Implicit Regularization of SGD via Thermophoresis
Authors: Mingwei Wei
David J. Schwab
Venue:  (2021)
Keywords: SGD
regularization
generalization
statistical mechanics
thermophoresis

Abstract: A central ingredient in the impressive predictive performance of deep neural networks is optimization via stochastic gradient descent (SGD). While some theoretical progress has been made, the effect of SGD in neural networks is still unclear, especially during the early phase of training. Here we generalize the theory of thermophoresis from statistical mechanics and show that there exists an effective entropic force from SGD that pushes to reduce the gradient variance. We study this effect in detail in a simple two-layer model, where the thermophoretic force functions to decreases the weight norm and activation rate of the units. The strength of this effect is proportional to squared learning rate and inverse batch size, and is more effective during the early phase of training when the model's predictions are poor. Lastly we test our quantitative predictions with experiments on various models and datasets.


Under review as a conference paper at ICLR 2021
IMPLICIT REGULARIZATION OF SGD VIA THER- MOPHORESIS
Anonymous authors Paper under double-blind review
ABSTRACT
A central ingredient in the impressive predictive performance of deep neural net- works is optimization via stochastic gradient descent (SGD). While some theoret- ical progress has been made, the effect of SGD in neural networks is still unclear, especially during the early phase of training. Here we generalize the theory thermophoresis from statistical mechanics and show that there exists effective force from SGD that pushes to reduce the gradient variance in certain parameter subspaces. We study this effect in detail in a simple two-layer model, where the thermophoretic force functions to decreases the weight norm and activation rate of the units. The strength of this effect is proportional squared learning rate and inverse batch size, and is more effective during the early phase of training when the model's predictions are poor. Lastly we test our quantitative predictions with experiments on various models and datasets.
INTRODUCTION
Deep neural networks have achieved remarkable success in the past decade on tasks that were out of reach prior to the era of deep learning. Yet fundamental questions remain regarding the strong performance of wver-parameterized models and optimization schemes that typically involve only first-order information, such as stochastic gradient descent (SGD) and its variants.
In particular, optimization via SGD is known in many cases to result in models that generalize better than those trained with full-batch optimization. To explain this, much work has focused on how SGD navigates towards so-called flat minima, which tend to generalize better than sharp minima (Hochreiter Schmichuber, 1997 Keskar et al. 2017). This has been argued by nonvacuous PAC- Bayes bounds (Dziugaite & Roy 2017) and Bayesian evidence (Smith & Le 2018). More recently, Wei & Schwab (2019) discuss how optimization via SGD pushes models to flatter regions within a minimal valley by decreasing the trace of the Hessian.
However, these perspectives apply to models towards the end of training, whereas it is known that proper treatment of hyperparameters during the early phase is vital. In particular, when training deep network one typically starts with a large learning rate and small batch size if possible. After training has progressed, the learning rate is annealed and decreased so that the model can be further trained to better fit the training set (Krizhevsky et al. 20122] Simonyan & Zisserman] [015 He et al.) 2016b at You et al. 2017: Vaswani al. (017). Crucially, using a small learning rate during the first phase of training usually leads to poor generalization and also result in large gradient variance practically (Jastrzebski et al., 2020: Faghri et al.) 2020).
However, limited theoretical work has been done to understand the effect of SGD on the early phase of training. Jasttrzbski al.] (2020) argue for the existence of "break-even" point on an SGD trajectory. This point depends strongly on the hyperparameter settings. They argue that the break- even point with large learning rate and small batch size tends to have a smaller leading eigenvalue of the Hessian spectrum, and this eigenvalue sets upper bound for the leading eigenvalue beyond this point. They also present experiments showing that large learning rate SGD will reduce the variance of the gradient. However their analysis focuses only the leading eigenvalue of the Hessian spectrum and requires the strong assumption that the loss function in the leading eigen- subspace is quadratic.

---

Under review as a conference paper at ICLR 2021
Meanwhile Li et al. (2000) studied the simple setting of two-layer neural networks. They demon- strate that in this model, training with large learning rate in the early phase tends to result in better generalization than training with small learning rate. To explain this, they hypothesize separation of features in the data: easy-to-gent toyeee--geer yet hard-to-fit features, and hard-to-generalize, easier- to-fit features. They argue that model trained with small learning rate will memorize easy-to- generalize, hard-to-fit patterns during phase one, and then generalize worse on hard-to-generalize, casicr-to-fit patterns, while the opposite scenario occurs when training with large learning rate. How- ever, this work relies heavily on the existence of these two distinct types of features in the data and the specific network architecture. Moreover, their analysis focuses mainly on learning rate instead of the effect of SGD.
In this paper, we study the dynamics of model parameter motion during SGD training by borrowing and generalizing the theory of thermophoresis from physics. With this framework, we show that during SGD optimization, especially during the early phase of training, the activation rate of hidden nodes is reduced is the growth of parameter weight norm. This effect is proportional to squared learning rate and inverse batch size. Thus, thermophoresis in deep learning acts as an implicit regularization that may improve the model'ss ability generalize.
We first give a brief overview of the theory of thermophoresis in physics in the next section. Then we generalize this theory to models beyond physics and derive particle mass flow dynamics microscop- ically, demonstrating the existence of thermophoresis and its relation to relevant hyperparameters. Then we focus simple two-layer model to study the effect of thermophoresis in detail. Notably, we find the thermophoretic force is strongest during the early phase of training. Finally, we test our theoretical predictions with a number of experiments, finding strong agreement with the theory.
THERMOPHORESIS IN PHYSICS
Thermophoresis, also known as the Soret effect, describes particle mass flow in response both diffusion and a temperature gradient. The effect was first discovered in electrolyte solutions (Lud-) wig. 1859 Soret 1897 Chipman 1926). However it was discovered in other systems such as gases, colloids, and biological fluids and solid (Janek et al. 2002: Köhler & Morozov] 2016).
Thermophoresis typically refers to particle diffusion
continuum with temperature gradient. In one method of analysis, the non-uniform steady-state density p is given by the "Soret Equilibrium"
(Eastman 1926 Tyrell & Colledge 1954, Wurger 2014).
Vp+ + PST VT =
where T is temperature and Sy is called the Soret coefficient.
In other work by de Groot & Mazur 1962), mass flow was calculated by non-equilibrium theory. They considered two types of processes for entropy balance: reversible process stands for the entropy transfer and an irreversible process corresponds to the entropy production, or dissipation. The resulting mass flow induced by diffusion and temperature gradient was found be
= -VD pD1 VT
where D is the Einstein diffusion coefficient and Dr is defined as thermal diffusion coefficient. Comparing the steady state in I and setting the flow to be zero, the Soret coefficient is simply
ST DT D
The Soret coefficient can be caleulated from molecular interaction potentials based specific molecular models (Wurger 2014).
3 THERMOPHORESIS IN GENERAL
In this section, we first study a kind of generalized random walk that has evolution equations for particle state with coordinate q {q: I==..... as
q = q -77[(1,))
(4)
2

---

Under review as a conference paper at ICLR 2021
where f is vector function, 7 and & are random variables, and nj is small number controlling the step size. Notice that this is a generalized inhomogeneous random walk for the particle. Before further analysis, it is noted that the evolution equations 4 is similar to SGD updates in machine learning and we will show this in the next section.
To isolate the effect of thermophoresis, assume the random walk is unbiased, in which P()f(q.E) a) = P()f(q.E) = -a),
(5)
for an arbitrary vector a. Thus there is no explicit force exerted on the particle. This simplification was used to demonstrate a residual thermophoretic force in the absence of a gradient. Including
gradients is straightforward and corresponds to an external field that creates a bias term. We also denote the probability density, which we also call the mass density, as p(q) and
(((g) = / ((a
(6)
so that (q) is the standard deviation of the random walk in the ith direction.
From a position q, we consider a subset of coordinate indices, U E (l..... ........,,, n). wherein
sign(Ji(q, x)) sigu(J (q,x)) and (:g;(9)
for all i, E U. We note here that indices will correspond to parameters when we study learning dynamics. The first property is necessary for our derivation. The second condition will be used at the end to conclude that each 9; decreases.
In order to study the dynamics of the particle and density function, we focus on the probability mass flow induced by the inhomogeneous random walk. We will show that there is always a flow from regions with larger (q) to those with smaller ()) for E U, which is a generalization of thermophoresis in physics.
Since 7 < 1, the movement of the particle will have a mean free path of g:(q) in ith direction. Therefore the random walk equation becomes
l Ø - (9)s,
(8)
where i 1,..., ... and Ç is a binary random variable with P(G = 11 = P(G: 1) 0.5. Moreover, from Eq. 7 we also have that C = C for all i and j E U.
Next we will show that the flow projecting on the subspace U is always toward smaller g, (q). Notice that although U can be multi-dimenssonal, the degree of freedom of the particle dynamics is 1 within U due to the sharing of the (s, and therefore the mass flow projecting on it is also I I-dimensional. For each i U, we define the average flow in this dimension to be the mass that enters q, from 9; minus the mass from the opposite direction 9,- From Eq. 8 and the assumption that 1) < 1, only mass close to q: will move across q at each step. We let the farthest mass that will flow across qi in step be q: A and 9: A,, where A and A positive. A and A, are thus defined implicitly by the equations: A 17g. (q A and A, (q A A-). respectively. Notice that if the random walk were homogeneous, we would have A = A,. In our inhomogeneous case, have A AT (9)(()) up to leading order of 17, and the next leading order will be calculated in order to compute the difference between A and A,
Now we are ready to calculate the mass flow through q. The mass flow projecting onto the subspace U is calculated by the mass through q from q A minus the mass from q A where A and A are as above for i U. It is straightforward to show that
A - A = 2n² Σ [,((9), + 0(72).
(9)
jeU
With this, we can compute the flow density, J. through q, finding
Σ (7(( Σ (:(())(pp()) -, Ei g. (())), p(g) O(1²),
J J= -
(10)
V
Eiev (
iEU iEU
3

---

Under review as a conference paper at ICLR 2021
plo (r)
pla 44)
+ A-
Figure I: Diagram of mass flow in generalized inhomogeneous random walk used in the derivation of the Soret coefficient.
where the derivation can be found Appendix A.3
This can be understood as described in Diagram Notice that this probability mass flow consists of two terms at order ry². The first represents diffu-
sion and the second corresponds to goal in this section, namely the flow due to thermophoresis. By the second property of the in Eq. 7 we find that the coefficient of thermophoresis (Soret coefficient), which is defined as
EiseU DI (
C == -1
(11)
2 Eieu g7(9)
I 0
(12)
negative. This means that there is effective force exerted a particle at position q towards the smaller variance regime (by analogy, the colder area). The coefficient is proportional to 17?.
4 MATHEMATICAL MODEL
4.1 TWO-LAYER MODEL
To study the physics behind SGD optimization in detail, we consider the simple setting of one- hidden layer neural networks. The network is function f R.M parameterized as follows:
V. Vo(Wx +b,,
N
M
Wayay b.).
1=1
We also write (xx for simplicity. The network has scalar output, which is widely used in regres- sion and binary classification. æ is the network input with dimension M, W and b are the weights and biases the first layer with dimension N x M and N respectively, where N is the number of hid- den nodes in the hidden layer, and a is the ReLU activation function defined as a (a) = max a).
The dataset is drawn i.i.d. from the data distribution, {(x, 8) (x, y) D(x, g) this paper we consider two cases, where either z; 2 0 or 2; ~ N'(0, 1 Here E Y and we denote the marginal distribution y as Dy. Finally, have the loss function L R x Y R
4.2 TRAINING
We consider optimization via SGD, where the gradient of the loss on batch of size |B] given by
A brief derivation can be found in Appendix A.2
2 Usually in convolutional neural networks or intermediate layers.
Often found when the data normalized.
4

---

Under review as a conference paper at ICLR 2021
B
VVn(V, W.
Σ
v (z
(13)
B
In our two-layer model, we have
Vv,f(x) 0 Σ
W Ki + bi),
Vw,,f(x)
Vi
W WaTE + b.).
k=1
M Vo.f(x) Vio
Wa + b,).
For an input vector x, we call the hidden node activated when o' E'E Wa + b,) 1, or equivalently War + b > O. We thus define the activation rate of the network to be
E
WA by)
(14)
N
This is an important concept to which we will return.
Henceforth, we drop the index i, since the dynamical equations are invariant with respect to node index, and write V = V. W; = Wij and b = b; by abuse of notation. We also denote
(V. W. b) = [V / (x)]²
hw (V. W. b) = ( (x))]
he W, b) == (x))
where E, denotes average over input x. We have the following property for the functions h: Property 4.1. Given W, Vf < V2 and
b2. have
(V,, W, b,) S (V,, W. b2).
hw (V1, W, b1) I lww(V, W. (2,b); ho[Vi, W, b,) I 1v2,,, W, W,b2): o' (Vi, W, W,b b1) < (V2, W,b2). Here we define S as min(b- a) 2 0. is straightforward to see the following:
Property 4.2. When the case x, 2 0 is considered, if V? I V2. W, I W2 and bj I bz. have
(V1,,, W1, bz) < h1 (V2, W2, bz),
Aw (V1, W2, b1) < (W2, W, by). hy W2, b1) < ho(V,, W, W,b)).
(V1, W, b1) < (V2, W,b)).
In our analysis, we focus for simplicity on binary classification tasks, where the loss is typically binary cross-entropy: I(f.v) y p(f) + v) In -(f)) and p(f) 1/(1 + ex)( We thus have
Vll(f = p(f) y.
(15)
Substituting into Eq. the mini-batch gradient becomes
B
1
VL b)
Σ pr. vf)xx)((()).
(16)
B
Our results also hold straightforwardly for squared error.
5

---

Under review as a conference paper at ICLR 2021
5 THERMOPHORESIS IN DEEP LEARNING
In this section, we will show that the parameters in the one hidden layer model and their dynamics approximately satisfy the criteria of the previous section and that the biases are pushed negative and V is suppressed during training, the effects of both of which are proportional to squared learning rate 17² and inverse batch size B|
The gradient that dominates model training is defined in 16 Because training samples are ii.d., the variance of the gradient is
LB
var VLB (V, W.b) = var
1
(p: y/ [[(xx)
(17)
B
1
var (p y) V fxx]
(18)
B
The gradient has two components: p- corresponding to y in equation and fx(x) corresponding to f(q. E). We assume that the dataset is unbiased, in which case Ply 0) P(y 1) 0.5 and
P(p - y/ a) P(p - = -a), and that y and V fx) are independent in the first period of training given that the dataset is complex and can't 't be learned by linear model. It straightforward to see that it satisfies Eq. |]
Next we will show that and b are always in the set of U, i.c. they satisfy the conditions of Eq. First, if V; 2 0, we have
Vv,((x) 0
W Wigay + b.),
(19)
/=1
20 0.
(20)
and
M Wa + b.),
Vo,((x = Vi0'
(21)
2 0.
(22)
Since we also have Property 4.1 the conditions in Eq. 7 are satisfied. If V. < 0, we consider coordinate transform that maps V, V, VV. is easy to show that Eq. is again satisfied after this transform.
Next we consider W. The gradient of with respect Wij is the product of Vo. and 2;. Iz for i = 1,..., M are always 2 O, which is usually the case in convolutional neural networks, it is easy to show that Wiy also set U and smaller W.y corresponds smaller variance according to Property 4.2 If .4 ~ (00 1), on the other hand, W is excluded from U.
For the following, we only consider the case where x; ~ (O 1). and
M
/ [(p
gv (V. W.,
y)
W Wayz + b,)] dd((z,)))
(23)
VIBI
V 2 (Wi,bi).
(24)
(V, 6,)
1
/ [(0 -o--iVo'
Wyar + 61)] =
(25)
V
22 W., b.)
(26)
where g is defined as in Eq. 6 Inserting these into Eq. to be
10 find the thermophoresis flow density
J/ = n²
(27)
B

---

Under review as a conference paper at ICLR 2021
Vidi d. V.. +V +V 230000
where v
p. This flow biases the model toward smaller by and smaller
2 +3
v with strength proportional to squared learning rate 17² and inverse batch size. We also note that i can be bounded by a function multiplying with a scalar (p- "))x(zz y). It is clear that this scalar measures the L-2 distance between model predictions and sample labels and decreases on average during training as prediction getting better. Thus thermophoresis is more effective during the early phase of training.
Therefore there exists an effective force that pushes to decrease the model's activation rate, defnnd in equation [4] and reduces the weight norm of the second layer. The strength of this force scales as
Fx 2 B
(28)
In Li & Liang 2018), Theorem 4.1 presents linear relation between learning rate and training iterations for target training error and small learning rate. This implies that if one uses a learning rate k times larger, the model will require & times fewer optimization steps for the same training performance. Together with our results, this implies the following: for the same model and initial- ization, comparing two optimization schemes with 7/1 S 1)2 each achieving a given training error, the activation rate for scheme 1 will be at least as large as that for scheme 2, i.e. 01 2 d2. Similarly, denoting the weight norm for scheme 1 (2) by 21 (12). we have that 11= 2 1/2-
Model sparsity can mean two different things: sparsity of the weights, and frequency with which units are activated, called the activation rate. Intuitively, a sparser model has a smaller capacity Bizopoulos & Koutsouris (2020); Kurtz et al. (2020); [Aghasi et al. (2017); [Ln et al. (2017). Fur- thermore, certain forms of model pruning have been shown to improve generalization Frankle & might surmise that smaller activation rate in general correlates with generalization. Moreover, in Appendix AA.5] we construct an upper bound of the Hessian norm and this upper bound monotonously depends on activation rate and weight norm. This also sheds light on the connection between sparsity, weight norm, and generalization.
Carbin (2019); Frankle al. (2020). Therefore
Our theory can also be generalized beyond two-layer models. We have shown that there exists an effective force in deep neural networks from SGD that reduces the gradient variance and have derived quantitative properties of it.
6 EXPERIMENTS
The essential result from the previous section is that there exists an effective force from SGD, anal- ogous to thermophoresis, that pushes to decrease the gradient variance, and in one-hidden-layer neural networks decreases the model's activation rate and reduces the weight norm of the second layer. The strength of the force is proportional to squared learning rate and inverse batch size. In this section, we present experiments to test these results. Further experiments can be found in the appendix.
First we consider a one hidden layer model with input dimension 100 and 100 hidden units. The input data, x, is distributed as (N, I) where is identity matrix, and the label is randomly chosen from (0. 1}. Batch size is set to and the learning rate is varied from 0.025 to 0.1. We calculate the activation rate and L2 norm of the vector V after each training iteration. The result for activation rate is shown the first row of Fig. 2 The leftmost plot shows activation rate as a function of true iteration the x-axis, and we that activation rate decreases during training, and the decreasing is more rapid with larger learning rate. In the middle plot we rescale the x-axis by a factor proportional to learning rate F This rescaling factor is to offset the movement difference due to learning rate difference. It is clear that even after this rescaling, we still observe that larger learning rates decrease activation rate faster. Finally, on the rightmost plot we rescale the x-axis with factor proportional to squared learning rate v We see that all trajectories now overlap, which matches our prediction in the previous section that decreasing rate is proportional to 2/².
largerr Vi V. < O.
Foor example, if raw iteration number for n 0.05 is 1000 and rescaled iteration number is also 1000, the rescaled iteration number for 0.1 is 1000 then its true iteration number 500.
7

---

Under review as a conference paper at ICLR 2021
I=01
Ir
6.88
1=0.5
Ire0.5
Activatien Rale
0,40
0.40
Activation
6.33
0.33
1r=0.1
fn00.5
1r=0.025
5000
2000
6000 Rescaled lterations (Ir"2)
lterationn
Rescalled lierations (Ir)
Ir00.01
1r:0.01
Ir=0.02 01
Gradient Vanaance
0.000216
fm.0.02
Grackent Variante
0.000055
Ir-0. 02
Gradient Variancce
0.000055
In-0 02
0.000224
0.000044
6.000044
2.000112
0.000122
60000122
20
50
20
32
10
Iterations
Rescaled lterations (Ir)
Rescaled iterations ((r2)
0.0007
Im:0.0
0.0077
0,000
Ir0000
Gradlent VardenetVaaaanne Vartance
2.0000
0m0000
Gradieen Varturce
0.0005
0.00.2
Gradient Vertarce
000000
0-00
0.0072
0.000s
00006
0.0001
0,0003
0.0003
0.0000
0.0000
0.0001
0.0001
0.000
10
30
20
50 40
30
20
90
40
50
Iterations
Rescaled lter derations (I))
Rescaled Iterations (1"2))
Figure 2: rows include rescaled x-axes as described in the main text. Top Row: Plots of activation rate as function of (rescaled) training iterations with different learning rates. The model two-layer fully-connected network with 100 hidden units. Training data is drawn from normal distribution. Middle Row: Plots of average gradient variance as function of (rescaled) training iterations with different learning rates in 6-layer fully-comnected neural networks. Training data is drawn from normal distribution. Bottom Row: Same as middle row but for 6-layer convolutional neural networks trained on Fashion-MNIST.
We next test our results for deep neural networks beyond the two-layer model. Instead of activation rate and weight norm, we plot the gradient variance predicted by our theory. Networks archi- tectures are 6-layer fully-connected with hidden layer sizes of 100 and 6-layer convolutional with 10 channels with kernel size of 5*5 and stride except the last fully-connected layer output. The results are shown in the second row Fig. and the third row of Fig. respectively.
Further experiments be found in Appendix A.6 where show that the evolution of the weight norm, scaling with batch size, and other results are consistent with our theoretical predictions. We also study other models and other datasets including CIFAR10.
7 CONCLUSION
In this paper we generalized the theory of thermophoresis from statistical mechanics and showed that there exists an effective thermophoretic force from SGD that pushes to reduce the gradient variance. We studied this effect in detail for simple two-layer model, where the thermophoretic force serves to decrease the weight norm and the activation rate of the units. We found that the strength this effect is proportional square of the learning rate, inversely proportional to batch size, and is more effective during the early phase training when the model's 's predictions are poor. We found good agreement between predictions and experiments various models and datasets.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Alireza Aghasi, Afshin Abdi, Nam Nguyen, and Justin Romberg, Net-trim: Convex pruning of deep neural networks with performance guarantee. Advances in Neural Information Processing Systems (NIPS), 2017.
Paschalis Bizopoulos and Dimitrios Koutsouris. /00,06690 2020.
Sparsely activated networks. arXiv preprint
John Chipman. The soret effect. Journal of the American Chemical Society, 48(10):2577-2589 1926.
S.R. de Groot and Mazur. Non-equlibrium Thermodynamics. North Holland Publishing, 1962.
Gintare Karolina Dziugaite and Daniel M Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. In UAI, 2017.
E. D. Eastman. Thermodynamics of non-isothermal systems. Journal of the American Chemical Society, 48(6):1482 1493, 1926.
Fartash Faghri, David Duvenaud, David J. Fleet, and Jimmy Ba. study of gradient variance in deep learning. arXiv preprint MrXiv:2007.04532 2020.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations (ICLR), 2019.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin. The lottery ticket hypothesis scale. arXiv preprint arXiv: 1tt.i:99300111,, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In ECCV, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016b.
Hochreiter and J. Schmidhuber. Flat minima. Neural Computation, 9(1):1-42, 1997.
Jürgen Janek, Carsten Korte, and Alan B. Lidiard. Thermodiffusion in lonic Solids -Model Exper- iments and Theory, pp. 146-183. Springer Berlin Heidelberg, 2002.
Stanislaw Jastrzebski, Maciej Szymezak, Stanislav Fort, Devansh Arpit. Jacek Tabor, Kyunghyun Cho, and Krzysztof Geras. The break-even point optimization trajectories of deep neural networks. In International Conference on Learning Representations (ICLR), 2020.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In ICLR, 2017.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In F. Pereira, C. J. C. Burges, L.. Bottou, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 25, pp. 1097-1105. Curran Associates, Inc., 2012.
Mark Kurtz, Justin Kopinsky, Rati Gelashvili, Alexander Matveev, John Carr, Michael Goin, William Leiserson, Sage Moore, Nir Shavit, and Dan Alistarh. Inducing and exploiting activa- tion sparsity for fast neural network inference. In International Conference Machine Learning (ICML), 2020.
Werner Köhler and Konstantin I. Morozov. The soret effect in liquid mixtures review. Journal of Non- Non-Equilibrium Thermodynamics, 41, 2016.
Yuanzhi Li and Yingyu Liang. Learning overparameterized neural networks via stochastic gradient descent structured data. In Advances in Neural Information Processing Systems (NIPS), 2018.

---

Under review as a conference paper at ICLR 2021
Yuanzhi Li, Colin Wei, and Tengyu Ma. Towards explaining the regularization effect of initial large learning rate in training neural networks. arXiv preprint arxiv:900..04555, 2020.
Lin, Yongming Rao, Jiwen Lu, and Jie Zhou. Runtime neural pruning. In Advances in Neural Information Processing Systems (NIPS), 2017.
C. Ludwig. Sitz. Ber. Akad. Wiss. Wien, 1859.
K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recogni- tion. In ICLR, 2015.
Samuel L. Smith and Quoe Le. A bayesian perspective generalization and stochastic gradient descent. In ICLR, 2018.
C. Soret. Arch Sci Phys Nat, 1897.
H. Tyrell and R. Colledge. Thermal diffusion potentials and the soret effect. Nature, 173:264-261 1954.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems (NIPS), 2017.
Mingwei Wei and David J Schwab. How noise affects the hessian spectrum in overparameterized neural networks. arXiv preprint arX1v:1910.00195. 2019.
Alois Wurger. soret equilibrium a non-equilibrium effect? arXiv preprint
2014.
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint Ar0: arXiv:1708.03888, 03888,, 2017.
10

---

Under review as a conference paper at ICLR 2021
A
APPENDIX
A.1 PROOF OF PROPERTY
Proof. By definition, we have
M
Wazy +b,) - o 2
W Wyzy + b2)
(29)
M
M
o'
Wyzz + bb)) 0'
Σ
Wazz +
(30)
==
j=1
For hyp
(V1, W.b.) = Exg. (x, V1, W.bi b1)
= E,
Wjaj + b1)
j=1
M
< E
Wjæz + b2)
(V2, W. b2)
Similarly, we have
M E, V2230'' hw (V1, W.b,
Wert + b1)
M
- E.g V22 20
WiT + bb)
< E Vv.V V22 270
WyTt b2)
A=1
=,,, (V2, W,b2)
Clearly the inequality also holds for has
A.2 DERIVATION OF EQ.
A
A p011q A) l) A7),
(31)
= > A
+ ロ ( + 0(7
(32)
jEU
27/² Σ
9 (9))))0(())000))
(33)
JEU
11

---

Under review as a conference paper at ICLR 2021
A.3
DERIVATION OF EQ.
10
+
pQq + A A lo A A7)
2
1
lpp(q
p(q
}]] (I |||- 11)))pp - -A7,,
2
1
+ A+) p --)
+
(1
+
2
A A-
A
+ 2
A
1
pqq A
2
A
1 EiEt ( + (A) - 4,)
A
A
[p(())
p(q A-),
2
A+ A-
p(q) + 0 (7) 9(((),p(a)) - -1p Eiser 9, (9)9, (9)0,9iqq)
T
Σ
Σ
VEeeu (g
IEU
VEU
A.4 SANITY CHECK OF GENERALIZED THEORY
If [ = and 9: (q) (:()),, the model will reduce to aforementioned physics model and the Soret coefficient reduces
2 g())g(())
(34)
K ng(g)
)31'
(35)
2
VT
(36)
where T is the effective temperature in the model. This result is consistent with thermophoresis model in physics.
A.5 SPARSITY, WEIGIT NORM AND THEIR RELATION TO GENERALIZATION
In this section, we demonstrate how sparsity related to the Hessian norm. We first denote the model's probabilistic prediction C-class classification as
exp =
(37)
Σ exp
where k is the probability for label k. is the data index, is model output and C' is the total number of categories. We consider cross entropy loss of the form
L(w) B
v log pk
(38)
k=1
where y/ sample labels and p stands for model probability prediction, similar the previous definition. We denote the loss for individual sample to be
L
skk log rk
(39)
The gradient with respect the model output is
(V.L")) 1/ + P
(40)
And it easy to show that the Hessian with respect to output is
(V2 1 Surrh pirl
(41)
12

---

Under review as a conference paper at ICLR 2021
Therefore the Hessian with respect to model parameters is
H = V (=L((((w)
(42)
VV V
(43)
(V + V.LV
(44)
= (V (V) "= )ar
(45)
To study the spectrum of the Hessian, we calculate the trace and have
Tr(H") = Tr((V ~ 2 (V =
(46)
=V)((V (V="))) w=
(47)
= =Tr(P*K) *T(P)))T((K...
(48)
(49)
where
P V.1"
(50)
Ozy ) Owly
Ozy aw!y
Kyn ΣΣ
(51)
The trace of K therefore can be calculated by chain rule,
Ozu )2 Owly
Tr(K)= ΣΣΣ
(52)
Σ (8ll)) (1
(53)
where 8 and h carry backward and forward information respectively. They are defined as
p[[p]] 6pm, WT When.,,''.... o'..WWl.. mi12t
(54)
h 1-1 = oWi-l o... G...Wreng
(55)
jm:
It can further be shown that
Σ iu
Jan WL nenn
0' wit1 d + o'14 M4421
o WL
n mng Onen
(56)
im
Tr(WLd'... ... www.o//............oo...... o'W'
(57)
< SPHoWWWWW/TrGoW--ww---- Tro
(58)
as well
ΣON / WW Tr (. (ooWl-1)
(59)
Together with the previous calculations and the definition of K, we have
Tr(K) < ||x||2 TT" To
(60)
= =||X||2 J B
(61)
Finally, we derive upper bound for the trace of the Hessian,
Tr(H") < [x(1) Σ T low" I 2 [
(62)
Notice that activation rate and weight norm control the magnitude of ||o and | Therefore smaller activation rate and weight norm lead to tiger upper bound of the Hessian trace and thus indicate smaller matrix norm. This analysis connects sparsity with Hessian norm, Hessiam trace specifically.
13

---

Under review as a conference paper at ICLR 2021
0-0
0.50
In00.1
1r:0.5
0.45
120.025
1r-00025
Activalon Rate
Activation Rate
9.40
0.99
0.40
0.35
0.33
0.85
|=00.1
1=0.5 5
0.25
In00 025
0.20
Iterations
Rescaled Iterations (r)
Rescaled Iterations (I+12)
Figure Plots of activation rate as functions of training iterations with different learning rate.
1.0
10
Weigt Norm
0.0
Weight Norm
0.8
0.8
0.8
0.8
0.7
0.7
Y=0.1
Ir:0.1
Ire0.1
1=00.5
I=:0.5
IneO. 5
0.8
Y=00.025
2000
4000
0033
2000
4000
I003
2000
I000 lterations (lr"2)
2003
Rescaled lterations (1)
Figure 4: Plots of L2 norm of V functions of training iterations with different learning rate.
A.6 MORE EXPERIMENTS
In this section, we design extensive experiments to test the obtained results. First of all, we consider binary classification with BCE loss. The model has the form
f(x) Vo (xx b)
(63)
Input dimension 10 and the number of hidden nodes is also 10. x is distributed NV I) where I is identity matrix and the label is randomly chosen from 10.
For the first hyperparameter setting, batch size is 1 and learning rate varies from 0.025 0.1. We calculate the activation rate and L. norm of the vector V after each training iteration. The result and Fig. respectively. Both figures
for activation rate and weight norm are shown in Fig. contain three plots. The leftmost plots correspond to plot with raw iteration x-axis. It is shown that both activation rate and weight norm are decreasing for cases. Additionally, the decreasing rate is larger with larger learning rate. The middle plots correspond to plot with rescaled iteration as x-axis, where the rescaled factor proportional to learning rate | This rescalation factor is to offset the movement difference due learning rate difference. It is clear that even after this rescalation, we still observe activation rate and weight norm difference for different learning rates. Lastly, the rightmost plots correspond plot with rescaled iteration x-axis, where the rescaled factor is proportional to squared learning rate 2 The result that all trajectories overlap with each other matches our prediction in the previous section, that decreasing rate proportional 172.
For the second hyperparameter setting, learning rate is fixed to be 0.05 and batch size varies from 3. Again we calculate the activation rate and L2 norm of the vector after each training
Forr example, if raw iteration number for 7 0.05 is 1000 and rescaled iteration number is also 1000, the rescaled iteration number for 0.1 is 1000 then its true iteration number 500.
14

---

Under review as a conference paper at ICLR 2021
0.50
bs=1
0.00
bs=1
0.45
bo=2
hs=2
Activation Raate
Activatioo Rale
0.35
0.33
0.26
0.86
0.0
Iterations
Rescaled iterations (1/bs)
Figure 5: Plots of activation rate as functions of training iterations with different batch size.
1.0
Weigh
0.6
Weight
0.
te:=1
bss1 b
0.6
bs:12
0.6
bs=3
bsm3
I0000
15000
20000
25000
10000
150000
20000
20000
lterations
Rescaled Iterations (1/bs))
Figure 6: Plots of L2 norm of V as functions of training iterations with different batch size.
iteration. The result for activation rate and weight norm are shown in Fig. 5 and Fig. 6 respectively. We observe similar tendency shows that both activation rate and weight norm are decreasing for all cases. While there exist decreasing rate difference in the left plots due to batch size discrepancy. This difference can be offset be rescaling x-axis according to proportional factor B/| which is the result in the right plots and it consistent with our theory prediction.
discussed in the previous hyperparameter setting. The result
Subsequently consider that second case discussed in the previous section. This is also a consider binary classification with BCE loss. The model, however, has the form
f(x = Vo(W*)
(64)
Input dimension is 10 and the number of hidden nodes is also 10. x is distributed uniformly between O and 1 as (O0, and the label is randomly chosen from 1).
The first setting is the same as the first setting in the aforementioned experiment. The result for activation and weight norm are shown in Fig. and Fig. respectively. Similar to previous experiment, both figures contain three plots. The lefumost, middle, right plots correspond to plot with raw iteration, rescaled iteration with factor 7) and rescaled iteration with factor ny respectively as x- axis. The result that all trajectories overlap with each other matches our prediction in the previous section, that decreasing rate is proportional to n?
The second setting is also similar to the second setting in the previous experiment where we fix learning rate to be 0.05 and vary batch size from 3. The result for activation rate and weight norm are shown in Fig. 9 and Fig. 10 respectively. The result shows that both activation rate and weight norm are decreasing for cases. While there exist decreasing rate difference the left plots due batch size discrepancy. This difference can be offset be rescaling x-axis according to
15

---

Under review as a conference paper at ICLR 2021
0.50
0.50
0.50
r=0.1
Ir:0.1
0.45
0.45
In:0.5
Activation Rale
r=0. 025
Activation Rate
0.40
In-0.00 0.000
0.35
0,36
0.50
0.90
0.25
0:2
0.25
0m00.5
0.20
(=0 025
0.25
1500
Iterations
Rescaled lterations (r)
Rescaled lterations (Ir2)
Figure 7: Plots of activation rate as functions of training iterations with different learning rate.
M
Ine0.1
0.50
r= 50.5
0.50
Ire0.5
Ined 025
Weight Norm
0.75
0.75
Weight Norn
6.75
0.70
Weigght
0.70
0.70
m
I=0,1
lm0
0.60
0.00
500
1000
2000
2000
2500
2000
500
1000
1300
2003
Rescaled Iterations (r)
Rescaled
Iterations
Figure 8: Plots of L2 norm of V as functions of training iterations with different learning rate.
proportional factor 1/|B], which is the result in the right plots and it is consistent with our theory prediction.
Furthermore, we use real image dataset CIFARIO for the next experiment instead of artificial data. The model in this experiment has one hidden layer with 300 hidden nodes. We first fix batch size to shown Fig. 11 and Fig. [2 respectively We then fix learning rate to be 0.02 and vary batch size. The result of activation rate and weight norm decreasing are shown in Fig. 13 and Fig. respectiivlll
be 1000 and vary learning rate 7). The result of activation rate and weight norm decreasing
is clear that the result matches our theoretical predictions. We skip detailed analysis here as is similar discussion in previous experiments.
bs 1
bss1
0.45
Azivalion Rate
0.40
Activatio Rate
0.35
0.50
6.80
0.25
0.25
0.00
611
0.15
Rescaled lterations (1/bs)
Figure Plots activation rate as functions of training iterations with different batch size.
16

---

Under review as a conference paper at ICLR 2021
0.55
0.85
bo=11
hs=1
0.40
b
0.00
b
bs=3
weight Norm
0.75
Weight Norm
0.75
6.70
0.85
0.90
0.88
Iterations
Rescaled Iter iterations (1/bs)
Figure Plots of L2 norm of V as functions of training iterations with different batch size.
0525
1re0
1=02
0.50
0.90
I==0.1
F=11
Inw0.05
1-005
Adtivaton Rate
Activation Rale
0.46
0.65
0.44
0.40
1=0.2
0.40
0.355
0.56
1=0.05
0.88
6360
00
25
5.0
7.5
10.0
12.5
15.0
00
23
5.0
7.5
10.0
12.
15.0
0.0
7,5
100
12.
15.0
Iterations
Rescaled Iterations (t)
Rescaled iteration (Ir12)
Figure 11: Plots of activation rate as functions of training iterations with different learning rate. Dataset is CIFAR10.
17

---

Under review as a conference paper at ICLR 2021
0.001225
000150
0:001100
I=0.11
0.0110
r=:..
Ir-011
0.0008
Weight Norm
0.00100
Weight Norm
0:001000
Weight Norm
0.001000
8:000975
0.0022
1=:022
000030
1n:0.55
0.000055
0,0000
0.0075
6.0
75
16.0
tterations
Rescaled Ilerations (I))
Rescaled Iterations (Ir'2
Figure 12: Plots of L.2 norm of V as functions of training iterations with different learning rate. Dataset is CIFARIO.
bs 1000
1000
500
500
bs 250
0.43
bs 250
Activation Fall
Activation Rale
0.40
0.86
6.00
0.25
Iterations
Rescaled Iterations (1/bs)
Figure 13: Plots of activation rate as functions of training iterations with different batch size. Dataset CIFAR10.
0.0011
bs-1000
bs-1000
hs:500
bss500
bs:220
ILC000S
bs=250
Weight Norm
Weigght
GLC00.0
0.009
RLCCERS
0,C0090
0,008
0.00085
0L00000
lterations
Rescaled Iterations (1/b)
Figure 14: Plots of L2 norm of V as functions of training iterations with different batch size. Dataset is CIFAR10.
18


Reviews:
[{"review_id": "vVHhYCEMP9H", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the implicit regularization effect of SGD from the Thermophoresis perspective. The authors find several quantities scale with square of learning rate over batch size, including activation rate and gradient variance. The theory, to my understanding, requires several strong assumptions, e.g., (1) zero true gradient, and (2) certain implicit independence condition. The empirical results in real dataset like CIFRAR-10 do not seem to support the conclusion, either. As someone from CS/Applied Math community, I personally encourage the authors to write the theory in a more mathematical manner: I find several of the current reasoning in the main text hard to grasp ---- maybe I am missing important Physics backgrounds.\n\nBelow are my detailed reviews ---- it is highly possible that I misunderstand something; if so please do clarify.\n\n# Theory\n\n- Contradicting statements in Page 2 the second paragraph vs. Page 2 Eq. (3). In specific you study steady state and let J to be zero. However when the dynamic reaches its steady state, it is no longer directly related to the \"early phase\", which is claimed as the focused object in Page 2 the second paragraph. \n\n- Eq. (5). I am super confused here. Does Eq. (5) imply the expectation of the gradient is zero??? This is not at all true for GD/SGD. I mean how can you effectively optimize an objective with zero (expected) gradient?\n\n- Eq. (9) and Eq. (10). I cannot understand the paragraphs for Eq. (9) and (10). Recall the definition of U in Eq. (7), I believe U should also depend on time t? Then what do you mean by projecting the flow on to the subspace U, where the subspace U is not even fixed? I tried to look at the appendix, but sorry I am unable to follow the abbreviated derivation.\n\n- Property 4.2. What is W here?\n\n- Below Eq. (18). Why P(p-y = a) = P(p-y = -a)??? To my understand, if we write down the complete formula, it reads \nP ( p(f(x_i, theta)) - y_i = a ) = P( p(f(x_i, theta)) - y_i = -a ), \nwhere theta is (W, V, b) the parameters. Even the dataset is unbiased, i.e., P(y = 0) = P(y=1), after theta being injected, we no longer have independence for p, thus I do not understand the above equality.\n\n\n- Below Eq. (22). Why can you consider a transformation that maps V to -V??? Intuitively this transformation changes a descent iterate to an ascent iterate and vice versa. Does not it change the behavior of the original dynamic??\n\n\n\n# Experiments\nThe results in toy datasets seem to be good. But when I look at Appendix A.6, the plots for CIFAR-10 do not seem to agree with your theory in my humble perspective. \n\n# Vague Statements\n- Page 7, the third paragraph. I understand sparsity of weights implies small capacity; however I do not see why sparsity of the activations also imply small capacity. In specific, it can be that the rate of activation is small, but for each data the activated neurons are different. In this case can you prune the network as stated in the paragraph?\n- The title and abstract emphasize the paper concerns SGD. But I fail to find a direct connection between SGD and the so called \"mass flow\" in Eq. (1). I suspect the tile and abstract are over claimed. \n\nOverall, I cannot give high scores to this paper. Again, I can miss important facts because of lacking Physics background. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "m4lsviAye_o", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\n\nThis paper uses the concept of thermophoresis from statistical mechanics to study certain aspects of the dynamics of SGD in neural network training. It is argued that this explains SGD’s tendency to reduce the gradient variance and that the strength of this effect depends on the setting of the step size and batch size.\n\n\n## Rating\n\nThe paper presents an interesting finding, namely, that SGD naturally seeks to reduce gradient variance. This is an interesting potential avenue to understand to effectiveness of SGD in neural network training. The main argument of the paper relies on a physics analogy. For a theoretical paper studying a mathematical process (SGD), the mathematical reasoning seems very vague and I have problems following it in various steps. I am listing some specific questions below. If the mathematical reasoning can be made precise and is explained clearly, this could be a good contribution. In its current form, I don’t think the paper plausibly support its claim of showing “that there exists an effective entropic force from SGD that pushes to reduce the gradient variance”, even for the simple model problem of a single-hidden layer neural net. I therefore recommend **rejection** for now, but would like to encourage the authors to respond to my questions below during the rebuttal phase.\n\n\n## Individual Comments and Questions\n\n1) The main argument is based on a physics analogy. I put a lot of effort into it, but I just can’t follow the chain of reasoning here. Up to Eq. (8), we are talking about a particle state $q$, which evolves stochastically in discrete time steps. That is a vector-valued discrete time stochastic process. In the paragraph after Eq. (8), the paper is talking about a notion of “mass flow through $q$”. What is that supposed to mean mathematically? It seems to me that we are talking about the probability of transitioning into state $q$ from any other state. Is that correct? I’m all for intuition, but for a theoretical paper, it still needs to be clear what mathematical quantities we are talking about.\n\n2) Eq. (7) restricts consideration to a subset of coordinates. This restriction is not discussed in the paper at all.\n    a) Why is that necessary? Is it a technicality or a can we really only say something about such a subset of coordinates.\n    b) Is there an intuitive meaning to that subset? The second condition in Eq. (7) seems mysterious and it is not checked whether this holds in the model problem of Section 5.\n    c) In the derivation of Eq. (9), why can the “mass flow” be restricted to that set $U$?\n    d) How and why can we generalize a finding that holds only on such subset? The abstract characterizes the finding as: “show that there exists an effective entropic force from SGD that pushes to reduce the gradient variance.” Is it really this general?\n    e) What would these subsets look like in practical SGD? I can see from Section 5, that such subsets might correspond to layers when using ReLU nonlinearities. But beyond that and when using nonlinearities that can take both positive and negative values, I can’t see where we would have sets of parameters, whose partial derivatives have the same sign **for all inputs $x$**.\n\n3) Glossing over the issues mentioned in point (1), the derivation of the main result relies on **a lot** of relatively crude approximations.\n    a) In Eq. (5), an unbiased random walk is assumed. Transferred to SGD, this would imply that the mean gradient is zero, which seems almost fundamentally at odds with the setting of gradient-based optimization. How can this be reconciled. Can a non-zero mean gradient be inserted into the model and can its effect be isolated? This is totally non-obvious to me.\n    b) Eq. (5) further implies a symmetric noise distribution, which is a very strong assumption. In the 2-layer NN studied in Section 5, some gradients are strictly positive irrespective of the input $x$ (e.g. Eq. 19), which is basically irreconcilable with the assumption of symmetric gradient noise.\n    c) Eq. (8) is a drastic simplification of the true process given by Eq. (4). It is approximating a continuous with a binary increment.\nApproximations are necessary when studying complex phenomena. But they need to be carefully discussed and justified. I feel that the paper is not delivering that.\n\n4) The experiments compare gradient variance (and related quantities) along trajectories obtained with different step sizes. It is observed that the per-step reduction in gradient variance per step is roughly proportional to the squared step size, which echos the theoretical findings. But how can we know that this reduction in gradient variance is not just an effect of the gradients becoming smaller overall? It would be much more interesting to see the gradient variance relative to the (squared) norm of the mean gradient as a quantity that characterizes “training progress” independent of the step size. For similar values of the gradient magnitude, does large-step SGD seek out regions with smaller gradient variance then small-step SGD?\n\n\n## Typos / Style\n\n- Second-to last paragraph of page 1: “Jastrzebski at al. (2020) argues” should be “argue”\n- Near bottom of page 1: “large learning SGD” should probably be “large learning rate SGD”\n- First paragraph of page 2: “than” should be “then”\n- Middle of page 3 says there is a flow “toward negative $g_i(q)$”. I think what you want to say here is that there’s a flow toward **smaller** $g_i(q)$, since $g_i(q)$ is a nonnegative quantity.\n- Middle of page 3: $\\eta \\ll 0$ should be $\\eta \\ll 1$\n\n\n## Update after Rebuttal\n\nThanks for the detailed reply. You have answered a number of my questions, but many of my main concerns remain, in particular points 3 b) and c) as well as 4). Furthermore, incorporating all the clarifications and additional discussions into the paper would in my opinion amount to a substantial revision beyond the usual scope of revisions after rebuttal. I will thus stick to my original rating and recommend rejection for this paper.", "extra_scores": {}, "extra_text": {}}, {"review_id": "FsF_V1kISu5", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "1. Summary:\n\nThis paper proposes that SGD has the implicitly bias of reducing gradient variance via the phenomenon of thermophoresis that masses tend to flow from regions with higher temperature / variance of random walk to regions with lower temperature / variance of random walk. In the setup of two-layer neural networks trained by SGD for binary classification, the authors show the analogous phenomenon that the model is biased towards smaller activation rate and the norm of its second layer weight. The dependence of the rate at which this phenomenon happens on the learning rate and batch size are verified in the experiments.\n\n2. Clearly state the recommendation (accept or reject) with one or two key reasons for this choice:\n\nI am recommending an accept to this paper, as this work presents a reasonable and empirically-supported theory for the implicit regularization effect of SGD on two-layer neural networks.\n\n3. Arguments for the recommendation and questions for the authors:\n\nThe authors did a good job explaining the general theory of thermophoresis as well as the derivation of the effect on the two-layer neural network model. The experiment agrees with the theoretical prediction quite nicely.\n\n4. Additional questions and comments:\n\na) On the bottom of Page 5, the authors claim that the results can be generalized to arbitrary loss. Is this obvious?\n\nb) The condition that $x_i$ is nonnegative seems unnatural in practice, because I think in practice techniques like data whitening are sometimes applied during preprocessing, which centers the input data. \n\nc) On Page 6, the authors mention “the set of U defined in the previous section”, but I couldn’t find it defined in Section 4, and also the set U defined in Section 3 seems like something separate to me - it is a set of indices rather than a set of weights.\n\nd) In the derivation on Page 3, how do we know that $\\Delta_i^+ = \\Delta_i^- = 0$ if $i \\notin U$? Because in the end, which indices belong to $U$ is not fixed, and the complement of a $U$ may also be a legitimate $U$ itself, for example.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper uses concepts from physics to make predictions about stochastic gradient descent. The reviews point to two issues. Firstly, the paper was not very accessible to those without a relevant background, and this is reflected in the low confidence rating reviewers gave. More importantly, two of the reviewers consistently pointed out 'vague mathematics' and oversimplification in the mathematical arguments.

The authors' feedback did not successfully address the reviewer's concerns, both R3 and R4 indicated there were outstanding concerns.

I should note that despite giving low confidence scores and stating that some concepts from physics are beyond their field of expertise, reviewers gave high quality reviews with detailed comments and questions, and subsequently participated in the discussion revisiting their reviews. This suggests that the low confidence is not a symptom of insufficient reviewer effort, but perhaps a consequence of an inaccessible paper.
Decision: Reject