Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: ## Summary

This paper uses the concept of thermophoresis from statistical mechanics to study certain aspects of the dynamics of SGD in neural network training. It is argued that this explains SGD’s tendency to reduce the gradient variance and that the strength of this effect depends on the setting of the step size and batch size.


## Rating

The paper presents an interesting finding, namely, that SGD naturally seeks to reduce gradient variance. This is an interesting potential avenue to understand to effectiveness of SGD in neural network training. The main argument of the paper relies on a physics analogy. For a theoretical paper studying a mathematical process (SGD), the mathematical reasoning seems very vague and I have problems following it in various steps. I am listing some specific questions below. If the mathematical reasoning can be made precise and is explained clearly, this could be a good contribution. In its current form, I don’t think the paper plausibly support its claim of showing “that there exists an effective entropic force from SGD that pushes to reduce the gradient variance”, even for the simple model problem of a single-hidden layer neural net. I therefore recommend **rejection** for now, but would like to encourage the authors to respond to my questions below during the rebuttal phase.


## Individual Comments and Questions

1) The main argument is based on a physics analogy. I put a lot of effort into it, but I just can’t follow the chain of reasoning here. Up to Eq. (8), we are talking about a particle state $q$, which evolves stochastically in discrete time steps. That is a vector-valued discrete time stochastic process. In the paragraph after Eq. (8), the paper is talking about a notion of “mass flow through $q$”. What is that supposed to mean mathematically? It seems to me that we are talking about the probability of transitioning into state $q$ from any other state. Is that correct? I’m all for intuition, but for a theoretical paper, it still needs to be clear what mathematical quantities we are talking about.

2) Eq. (7) restricts consideration to a subset of coordinates. This restriction is not discussed in the paper at all.
    a) Why is that necessary? Is it a technicality or a can we really only say something about such a subset of coordinates.
    b) Is there an intuitive meaning to that subset? The second condition in Eq. (7) seems mysterious and it is not checked whether this holds in the model problem of Section 5.
    c) In the derivation of Eq. (9), why can the “mass flow” be restricted to that set $U$?
    d) How and why can we generalize a finding that holds only on such subset? The abstract characterizes the finding as: “show that there exists an effective entropic force from SGD that pushes to reduce the gradient variance.” Is it really this general?
    e) What would these subsets look like in practical SGD? I can see from Section 5, that such subsets might correspond to layers when using ReLU nonlinearities. But beyond that and when using nonlinearities that can take both positive and negative values, I can’t see where we would have sets of parameters, whose partial derivatives have the same sign **for all inputs $x$**.

3) Glossing over the issues mentioned in point (1), the derivation of the main result relies on **a lot** of relatively crude approximations.
    a) In Eq. (5), an unbiased random walk is assumed. Transferred to SGD, this would imply that the mean gradient is zero, which seems almost fundamentally at odds with the setting of gradient-based optimization. How can this be reconciled. Can a non-zero mean gradient be inserted into the model and can its effect be isolated? This is totally non-obvious to me.
    b) Eq. (5) further implies a symmetric noise distribution, which is a very strong assumption. In the 2-layer NN studied in Section 5, some gradients are strictly positive irrespective of the input $x$ (e.g. Eq. 19), which is basically irreconcilable with the assumption of symmetric gradient noise.
    c) Eq. (8) is a drastic simplification of the true process given by Eq. (4). It is approximating a continuous with a binary increment.
Approximations are necessary when studying complex phenomena. But they need to be carefully discussed and justified. I feel that the paper is not delivering that.

4) The experiments compare gradient variance (and related quantities) along trajectories obtained with different step sizes. It is observed that the per-step reduction in gradient variance per step is roughly proportional to the squared step size, which echos the theoretical findings. But how can we know that this reduction in gradient variance is not just an effect of the gradients becoming smaller overall? It would be much more interesting to see the gradient variance relative to the (squared) norm of the mean gradient as a quantity that characterizes “training progress” independent of the step size. For similar values of the gradient magnitude, does large-step SGD seek out regions with smaller gradient variance then small-step SGD?


## Typos / Style

- Second-to last paragraph of page 1: “Jastrzebski at al. (2020) argues” should be “argue”
- Near bottom of page 1: “large learning SGD” should probably be “large learning rate SGD”
- First paragraph of page 2: “than” should be “then”
- Middle of page 3 says there is a flow “toward negative $g_i(q)$”. I think what you want to say here is that there’s a flow toward **smaller** $g_i(q)$, since $g_i(q)$ is a nonnegative quantity.
- Middle of page 3: $\eta \ll 0$ should be $\eta \ll 1$


## Update after Rebuttal

Thanks for the detailed reply. You have answered a number of my questions, but many of my main concerns remain, in particular points 3 b) and c) as well as 4). Furthermore, incorporating all the clarifications and additional discussions into the paper would in my opinion amount to a substantial revision beyond the usual scope of revisions after rebuttal. I will thus stick to my original rating and recommend rejection for this paper.
review_title: Interesting finding, but the mathematical reasoning is imprecise

final_score: rating: 7: Good paper, accept
review_content: 1. Summary:

This paper proposes that SGD has the implicitly bias of reducing gradient variance via the phenomenon of thermophoresis that masses tend to flow from regions with higher temperature / variance of random walk to regions with lower temperature / variance of random walk. In the setup of two-layer neural networks trained by SGD for binary classification, the authors show the analogous phenomenon that the model is biased towards smaller activation rate and the norm of its second layer weight. The dependence of the rate at which this phenomenon happens on the learning rate and batch size are verified in the experiments.

2. Clearly state the recommendation (accept or reject) with one or two key reasons for this choice:

I am recommending an accept to this paper, as this work presents a reasonable and empirically-supported theory for the implicit regularization effect of SGD on two-layer neural networks.

3. Arguments for the recommendation and questions for the authors:

The authors did a good job explaining the general theory of thermophoresis as well as the derivation of the effect on the two-layer neural network model. The experiment agrees with the theoretical prediction quite nicely.

4. Additional questions and comments:

a) On the bottom of Page 5, the authors claim that the results can be generalized to arbitrary loss. Is this obvious?

b) The condition that $x_i$ is nonnegative seems unnatural in practice, because I think in practice techniques like data whitening are sometimes applied during preprocessing, which centers the input data. 

c) On Page 6, the authors mention “the set of U defined in the previous section”, but I couldn’t find it defined in Section 4, and also the set U defined in Section 3 seems like something separate to me - it is a set of indices rather than a set of weights.

d) In the derivation on Page 3, how do we know that $\Delta_i^+ = \Delta_i^- = 0$ if $i \notin U$? Because in the end, which indices belong to $U$ is not fixed, and the complement of a $U$ may also be a legitimate $U$ itself, for example.
review_title: Review

final_score: rating: 3: Clear rejection
review_content: This paper studies the implicit regularization effect of SGD from the Thermophoresis perspective. The authors find several quantities scale with square of learning rate over batch size, including activation rate and gradient variance. The theory, to my understanding, requires several strong assumptions, e.g., (1) zero true gradient, and (2) certain implicit independence condition. The empirical results in real dataset like CIFRAR-10 do not seem to support the conclusion, either. As someone from CS/Applied Math community, I personally encourage the authors to write the theory in a more mathematical manner: I find several of the current reasoning in the main text hard to grasp ---- maybe I am missing important Physics backgrounds.

Below are my detailed reviews ---- it is highly possible that I misunderstand something; if so please do clarify.

# Theory

- Contradicting statements in Page 2 the second paragraph vs. Page 2 Eq. (3). In specific you study steady state and let J to be zero. However when the dynamic reaches its steady state, it is no longer directly related to the "early phase", which is claimed as the focused object in Page 2 the second paragraph. 

- Eq. (5). I am super confused here. Does Eq. (5) imply the expectation of the gradient is zero??? This is not at all true for GD/SGD. I mean how can you effectively optimize an objective with zero (expected) gradient?

- Eq. (9) and Eq. (10). I cannot understand the paragraphs for Eq. (9) and (10). Recall the definition of U in Eq. (7), I believe U should also depend on time t? Then what do you mean by projecting the flow on to the subspace U, where the subspace U is not even fixed? I tried to look at the appendix, but sorry I am unable to follow the abbreviated derivation.

- Property 4.2. What is W here?

- Below Eq. (18). Why P(p-y = a) = P(p-y = -a)??? To my understand, if we write down the complete formula, it reads 
P ( p(f(x_i, theta)) - y_i = a ) = P( p(f(x_i, theta)) - y_i = -a ), 
where theta is (W, V, b) the parameters. Even the dataset is unbiased, i.e., P(y = 0) = P(y=1), after theta being injected, we no longer have independence for p, thus I do not understand the above equality.


- Below Eq. (22). Why can you consider a transformation that maps V to -V??? Intuitively this transformation changes a descent iterate to an ascent iterate and vice versa. Does not it change the behavior of the original dynamic??



# Experiments
The results in toy datasets seem to be good. But when I look at Appendix A.6, the plots for CIFAR-10 do not seem to agree with your theory in my humble perspective. 

# Vague Statements
- Page 7, the third paragraph. I understand sparsity of weights implies small capacity; however I do not see why sparsity of the activations also imply small capacity. In specific, it can be that the rate of activation is small, but for each data the activated neurons are different. In this case can you prune the network as stated in the paragraph?
- The title and abstract emphasize the paper concerns SGD. But I fail to find a direct connection between SGD and the so called "mass flow" in Eq. (1). I suspect the tile and abstract are over claimed. 

Overall, I cannot give high scores to this paper. Again, I can miss important facts because of lacking Physics background. 

review_title: Not Fully Clear about the Mathematics / Physics, Not Ideal Empirical Results in Real Datasets

Meta-review: This paper uses concepts from physics to make predictions about stochastic gradient descent. The reviews point to two issues. Firstly, the paper was not very accessible to those without a relevant background, and this is reflected in the low confidence rating reviewers gave. More importantly, two of the reviewers consistently pointed out 'vague mathematics' and oversimplification in the mathematical arguments.

The authors' feedback did not successfully address the reviewer's concerns, both R3 and R4 indicated there were outstanding concerns.

I should note that despite giving low confidence scores and stating that some concepts from physics are beyond their field of expertise, reviewers gave high quality reviews with detailed comments and questions, and subsequently participated in the discussion revisiting their reviews. This suggests that the low confidence is not a symptom of insufficient reviewer effort, but perhaps a consequence of an inaccessible paper.
Decision: Reject