Venue: ICLR 2018 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors discuss the regularized objective function minimized by standard SGD in the context of neural nets, and provide a variational inference perspective using the Fokker-Planck equation. They note that the objective can be very different from the desired loss function if the SGD noise matrix is low rank, as evidenced in their experiments.

Overall the paper is written quite well, and the authors do a good job of explaining their thesis. However I was unable to identify any real novelty in the theory: the Fokker-Planck equation has been widely used in analysis of stochastic noise in MCMC samplers in recent years, and this paper mostly rephrases those results. Also the fact that SGD theory only works for isotropic noise is well known, and that there is divergence from the true loss function in case of low rank noise is obvious. Thus I found most of section 3 to be a reformulation of known results, including Theorem 5 and its proof.

Same goes for section 5; the symmetric- anti symmetric split is a common technique used in the stochastic MCMC literature over the last few years, and I did not find any new insight into those manipulations of the Fokker-Planck equation from this paper.

Thus I think that although this paper is written well, the theory is mostly recycled and the empirical results in Section 4 are known; thus it is below acceptance threshold due to lack of novelty.
review_title: Well written, but lacking in novelty.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The paper takes a closer look at the analysis of SGD as variational inference, first proposed by Duvenaud et al. 2016
and Mandt et al. 2016. In particular, the authors point out that in general, SGD behaves quite differently from Langevin diffusion due to the multivariate nature of the Gaussian noise. As the authors show based on the Fokker-Planck equation of the underlying stochastic process, there exists a conservative current (a gradient of an underlying potential) and a non-conservative current (which might induce stationary persistent currents at long times). The non-conservative part leads to the fact that the dynamics of SGD	may show oscillations, and these oscillations may even prevent the algorithm from converging to the 'right' local optima. The theoretical analysis is carried-out very nicely, and the theory is supported by experiments on two-dimensional toy examples, and Fourier-spectra of the iterates of SGD.

This is a nice paper which I would like to see accepted. In particular I appreciate that the authors stress the importance
of 'non-equilibrium physics' for understanding the SGD process. Also, the presentation is quite clear and the paper well written.

There are a few minor points which I would like to ask the authors to address:

1. Why cite Kingma and Welling as a source for variational inference in	section 3.1? VI is a much older	field, and Kingma and Welling proposed a very special form of VI, namely amortized VI with inference networks. A better citation would be Jordan et	al 1999.

2. I'm not sure how much to trust the Fourier-spectra. In particular, perhaps the deviations from Brownian motion could also be due to the discrete	nature of SGD (i.e. that the continuous-time formalism is only an approximation of a discrete process). Could you elaborate on this?

3. Could you give the reader more details on how the uncertainty estimates on the Fourier transformations were obtained?

Thanks.
review_title: A variational analysis of SGD as a non-equilibrium process.

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper develop theory to study the impact of stochastic gradient noise for SGD, especially for deep neural network models. It is shown that when the gradient noise is isotropic normal, SGD converges to a distribution tilted by the original objective function. However, when the gradient noise is non isotropic normal, which is shown common in many models especially in deep neural network models, the behavior of SGD is intriguing, which will not converge to the tilted distribution by the original objective function, sometimes more interestingly, will converge to limit cycles around some critical points of the original objective function. The paper also provides some hints on why using SGD can get good generalization ability than gradient descend.

I think the finding of this paper is interesting, and the technical details are correct. I still have the following comments.

First, Assumption 4 seems a bit too abstract. It is not easy to see what the assumption means. It would be better if an example is given, which is verified to satisfy the assumption.

Another comment is related to the overall content of this paper. Thought the paper point out that SGD will have the out-of-equilibrium behavior when the gradient noise is non isotropic normal, it remains to show how far away this stationary distribution is from the original distribution defined by the objective function.
review_title: An interesting paper on analyzing the impact of gradient noise for SGD

Meta-review: Dear authors,

Based on the comments and your rebuttal, I am glad to accept your paper at ICLR.
Decision: Accept (Poster)