Title: Entropy-SGD: Biasing Gradient Descent Into Wide Valleys
Authors: Pratik Chaudhari
Anna Choromanska
Stefano Soatto
Yann LeCun
Carlo Baldassi
Christian Borgs
Jennifer Chayes
Levent Sagun
Riccardo Zecchina
Venue: ICLR 2017
Keywords: Deep learning
Optimization

Abstract: This paper proposes a new optimization algorithm called Entropy-SGD for training deep neural networks that is motivated by the local geometry of the energy landscape. Local extrema with low generalization error have a large proportion of almost-zero eigenvalues in the Hessian with very few positive or negative eigenvalues. We leverage upon this observation to construct a local-entropy-based objective function that favors well-generalizable solutions lying in large flat regions of the energy landscape, while avoiding poorly-generalizable solutions located in the sharp valleys. Conceptually, our algorithm resembles two nested loops of SGD where we use Langevin dynamics in the inner loop to compute the gradient of the local entropy before each update of the weights. We show that the new objective has a smoother energy landscape and show improved generalization over SGD using uniform stability, under certain assumptions. Our experiments on convolutional and recurrent neural networks demonstrate that Entropy-SGD compares favorably to state-of-the-art techniques in terms of generalization error and training time.

TL;DR: This paper focuses on developing new optimization tools for deep learning that are tailored to exploit the local geometric properties of the objective function.

Review discussion:
We have updated the "Related work" section of the paper with a discussion of [HS97] and [KS16].
[HS97] Hochreiter, Sepp, and Jürgen Schmidhuber. "Flat minima." Neural Computation 9.1 (1997): 1-42.
[KS16] Keskar, Nitish Shirish, et al. "On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima." arXiv:1609.04836 (2016).
Section 4.4: The original loss function is denoted by f(x) and right before lemma (2), related to the eigenvalues of hessian of f(x), it says either lambda_min>=c (with c>=0) or lambda_max<=-2*gamma-c (with c>=0 and gamma>=0), which respectively imply that at each point x, either all eigenvalues are nonnoegative, or all are nonpositive. If I understand this correctly, then the analysis does not consider points which have mixed-sign eigenvalues. Did I misunderstand something?
The proof works for both positive and negative eigenvalues. The assumption (in Sec. 4.4) should read "assume |\lambda (\nabla^2 f(x))| \geq 2 \gamma + c for some small c > 0". This can be made slightly stricter to read "assume that \nabla^2 f(x) does not have any eigenvalue between [-2\gamma-c,c]". We have rectified this in the paper.
This paper presents a principled approach to finding flat minima. The motivation to seek such minima is due to their better generalization ability. The idea is to add to the original loss function a new term that exploits both width and depth of the objective function.
In fact, the regularization term can be interpreted as Gaussian convolution of the exponentiated loss.
Therefore, the introduced regularization term is essentially Gaussian smoothed version of the exponentiated loss. The smoothing obviously tends to suppress sharp minima.
Overall, developing such regularization term based on thermodynamics concepts is very interesting. I have a couple of concerns that the authors may want to clarify in the rebuttal.
1.
When reporting the generalization performance, the experiments report the number of epochs; showing the proposed algorithm reaches better generalization in fewer epochs than plain SGD. Is this the number of epochs it takes by line 7 of your algorithm, or it is the total number of epochs (line 3 and 7 all combined)? If the former, it is not a fair comparison. If you multiply the number of epochs of SGD (line 7) by the number iterations it takes to approximate Langevin dynamics, it seems you obtain little gain against plain SGD.
2. The proposed algorithm approximates the smoothed "exponentiated" loss (by smoothing I refer to convolution with the Gaussian). I am wondering how it compares against simpler idea of smoothing the original loss (dropping exponentiation)? Is the difference only in the motivation (e.g. thermodynamics interpretation) or it is deeper, e.g. the proposed scheme lends itself to more accurate approximation and/or achieves better generalization bound (in terms of the attained smoothness)? Smoothing the cost function without exponentiation allows simpler approximation (Monte Carlo integration instead of MCMC), e.g. see section 5.3 of https://arxiv.org/pdf/1601.04114
3. Section 4.4. Thank you for revising the statements related to the eigenvalues of the Hessian.
However, even in the revised version, there seems to be some discrepancy. You "assume no eigenvalue of the Hessian lies in the set [−2γ −c, c] for some small c > 0". This essentially says the eigenvalues are far from zero. Such assumption seems to be in the opposite direction of the reality: the plots of eigenvalues (Figure 1) show most eigenvalues are indeed close to zero.
4. Theorem 3 from Hardt 2015: The way you quote it differs from the original paper. Are you referring to the Theorem 3.12 of Hardt's paper? If so, why the difference, including elimination of dependency on constant c in the exponent of T?
>> Thm. 3 does not have the constant c in Hardt et. al. '15
We set the constant c of Hardt et. al., to 1 in our statement to avoid confusion with our constant, with is also called c (our statement has the learning rate \eta_t \leq 1/t instead of their original \eta_t \leq c/t). This does not change the result qualitatively.
Please see the "updates to the paper" comment above for the response to the questions regarding (i) the eigenvalue assumption, (ii) thorough experimental results and, (iii) smoothing the original loss vs. local entropy.
The paper introduces a new regularization term which encourages the optimizer
to search for a flat local minimum of reasonably low loss instead of seeking a
sharp region of a low loss. This is motivated by some empirical observations that
local minima of good generalization performance tend to have flat shape.
To achieve this, a regularization term based on the free local energy is proposed
and the gradient of this term, which do not have tractable closed-form solution,
is obtained by performing Monte Carlo estimation using SGLD sampler. In the
experiments, the authors show some evidence of the flatness of good local
minima, and also the performance of the proposed method in comparison to the
Adam optimizer.
The paper is well and clearly written. I enjoyed reading the paper. The connection
to the concept of free energy in optimization framework seems interesting. The
motivation of pursuing flatness is also well analyzed with a few experiments. I'm
wondering if the first term in eqn. (8) is correct. I guess it should be f(x') not f(x)?
Also, I'm wondering why the authors did not add the experiment results on RNN in
the evaluation of the performance because char-lstm for text generation was
already used for the flatness experiments. I think adding more experiments on
various models and applications of deep architectures (e.g., RNN, seq2seq, etc.)
will make the author's claim more persuasive. I also found the mixed usage of the
terminology, e.g., free energy and free entropy, a bit confusing.
>> Eqn. 8 should have f(x')
Thanks, it is fixed now.
>> experiments on RNNs
Thanks for suggesting this. We have included results for word-level and character-level text prediction on PTB and char-LSTM with War and Peace. We not only obtain a better test perplexity than a competitive baseline with SGD, but also train in about half as much wall-clock time.
Please see the "updates to the paper" comment above for more details.
Acknowledging the author responses to improve the paper w.r.t the reviewer comments, I changed the rating to 7.
Overview:
This paper introduces a biasing term for SGD that, in theoretical results and a toy example, yields solutions with an approximately equal or lower generalization error. This comes at a computational cost of estimating the gradient of the biasing term for each iteration through stochastic gradient Langevin dynamics, approximating an MCMC sample of the log partition function of a modified Gibbs distribution. The cost is equivalent to adding an inner for-loop to the standard SGD algorithm for each minibatch.
Pros:
- Reviews and distills many results and theorems from past 2 decades that suggest a promising way forward for increasing the generalizability of deep neural networks
- Generally very well written and well presented results, with interesting discussion of eigenvalues of Hessian as a way to characterize “flat” minima
- Promising mathematical arguments suggest that E-SGD has generalization error bounded below by SGD, motivating further research in the area
Cons / points suggested for a rebuttal:
(1) One claim of the paper given in the abstract is ”experiments on competitive baselines demonstrate that Entropy-SGD leads to improved generalization and has the potential to accelerate training.“ This does not appear to be supported by the current set of experiments. As the authors comment in the discussion section, “In our experiments, Entropy-SGD results in a comparable generalization error as SGD, but always has a lower cross-entropy loss.” It's not clear to me how to reconcile those two claims.
(2) Similarly, the claim of accelerated training is not convincingly supported in the present version of the paper. Vanilla SGD requires a single forward pass through all M minibatches during one epoch for a parameter update, but the new method, E-SGD requires, L*M forward passes during one epoch where L is the number of Langevin updates, which require a minibatch sample each. This could in fact mean that E-SGD has worse computational complexity to reach the same point. In a remark on p.9, the authors note that a single epoch is defined to be “the number of parameter updates required to run through the dataset once.” It’s not clear to me how this answers the objection to a factor of L additional computations required for the inner-loop SGLD iterations. SGLD appears to introduces a potentially costly tradeoff that must be carefully managed by a user of E-SGD.
(3) As the previous two points suggest, the paper could use some attention to the magnitude of the claims.
For example, the introduction reads “Actively biasing towards wide valleys aids generalization, in fact, we can optimize solely the free energy term to obtain similar generalization error as SGD on the original loss function.“ According the the values reported on pp.9-10, only on MNIST is the generalization error, using only the free energy term (the log partition function of the modified Gibbs distribution), equivalent to using only the SGD loss function. This corresponds to setting rho to 0 in equation (6). On CIFAR-10, rho = 0.01 is used.
(4) Another contribution of this paper, the characterization of the optimization landscape in terms of the eigenvalues of the Hessian and low generalization error being associated with flat local extrema, is helpful and interesting. I found the plots clear and useful. As another reviewer has already pointed out, there are high-level similarities to “Flat Minima” by Hochreiter and Schmidhuber (1997). The authors have responded already by adding a paragraph that helpfully explores some differences with H&S 1997.
However, the similarities should also be carefully identified and mentioned. H&S 1997 includes detailed theoretical analysis that could be helpful for future work in this area, and has independently discovered a similar approach to training generalizable networks.
(5) It's not clear how the assumption about the eigenvalues that were made in section 4.4 / Appendix B affect the application of this result to real-world problems. What magnitude of c>0 needs to be chosen? Does this correspond to a measurable characteristic of the dataset? It's a little mysterious in the current version of the paper.
Please see the "updates to the paper" comment above for the response to the questions regarding (i) the eigenvalue assumption and, (ii) thorough experimental results.
>> how to reconcile "Entropy-SGD obtains better generalization but results in lower cross-entropy loss"
The confusion is probably caused by our omission, it should read "... lower cross-entropy loss on the training set" (fixed now). This experimental observation suggests that there exist wide valleys that are deeper in the energy landscape that also generalize well; Entropy-SGD manages to find them while SGD gets stuck at a higher loss.
>> \rho = 0.01 on CIFAR-10 but \rho = 0 on MNIST
We have modified the algorithm to only train with the local entropy objective, i.e., \rho = 0 always (cf. Eqn. 6). Using scoping we can obtain better results on all our networks both in terms of generalization error and wall-clock time.
>> discuss similarities to Hochreiter and Schmidhuber '97 [HS97]
We have expanded our discussion of [HS97] in Sec. 2 to include this discussion: While the motivations are exactly the same, similarities with their exact formulation are only conceptual in nature, e.g., in a flat minimum, the local entropy is a direct measure of the width of the valley which is similar to their usage of Hessian. The Gibbs variant to averaging in weight space in their analysis (Eqn. 33, pg. 22 of [HS97]) is similar to the averaging in Eqn. 7 of our paper.
We agree with the reviewer that the elaborate analysis of generalization of [HS97] using the Gibbs formalism is a promising direction. In our case, we benefit from similar elaborate and technical results introduced for uniform stability [BE02].
>> experiments are on a toy example
Please note that our results on CIFAR-10 are without any data augmentation, the best result for this is 6.55% error using ELU units by [CUH15]. To our knowledge, our baseline on CIFAR-10 (7.71 \pm 0.19% error with SGD), is the best reported result for the popular All-CNN-C network [S14], which is a medium-sized model with about 1.6 million weights. The largest model we have experimented with is an LSTM on the PTB dataset with 66 million weights and we achieve better test perplexity than the original authors [ZSV14] in half as much wall-clock time.
[HS97] Hochreiter, S. and Schmidhuber, J. (1997), Flat Minima.
[BE02] Olivier Bousquet and Andre Elisseeff. Stability and generalization. JMLR, 2002.
[CUH15] Clevert, Djork-Arné, Thomas Unterthiner, and Sepp Hochreiter. "Fast and accurate deep network learning by exponential linear units (elus)." arXiv:1511.07289 (2015).
[S14] Springenberg, Jost Tobias, et al. "Striving for simplicity: The all convolutional net." arXiv:1412.6806 (2014).
[ZSV14] Zaremba, Wojciech, Ilya Sutskever, and Oriol Vinyals. "Recurrent neural network regularization." arXiv:1409.2329 (2014).
We thank the reviewers and the area chair for their insightful feedback. We have incorporated all comments into our current draft. We first discuss the updates to the paper and address common questions raised by the reviewers. We have also posted individual comments to the reviewers to address specific questions.
Updates
a) We have updated the experimental section of the paper with new experiments on MNIST, CIFAR-10 and two datasets on RNNs (PTB and char-LSTM).
b) We have modified the algorithm to introduce a technique called "scoping". This increases the scope parameter \gamma as training progresses instead of fixing it and has the effect of exploring the parameter space in the beginning of training (Sec. 4.3).
As a result of this, we can now train all our networks with only the local entropy loss instead of treating it as a regularizer (Eqn. 6).
c) For a fair comparison of the training time, we now plot the error curves in Figs. 4, 5 and 6 against the "effective number of epochs", i.e., the number of epochs of Entropy-SGD is multiplied by the number of Langevin iterations L (we set L=1 for SGD/Adam).
Thus the x-axis is a direct measure of the wall-clock time agnostic to the underlying implementation and is proportional to the number of back-props as suggested.
We obtain significant speed-ups with respect to our earlier results due to scoping and the wall-clock training time for all our networks with Entropy-SGD is now comparable to SGD/Adam.
In fact, Entropy-SGD is almost twice as fast as SGD on our experiments on RNNs and also obtains a better generalization error (cf. Fig. 6). The acceleration for CNNs on MNIST and CIFAR-10 is about 20%.
Table 1 (page 11) summarizes the experimental section of the paper.
d) Improved exposition of the algorithm in Sec. 4.2 that includes intuition for hyper-parameter tuning. We have expanded the discussion of experiments in Sec. 5.3, 5.4 to provide more details and insights that relate to the energy landscape of deep networks.
e) Appendix C discusses the possible connections to variational inference (this is the same material as the discussion with the AC below). Sec. C.1 presents an experimental comparison of local entropy vs. SGLD. We note here that our results using Entropy-SGD for both CNNs and RNNs are much better than vanilla SGLD in significantly smaller (~3-5x) wall-clock times.
Response to the reviewers:
>> smoothing of the original loss vs. local entropy
We discuss this in detail in the related work in Sec. 2 and Appendix C.
While smoothing the original loss function using convolutions or averaging the gradient over perturbations of weights reduces the ruggedness of the energy landscape, it does not help with sharp, narrow valleys. Local entropy introduces a measure that focuses on wide local minima in the energy landscape (cf. Fig. 2 which has a "global" minimum at a wide valley); this is the primary reason for its efficacy. Smoothing the loss function can also, for instance, generate an artificial local minimum between two close by sharp valleys, which is detrimental to generalization.
>> unrealistic eigenvalue assumption in Sec. 4.3
We have clarified this point in Remark 4. Our analysis employs an assumption that the Hessian \nabla^2 f(x) does not have eigenvalues in the set [-2\gamma-c, c] for some c > 0. This is admittedly unrealistic, for instance, the eigenspectrum of the Hessian in Fig. 1 has a large fraction of its eigenvalues almost zero. Let us note though that Fig. 1 is plotted at a local minimum, from our experiments, the eigenspectrum is less sparse in the beginning of training.
We would like to remark that the bound on uniform stability in Thm. 3 by Hardt et al. assumes global conditions on the smoothness of the loss function; one imagines that Eqn. 9 remains qualitatively the same (in particular, with respect to the number of training iterations) even if this assumption is violated to an extent before convergence happens. Obtaining a rigorous generalization bound without this assumption requires a dynamical analysis of SGD and seems out of reach currently.
The draft has undergone a major revision and my comments are handled very well in the new version. I think now the draft is in great shape, both from formulation viewpoint (connection to principles in statistical physics) and practical impact (Entropy-SGD is shown to beat SGD and SGLD in terms of wall-clock time).
As a result of these improvements, I am raising my rating for this paper by one level.
The major revisions to the paper have answered each of the concerns I raise in the first review. The authors have also improved the technique by introducing an enhancement to the algorithm through the reverse-annealing of the scoping parameter.
Hence, the evidence presented for the practical value of the technique is much improved, so that the paper has a clearer pragmatic as well as theoretical value. The ties of this work to previous studies is also much improved. These changes motivate the update in my review status to a clear accept.
Congratulations to the authors for the high points on the updated PDF. I hope I am not very late in asking two short questions. Will you be opensourcing the code for the method & how does the method relate to the discussions in some other recent ICLR papers
[1] Understanding deep learning requires rethinking generalization
[2] Snapshot Ensembles: Train 1, Get M for Free
[3] On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
They are only tangentially related I know but I was curious how the motivation of the method fits into the generalization arguments of [1,3] and whether the ensembling trick of [2] can be used in conjunction with Entropy-SGD to reap benefits of both.
We plan to open-source the code and examples.
We could speculate on how our method relates to the experiments of [1] but we would rather leave this to further investigation.
[2] is quite different from the replica theoretic motivations of Entropy-SGD, the latter creates one good model.
While it is possible to create ensembles of networks with more local scoping, the benefits might be offset by significantly longer test times.
The metric in [3] could be considered as a proxy for local entropy. Entropy-SGD is a specialized algorithm to lead to flat minima, obtaining the same effect using parameters such as batch-size, dropout, batch-normalization, data augmentation etc. is quite hard.

Meta-review:
role: AC
data: comment: I agree with the motivations of the method, but this paper seems to be mostly re-inventing stochastic variational inference.  The SVI ELBO estimate also encourages the approximate posterior mean to head in directions that have high likelihood times volume.

More broadly, the marginal likelihood is also trying to achieve the same tradeoffs, which is the motivation for MCMC methods such as SGLD in the first place.  So It seems bizarre that you would compare Adam against a complicated method with SGLD in the inner loop, without comparing against the much simpler SGLD as well.

Finally, it's frustrating that you discuss free energy and entropy at length in words without giving their precise definitions.  If you did so, I suspect it would become clear that you're proposing maximizing the marginal likelihood, which is definitely a great idea but already well-known.  How does the entropy you optimize relate to the marginal likelihood and the variational ELBO?
role: Author
data: value: comment: We are happy to provide clarifications to the questions posed. We enclose them below.

"mostly re-inventing stochastic variational inference"
SVI computes the posterior distribution of hidden variables given data by maximizing the ELBO. Please note that we do not have hidden variables in our formulation; "x" are the variables of our objective f(x), not data. We are interested in maximizing the free energy F(x,\gamma) as defined in Eqn. 4. This is the *free energy of the energy landscape* (defined via a Gibbs distribution). This is thus unrelated to marginal likelihood or variational ELBO where one can impose entropic constraints on the posterior distribution of the *hidden variables* via a prior on them.

"I suspect it'd become clear that you are maximizing the marginal likelihood"
Please note that we are not maximizing the marginal likelihood, as discussed above.

"how is this better than SGLD", "bizarre that you compare against Adam instead of SGLD"
SGLD is an MCMC algorithm that draws samples from a given distribution. If the step-size goes to zero slowly enough, akin to all MCMC algorithms, it converges to the maximum of the likelihood in exponentially long time-scales (see algorithms like SGHMC [CFG14], Santa [CCG15] etc. that train Bayesian neural networks using MCMC algorithms). Note that SGLD does not optimize the marginal likelihood because there is no notion of hidden variables in vanilla SGLD. We provide a brief review of SGLD in Appendix A of our paper.

We do not know of any results in the literature that train large deep networks such as the one used for CIFAR to get competitive error rates using SGLD. We would like to emphasize that Entropy-SGD simply uses MCMC sampling to estimate Eqn. 7 but it is unrelated to SGLD otherwise.

On the other hand, Adam is an algorithm for computing the maximum of a the likelihood of data given parameters or (equivalently, minimize the loss function). Entropy-SGD is an algorithm designed for minimizing the loss function f(x), in particular, it is not an MCMC algorithm that draws samples from the likelihood. It however does not explicitly do so and instead maximizes the local entropy. We therefore compare Entropy-SGD with state-of-the-art algorithms for training deep networks like Adam and SGD.

"frustrating that you discuss free energy and entropy without precise definitions"
Local entropy (local free energy) is formally defined in Def. 1 (Eqn. 4) but it is already introduced on pg. 2 in the Introduction. The discussion towards the end of Sec. 3 (pg. 5, first line) defines the classical entropy of a Gibbs distribution. The beginning of Sec. 3 based on Fig. 2 is intended to explain things to the reader at an intuitive level before proceeding to the formal definitions which are nevertheless present.

Free energy and classical entropy are related by the informal description "free energy = internal energy - temperature x entropy". Formally, the relation is
    F(\beta) = U(\beta) - \frac{1}{\beta} S(\beta)
where the log-partition function (free energy) is defined as F(\beta) = -\beta^{-1} \log Z(\beta), the internal energy is U(\beta) = \partial_\beta (\beta F(\beta)) and S(\beta) = \beta^2 \partial_\beta (F(\beta)) is the classical entropy.

Note that the above equation is about the entire Gibbs distribution, in our work we define a "local free energy" F(x, \beta) via the modified Gibbs distribution in Eqn. 3. We can add this discussion to the paper.

[CFG14] Chen, Tianqi, Emily B. Fox, and Carlos Guestrin. "Stochastic Gradient Hamiltonian Monte Carlo." ICML. 2014.
[CCF15] Chen, Changyou, et al. "Bridging the gap between stochastic gradient mcmc and stochastic optimization." arXiv:1512.07962 (2015).
role: AC
data: value: comment: Thanks for the detailed reply.  However, I feel like we're talking past each other on a few points:

1) You explain that SVI wouldn't be appropriate here because there aren't latent variables.  But I'm suggesting that you're effectively approximately integrating out, or sampling from, the posterior over neural network parameters.  That is, you only have one set of latent variables, which are the parameters of your network.

2) You assert it wouldn't be appropriate to compare against SGLD because it's a sampler and you're introducing an optimizer.  However, the main motivation for your method seems to be better generalization error, not a better optimization of the original objective.  Better generalization is the reason why people advocate using approximate inference such as SGLD to choose weights instead of maximum likelihood.

I realize you claim to not be doing approximate inference, but since SGLD is a strictly simpler method than the one you propose which also has the same motivations, you should compare against it.

3) Thanks for spelling out the free energy in terms of F, U and \beta - however I would find it much more illuminating if you were to write these quantities in terms of integrals of log-probabilities.

Again, the entire motivation for this method seems to be the same as for approximate inference methods, to the point where I suspect you might be re-inventing existing methods.  So a clearer conceptual and experimental comparison would be helpful.
role: Author
data: value: comment: To address the objections 1) and 3), let \Xi denote the dataset, z denote the weights and x be the parameters of a variational distribution q_x(z). The ELBO can then be written as
(i) \log p(\Xi)
        \geq E_{z \sim q_x(z)} [\log p(\Xi | z)] - KL(q_x(z) || p(z))
and maximized with respect to x. The distribution p(z) is a fixed (parameter-free) prior, which one has to postulate.

On the other hand, Eqn. 4 in the paper can be used to write the log of local entropy as:
(ii) \log F(x,\gamma)
        = \log \int_{z \in Z} e^{-f(z; \Xi) - \gamma/2 |x-z|^2} dz
        \geq \int_{z \in Z} [-f(z; \Xi) - \gamma/2 |x-z|^2] dz;
where f(z) = -\log p(\Xi | z). We are unaware of a general way to choose a prior p(z) and a variational family q_x(z) that that makes (i) resemble (ii), and therefore interpret our method as “integrating out […] the posterior over neural network parameters.” The only way we can do so is to pick a specific “prior" that depends on the parameter x (hence, not really a prior). For instance, one could choose a uniform variational family (say, q_x(z) \propto constant for |x-z| \leq C and zero otherwise) and a Gaussian “prior" with mean x (\log p(z) = -\gamma/2 |x-z|^2) to make (ii) resemble ELBO. In this case p(z) would not be fixed, but it would “move” along with the current iterate x.

This "moving prior" is a crucial feature of our proposed algorithm. The gradient of local entropy (Eqn. 7 in the paper) clarifies this further:
    dF  = -\gamma (x - E_{z \sim r(z; x)} [z]);
where the distribution r(z; x) is
    r(z; x) \propto p(\Xi | z) \exp(-\gamma/2 |x-z|^2);
i.e. it contains a data likelihood term with a prior that "moves" along with the current iterate x.

Concerning the relation to SGLD, consider Belief Propagation (BP). Our proposed algorithm relates to the "focusing-Belief Propagation" variant (fBP), rather than the standard one [BBC16]. The difference between BP and fBP is analogous to that between SGLD and Entropy-SGD: the latter operates on a transformation of the energy landscape of the former, exploiting local entropic effects. This difference is crucial and indeed related to the "moving prior" of the previous discussion; plain SGLD (or BP) can only trade energy for entropy via the temperature parameter which does not allow for direct use of any geometric information of the landscape and does not help with narrow minima.

In view of your comment 2), we also implemented SGLD for LeNet on MNIST and All-CNN-BN on CIFAR and will add the following results to our paper: After a hyper-parameter search, the best we obtained were a test error of LeNet on 0.63 \pm 0.1% on MNIST after 300 epochs and 9.89 \pm 0.11% on All-CNN-BN after 500 epochs. Disregarding the slow convergence of SGLD, its generalization error is slightly worse than the results in our paper, viz. 0.48% with Entropy-SGD on LeNet (0.51% with SGD) and 8.65% with Entropy-SGD on All-CNN-BN (8.3% with SGD). For comparison, the authors in [CCF15] report an error of 0.71% with SGLD on MNIST with a slightly larger network (0.47% with Santa), there are no results in the literature where MCMC methods perform comparably to SGD on larger networks.

[BBC16] Baldassi, Carlo, et al. "Unreasonable effectiveness of learning neural networks: From accessible states and robust ensembles to basic algorithmic schemes." PNAS (2016).
[CCF15] Chen, Changyou, et al. "Bridging the gap between stochastic gradient MCMC and stochastic optimization." arXiv:1512.07962 (2015).
role: AC
data: value: comment: Ok, great, you're not doing exactly the same thing as stochastic variational inference.  But the point is that you're proposing a method whose motivations are almost identical to those of Bayesian methods, and which looks very similar to SVI in practice, and which using Bayesian methods in the inner loop.

So it's kind of obnoxious to not make a clear and explicit comparison in the paper between your new method and the existing, standard methods which are extremely similar.  I think the physics notation obfuscates what's really going on.  I would suggest adding the above discussion to the paper.

Would it be fair to say that your algorithm is SVI but with a moving prior?  Because that would be a nice connection to make, and it would inform the design of inference algorithms more generally.

Regarding experiments, I would also strongly recommend that you use time or gradient evaluations on the x-axis.  As it stands it's not clear whether or not your new algorithm gets better performance in the same amount of time.  Also, I still think it's strange that you don't compare against SVI.
role: Author
data: value: comment: We have updated the paper to include a detailed discussion about SVI and added experiments comparing with SGLD in Appendix C (also see the discussion in the last paragraph of Sec. 2). As already remarked in our previous comment, characterizing local entropy as "SVI with a moving prior" would be inaccurate as their relation is just conceptual and not rigorous.

>> change x-axis for error curves
We have updated the experiments in the paper and now plot all error curves against the "effective number of epochs", i.e., the number of epochs for Entropy-SGD is multiplied by L (we set L=1 for SGD/Adam). Note that this is proportional to the number of back-props as suggested.

Please see the "updates to the paper" comment above for more details.
role: PC
data: comment: This paper presents both an analysis of neural net optimization landscapes, and an optimization algorithm that encourages movement in directions of high entropy. The motivation is based on intuitions from physics.
 
 Pros
  - the main idea is well-motivated, from a non-standard perspective.
  - There are lots of side-experiments supporting the claims for the motivation.
 Cons
  - The propose method is very complicated, and it sounds like good performance depended on adding and annealing yet another hyperparameter, referred to as 'scoping'.
  - The motivating intuition has been around for a long time in different forms. In particular, the proposed method is very closely related to stochastic variational inference, or MCMC methods. Appendix C makes it clear that the two methods aren't identical, but I wish the authors had simply run SVI with their proposed modification, instead of appearing to re-invent the idea of maximizing local volume from scratch. The intuition that good generalization comes from regions of high volume is also exactly what Bayes rule says.
 
 In summary, while there is improvement for the paper, the idea is well-motivated and the experimental results are sound.

Decision: Accept (Poster)