Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: Section 4.4: The original loss function is denoted by f(x) and right before lemma (2), related to the eigenvalues of hessian of f(x), it says either lambda_min>=c (with c>=0) or lambda_max<=-2*gamma-c (with c>=0 and gamma>=0), which respectively imply that at each point x, either all eigenvalues are nonnoegative, or all are nonpositive. If I understand this correctly, then the analysis does not consider points which have mixed-sign eigenvalues. Did I misunderstand something?
review_title: Allowing eigenvalues with mixed signs?

final_score: rating: null
review_content: -
review_title: -

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Overview: 

This paper introduces a biasing term for SGD that, in theoretical results and a toy example, yields solutions with an approximately equal or lower generalization error. This comes at a computational cost of estimating the gradient of the biasing term for each iteration through stochastic gradient Langevin dynamics, approximating an MCMC sample of the log partition function of a modified Gibbs distribution. The cost is equivalent to adding an inner for-loop to the standard SGD algorithm for each minibatch.

Pros:
- Reviews and distills many results and theorems from past 2 decades that suggest a promising way forward for increasing the generalizability of deep neural networks
- Generally very well written and well presented results, with interesting discussion of eigenvalues of Hessian as a way to characterize “flat” minima
- Promising mathematical arguments suggest that E-SGD has generalization error bounded below by SGD, motivating further research in the area

Cons / points suggested for a rebuttal:
(1) One claim of the paper given in the abstract is ”experiments on competitive baselines demonstrate that Entropy-SGD leads to improved generalization and has the potential to accelerate training.“ This does not appear to be supported by the current set of experiments. As the authors comment in the discussion section, “In our experiments, Entropy-SGD results in a comparable generalization error as SGD, but always has a lower cross-entropy loss.” It's not clear to me how to reconcile those two claims.

(2) Similarly, the claim of accelerated training is not convincingly supported in the present version of the paper. Vanilla SGD requires a single forward pass through all M minibatches during one epoch for a parameter update, but the new method, E-SGD requires, L*M forward passes during one epoch where L is the number of Langevin updates, which require a minibatch sample each. This could in fact mean that E-SGD has worse computational complexity to reach the same point. In a remark on p.9, the authors note that a single epoch is defined to be “the number of parameter updates required to run through the dataset once.” It’s not clear to me how this answers the objection to a factor of L additional computations required for the inner-loop SGLD iterations. SGLD appears to introduces a potentially costly tradeoff that must be carefully managed by a user of E-SGD.

(3) As the previous two points suggest, the paper could use some attention to the magnitude of the claims. For example, the introduction reads “Actively biasing towards wide valleys aids generalization, in fact, we can optimize solely the free energy term to obtain similar generalization error as SGD on the original loss function.“ According the the values reported on pp.9-10, only on MNIST is the generalization error, using only the free energy term (the log partition function of the modified Gibbs distribution), equivalent to using only the SGD loss function. This corresponds to setting rho to 0 in equation (6). On CIFAR-10, rho = 0.01 is used.

(4) Another contribution of this paper, the characterization of the optimization landscape in terms of the eigenvalues of the Hessian and low generalization error being associated with flat local extrema, is helpful and interesting. I found the plots clear and useful. As another reviewer has already pointed out, there are high-level similarities to “Flat Minima” by Hochreiter and Schmidhuber (1997). The authors have responded already by adding a paragraph that helpfully explores some differences with H&S 1997. However, the similarities should also be carefully identified and mentioned. H&S 1997 includes detailed theoretical analysis that could be helpful for future work in this area, and has independently discovered a similar approach to training generalizable networks.

(5) It's not clear how the assumption about the eigenvalues that were made in section 4.4 / Appendix B affect the application of this result to real-world problems. What magnitude of c>0 needs to be chosen? Does this correspond to a measurable characteristic of the dataset? It's a little mysterious in the current version of the paper.

review_title: Well-written paper explores promising direction for training generalizable deep neural network, but empirical results are too preliminary to support some arguments made

final_score: rating: 6: Marginally above acceptance threshold
review_content: __Note__: An earlier version of the review (almost identical to the present one) for an earlier version of the paper (available on arXiV) can be found here: http://www.shortscience.org/paper?bibtexKey=journals/corr/1611.01838#csaba
The only change concerns relation to previous work.

__Problem__: The problem considered is to derive an improved version of SGD for training neural networks (or minimize empirical loss) by modifying the loss optimized to that the solution found is more likely to end up in the vicinity of a minimum where the loss changes slowly ("flat minima" as in the paper of Hochreiter and Schmidhuber from 1997). 

__Motivation__: It is hypothetised that flat minima "generalize" better. 

__Algorithmic approach__: Let $f$ be the (empirical) loss to be minimized. Modify this to $$\overline{f}(x) = \rho f(x) - \log \int \exp(-\frac{\gamma}{2}\||z\||^2-f(x+z))\,dz$$ with some $\rho,\gamma>0$ tunable parameters. For $\||z\||^2\gg \frac{1}{\gamma}$, the term $\exp(-\frac{\gamma}{2}\||z\||^2)$ becomes very small, so effectively the second term is close to a constant times the integral of $f$ over a ball centered at $x$ and having a radius of $\propto \gamma^{-1/2}$. This is a smoothened version of $f$, hence one expects that by making this term more important then the first term, a procedure minimizing $\bar f$ will be more likely to end up at a flat minima of $f$. Since the gradient is somewhat complicated, an MCMC algorithm is proposed ("stochastic gradient Langevin dynamics" from Welling and Teh, 2011).

__Results__: There is a theoretical result that quantifies the increased smoothness of $\overline{f}$, which is connected to stability and ultimately to generalization through citing a result of Hardt et al. (2015). Empirical results show better validation error on two datasets: MNIST and CIFAR-10 (the respective networks are LeNet and All-CNN-C). The improvement is in terms of reaching the same validation error as with an "original SGD" but with fewer "epochs".

__Soundness, significance__: The proof of the __theoretical result__ relies on an arbitrary assumption that there exists some $c>0$ such that no eigenvalue of the hessian of $f$ lies in the set $[-2\gamma-c,c]$ (the reason for the assumption is because otherwise a uniform improvement cannot be shown). For $\rho=0$ the improvement of the smoothness (first and second order) is a factor of $1/(1+c/\gamma)$. The proof uses Laplace's method and is more a sketch than a rigorous proof (error terms are dropped; it would be good to make this clear in the statement of the result).

In the experiments the modified procedure did not consistently reach a smaller validation error. The authors did not present running times, hence it is unclear whether the procedure's increased computation cost is offset by the faster convergence. 

__Evaluation__: It is puzzling why a simpler smoothing, e.g., $$\overline{f}(x) = \int f(x+z) g(z) dz$$ (with $g$ being the density of a centered probability distribution) is not considered. The authors note that something "like this" may be infeasible in "deep neural networks" (the note is somewhat vague). However, under mild conditions, $\frac{\partial}{\partial x} \overline{f}(x) = \int \frac{\partial}{\partial x} f(x+z) g(z) dz$, hence, for $Z\sim g(\cdot)$, $\frac{\partial}{\partial x} f(x+Z)$ is an unbiased estimate of $\frac{\partial}{\partial x} \overline{f}(x)$, whose calculation is as cheap as that of vanilla SGD. Also, how much the smoothness of $f$ changes when using this approach is quite well understood.

__Related work__: It is also strange that the specific modification that appears in this paper was proposed by others (Baldassi et al.), whom this paper also cites, but without giving these authors credit for introducing local entropy as a smoothing technique. 
review_title: Smoothing the error surface to improve generalization; high level idea may be good, details are not entirely convincing.

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: This paper presents a principled approach to finding flat minima. The motivation to seek such minima is due to their better generalization ability. The idea is to add to the original loss function a new term that exploits both width and depth of the objective function. In fact, the regularization term can be interpreted as Gaussian convolution of the exponentiated loss. Therefore, the introduced regularization term is essentially Gaussian smoothed version of the exponentiated loss. The smoothing obviously tends to suppress sharp minima.

Overall, developing such regularization term based on thermodynamics concepts is very interesting. I have a couple of concerns that the authors may want to clarify in the rebuttal.

1. When reporting the generalization performance, the experiments report the number of epochs; showing the proposed algorithm reaches better generalization in fewer epochs than plain SGD. Is this the number of epochs it takes by line 7 of your algorithm, or it is the total number of epochs (line 3 and 7 all combined)? If the former, it is not a fair comparison. If you multiply the number of epochs of SGD (line 7) by the number iterations it takes to approximate Langevin dynamics, it seems you obtain little gain against plain SGD.

2. The proposed algorithm approximates the smoothed "exponentiated" loss (by smoothing I refer to convolution with the Gaussian). I am wondering how it compares against simpler idea of smoothing the original loss (dropping exponentiation)? Is the difference only in the motivation (e.g. thermodynamics interpretation) or it is deeper, e.g. the proposed scheme lends itself to more accurate approximation and/or achieves better generalization bound (in terms of the attained smoothness)? Smoothing the cost function without exponentiation allows simpler approximation (Monte Carlo integration instead of MCMC), e.g. see section 5.3 of https://arxiv.org/pdf/1601.04114

3. Section 4.4. Thank you for revising the statements related to the eigenvalues of the Hessian. However, even in the revised version, there seems to be some discrepancy. You "assume no eigenvalue of the Hessian lies in the set [−2γ −c, c] for some small c > 0". This essentially says the eigenvalues are far from zero. Such assumption seems to be in the opposite direction of the reality: the plots of eigenvalues (Figure 1) show most eigenvalues are indeed close to zero.

4. Theorem 3 from Hardt 2015: The way you quote it differs from the original paper. Are you referring to the Theorem 3.12 of Hardt's paper? If so, why the difference, including elimination of dependency on constant c in the exponent of T?  

review_title: Insightful Work

final_score: rating: 7: Good paper, accept
review_content: The paper introduces a new regularization term which encourages the optimizer 
to search for a flat local minimum of reasonably low loss instead of seeking a 
sharp region of a low loss. This is motivated by some empirical observations that
local minima of good generalization performance tend to have flat shape. 
To achieve this, a regularization term based on the free local energy is proposed
and the gradient of this term, which do not have tractable closed-form solution, 
is obtained by performing Monte Carlo estimation using SGLD sampler. In the 
experiments, the authors show some evidence of the flatness of good local 
minima, and also the performance of the proposed method in comparison to the
Adam optimizer. 

The paper is well and clearly written. I enjoyed reading the paper. The connection
to the concept of free energy in optimization framework seems interesting. The 
motivation of pursuing flatness is also well analyzed with a few experiments. I'm
wondering if the first term in eqn. (8) is correct. I guess it should be f(x') not f(x)?
Also, I'm wondering why the authors did not add the experiment results on RNN in
the evaluation of the performance because char-lstm for text generation was 
already used for the flatness experiments. I think adding more experiments on 
various models and applications of deep architectures (e.g., RNN, seq2seq, etc.) 
will make the author's claim more persuasive. I also found the mixed usage of the
terminology, e.g., free energy and free entropy, a bit confusing. 

review_title: official review

Meta-review: This paper presents both an analysis of neural net optimization landscapes, and an optimization algorithm that encourages movement in directions of high entropy. The motivation is based on intuitions from physics.
 
 Pros
  - the main idea is well-motivated, from a non-standard perspective.
  - There are lots of side-experiments supporting the claims for the motivation.
 Cons
  - The propose method is very complicated, and it sounds like good performance depended on adding and annealing yet another hyperparameter, referred to as 'scoping'.
  - The motivating intuition has been around for a long time in different forms. In particular, the proposed method is very closely related to stochastic variational inference, or MCMC methods. Appendix C makes it clear that the two methods aren't identical, but I wish the authors had simply run SVI with their proposed modification, instead of appearing to re-invent the idea of maximizing local volume from scratch. The intuition that good generalization comes from regions of high volume is also exactly what Bayes rule says.
 
 In summary, while there is improvement for the paper, the idea is well-motivated and the experimental results are sound.
Decision: Accept (Poster)