Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Update after response: I appreciate the authors making their contributions clearer, and adding details about the training loss and error. I have increased my score accordingly.

Original Review:

This paper presents an empirical evaluation of whether flatness correlates with generalization using a few different definitions of flatness - local energy and local entropy. The authors study two training procedures - entropy sgd and replicated sgd, and show that deep networks trained using these procedures are able to locate flatter solutions that also generalize better.

While the paper presents some interesting empirical confirmation of the correlation between local entropy/energy and generalization, the algorithms presented in this paper have also been defined in previous work, and this phenomenon has been repeatedly observed with different definitions of flatness [1,2]. The authors also do not present any reasons to expect that generalization is related to flatness that are grounded in theory. The contributions of this paper thus seem to be confirmation of previously observed phenomena [3,4].

Moreover, in the experiments it is not clear whether the solutions that are reached by the training procedures actually correspond to local or global minima of the cross-entropy loss function (There is also the issue that minimizers of the cross-entropy loss function occur at infinity). The authors do not report the training loss at the solutions that they choose to plot, and the training error also does not seem to be zero (plots corresponding to Figure 3). I am not sure whether it makes sense to call these solutions minima, and compare their flatness.


[1] Neyshabur, B., Bhojanapalli, S., McAllester, D., & Srebro, N. (2017). Exploring generalization in deep learning. In Advances in neural information processing systems (pp. 5947-5956). 

[2] Keskar, N. S., Mudigere, D., Nocedal, J., Smelyanskiy, M., & Tang, P. T. P. (2016). On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836.

[3] Baldassi, C., Pittorino, F., & Zecchina, R. (2020). Shaping the learning landscape in neural networks around wide flat minima. Proceedings of the National Academy of Sciences, 117(1), 161-170.

[4] Chaudhari, P., Choromanska, A., Soatto, S., LeCun, Y., Baldassi, C., Borgs, C., ... & Zecchina, R. (2019). Entropy-sgd: Biasing gradient descent into wide valleys. Journal of Statistical Mechanics: Theory and Experiment, 2019(12), 124018.
review_title: Official Blind Review #3

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper studies local entropy measures for characterizing flat regions in the energy landscape of deep networks. The paper discusses, at length, two previously proposed algorithms named Entropy-SGD and Replicated-SGD and demonstrates, using (i) controlled experiments where Belief Propagation (BP) can be used to estimate the local entropy integral precisely, and (ii) empirical results on deep networks that flatter minima generalize better.

This paper presents a systematic analysis of the hypothesis that flat minima generalize better. This has been the subject of much debate in the recent literature because flatness is often characterized using spectral norm of the Hessian. The paper revisits the line of work that initiated this debate and shows that flatness, as measured by local entropy instead, indeed correlates with good generalization.

The paper lacks in terms of novelty in the sense that it does not propose a “new” algorithm, but the reviewer believes that not all publications need to. A paper like this with carefully constructed constructed and a clear conclusion is also a valuable addition to this debate. I am however concerned about the incremental value of this manuscript in view of the long line of work that it builds upon. I am recommending a weak accept but I am willing to increase my score if the authors make a convincing case against this concern.

Comments.

1. The paper, as noted above, heavily builds upon existing work, in particular series of works of Baldassi et al. and Chaudhari et al. The papers introduce local entropy, focussing, replicated SGD, BP for calculating the local entropy etc. Entropy-SGD/Parle were also shown to work well for state-of-the-art deep networks then. The authors should state clearly what the concrete contributions of this manuscript are.
2. I like the section on measuring flatness for shallow architectures using BP. It would be good to include the BP calculations in the Appendix.
3. The idea of making the cross-entropy loss invariant to the scale of the weights by keeping them normalized can also be used with deep networks using weight normalization https://arxiv.org/abs/1602.07868.
4. Can you characterize the shape of the wide region? I believe this will lend more insight in why wide minima generalize better for deep networks.
review_title: good paper but unclear what the contributions are

final_score: rating: 7: Good paper, accept
review_content: The paper studies (1) the relationship between the flatness of minima and their generalization properties, and (2) the connection between two measures of flatness, known as local entropy and local energy. Through a series of experiments, the authors show that the two measures are highly correlated and correlate well with generalization. They also empirically show that Entropy-SGD and Replicated-SGD, when used to explicitly optimize the local entropy, are able to flatter and better minima (in terms of lower generalization errors).

Clearly, the contributions of this work are not on either new algorithms or new measures, but rather it is a comprehensive empirical study that nicely brings seemingly different things together. I find the results interesting and the findings influential.

I have two clarifying questions:

(1) About activation functions in Section 5 and Section 6: For the results presented in Table 1, the deep networks use ReLU non-linearities, while in Section 5, the sign activation is shown in Equation 7 but then switched to an approximation by tanh(beta x). Can the authors clarify why using ReLU leads to an issue here? Please explain this “All these algorithms require a differentiable objective” and the use of a new loss function.

(2) Figure 3 and Figure 5, the training error difference of eSGD consistently decrease and gradually becomes smaller than that of SGD and rSGD. Does this have anything to do with the Langevin sampling? Can the authors give an explanation?

Minor comments:

+) In Equation (1), the loss L(w) is with respect to w, but what is L*(w, beta, gamma)? The same with d*(w, beta, gamma).

+) In Equation (3), E_train(w) is not defined. When \it{d}=0, then Omega(-d(w’, w)) in the denominator seems to be 0, which is strange?

+) Last paragraph before Section 6: do you mean Figure 1 instead of 6?. Also, there are some inconsistencies in the referencing figures: “Fig. 6”, “fig. 3” and “Figures 4 and 5”. The same problem is with equations: “(1)” vs “eq. (4)” and “Eq. (4)”.
review_title: Official Review #2

final_score: rating: 5: Marginally below acceptance threshold
review_content: Authors essentially study the generalization properties of networks trained via two different algorithms, Entropy SGD (eSGD) and Replicated SGD (rSGD)  against networks trained via SGD. Both eSGD and rSGD are algorithms that have been previously designed using the notion of entropy guided modified loss function. In this paper the authors compare the performance of broadly these three categories of networks in terms of Local Entropy (Eq. 3) and local energy (Eq. 4) (i.e. difference in training error when the weights are perturbed by factor \sigma), which was developed in [1]. With this the authors aim to find a correlation between networks with close to zero entropy and networks with lower local energy. 

The first experiment uses a simple two layer neural network with fixed final layer weights and studies both local entropy and local energy as a function of weight perturbation. (I tried hard to find where Figure 1 is explicitly mentioned but I simply couldn't. I assume the experiments in Figure 1 correspond to the network eq (7) - please make this explicit). 

Second set of experiments include simple classification test accuracies on deeper architectures such as ResNet110, DenseNet, EfficientNet and PyramidNet [this is not novel, just a replication and extension to newer architectures] (not including local energy, if I understand correctly.)

Third set of experiments are the main contribution of this paper - testing the correlation of local energy and local entropy - this is done on ResNet-18 trained on CIFAR10 dataset. While this does shed some light on the correlation of local entropy and local energy in promoting flatness, I'm not sure if one experiment on CIFAR10/Resnet18 is sufficient to justify correlations in deeper architectures that have been trained on larger datasets. What would have been more useful is to see if the same results translate for the models discussed in Section 6.1. 

In conclusion, I do think this is an interesting correlation but not enough experiments are provided to justify this correlation. 

References

[1] Fantastic Generalization Measures and Where to Find Them, Jiang et al ICLR 2019.
review_title: Ablation study on local entropy and local energy measures and how they correlate to generalization

Meta-review: This paper studies the link between generalization behavior and "flatness" of the loss landscape in deep networks. Specifically, the authors study two measures of flatness (local entropy and local energy), and show that these two measurements are strongly correlated with one another. Moreover they show via a careful set of numerical experiments that two previously proposed algorithms (entropy SGD and replica SGD) that optimize for local entropy tend to both find flatter minima as well as provide better generalization.

Despite the fact that the paper proposes no new models or algorithms, the experiments are compelling and provide non-trivial insights into predicting generalization behavior of deep networks, as well as solid evidence on the benefits of entropy regularization in SGD. The authors also seem to have satisfactorily answered the (numerous) initial concerns raised by the authors. Overall, I recommend an accept.
Decision: Accept (Poster)