Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper compares two well-known flat-minima-seeking methods, stochastic weight averaging (SWA) and sharpness aware minimization (SAM). Based on the geometric loss landscape analysis, the paper suggests combining SWA and SAM. Through experiments conducted on a wide range of tasks, the paper attempts to address when and why each method fails. 
strengths_and_weaknesses: **Strengths**
- The paper tackles a significant and practically-relevant problem supported by a fair amount of experiments conducted across various tasks and domains. There is a need for comparative studies of flat-minima-seeking algorithms.
- Overall, the paper is well-written and easy to follow.

**Weaknesses**

*(W1) Robustness to various settings:*
- In Section 3, it is unclear whether the findings in Figure 2 can be generalized to other settings. For instance, is the result robust to varying hyperparameters (e.g. SAM's radius) or even to different random seeds? Considering that this is an analysis paper, the robustness of the findings should be highlighted.
- Although the authors stated the scope of the work in Section 2.4, it is questionable whether the results would hold under variants of SWA and SAM (e.g., Adaptive SAM, Fast Geometric Ensemble).

*(W2) Credibility of linear interpolation analysis:*
- Regarding observations 3 and 4 in Section 3.1, one may consider asymmetric valleys [1]. While SAM solutions are closer to the sharper basin in certain directions, it is not guaranteed to be true in all directions. 
- I cannot find a 2D plot in the Appendix, which was mentioned in line 168. 
- It remains unclear how SAM+SWA solutions escape the saddle point in Fig 3.

*(W3) Novelty of proposed algorithm:*
- The proposed way of mixing SWA and SAM outperforms baselines in half of the proposed cases. In this regard, it lacks novelty. 
- Further, the motivation, i.e., observation 4, needs to be enhanced in line with the previous comment (W2). For example, I wonder if dominant Hessian eigenvalue analysis would hold under different settings since it is well-known that such analysis is not translation invariant w.r.t. neural network parameters.

*(W4) Lack of interpretation:*
- While the findings in Section 4.4 are interesting, it is mostly devoted to when SWA and SAM fail, not why. In order to conduct future research based on the paper’s observations, well-reasoned interpretations or hypotheses are crucial. 
- As in A.2, a study concerning global flatness and well-connectedness or other distance metrics (e.g., CKA) would be worthwhile [2]. 

**References**\
[1] He, Haowei, Gao Huang, and Yang Yuan. "Asymmetric valleys: Beyond sharp and flat local minima." Advances in neural information processing systems 32 (2019). \
[2] Yang, Yaoqing, et al. "Taxonomizing local versus global structure in neural network loss landscapes." Advances in Neural Information Processing Systems 34 (2021): 18722-18733.

------------------------------------------------
(Post-rebuttal) I have updated my score to 5. 

questions: - Do SWA solutions and non-flat baseline solutions share optimization trajectories? If so, observation 1 seems to be a naïve consequence.
- In Section 3, is Adam optimizer used for SWA and SAM in GIN/Code2 task? Or, only for non-flat solutions?
- Regarding line 296, I cannot find an explanation in the Appendix as to why SWA and transformers do not work well together.

limitations: The paper provided limitations along with future research directions in Section 5.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper focuses on two famous flat-minima optimizers, SAM and SWA. They provide lots of experiment results to analyze the solution of these two optimizers and find that they focus on different aspects. Finally, they try to combine them to get better performance than SAM or SWA and conduct some experiments in a variety of tasks to help us better understand these two algorithms.
strengths_and_weaknesses: Strengths:

1. This paper provides lots of experiment results in various tasks to analyze SAM and SWA, which help us better understand them.

2. The proposed method is very clear and simple to reproduce.

3. SWA+SAM can achieve better performance in some tasks.

4. The figures and writing are very easy to understand.

Weakness:

1. Although the paper provides lots of experiment results, some important experiments are not included in this paper. For example, some related work illustrate that SAM can achieve a huge improvement in ViT training on ImageNet (5% improvement). I think this is very important for people to better understand the relationship between SAM and SWA. 

2. Some comparison with current state of the art methods, such as GSAM.

3. The accuracy of SAM+SWA is not better than vanilla SAM in some experiments. In addition, in some tasks the performance are very close. I think that will also limit the contribution of this paper.

4. About the novelty of this paper. Although the paper focus on a very important problem, but the proposed method try to combine current two methods, SAM and SWA, which will also weak the contribution of this paper.
questions: 1. In the experiments, whether the distributed setting is used. The main reason is that we find this setting is very important for SAM, which may also affect the analysis of SAM.

2. We know that data augmentation can improve the generalization of neural networks. When using data augmentation, the improvement of SAM will be reduced. What is the relationship between data augmentation, SAM, and SWA.


limitations: The paper provides some experiment analysis to illustrate the properties of SAM and SWA. 

1. The contribution of this paper. Combining SAM and SWA is not novel for me.

2. The performance of SWA+SAM is not very great for me. In some tasks, the performance are very close between SAM+SWA and SAM/SWA and in some experiments SAM+SWA will hurt the performance.

3. The experiment results of ViT training on ImageNet. I think this experiment is very important for SAM since SAM can achieve about 5% improvement on ImageNet.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors perform a detailed comparison of stochastic weight averaging (SWA) and sharpness-aware minimization (SAM), first with loss landscape analysis and then with a broad set of benchmarks. The authors find several surprising results, and in particular show that combining the two methods, it is often possible to achieve improvements over both of them, leading to consistently good performance.
strengths_and_weaknesses: **S** = strength, **W** = weakness, **Q** = question.

**S1**. The authors perform a detailed benchmark study that was missing from the literature. It is interesting to see that SAM and SWA are in fact quite complimentary, without one method clearly dominating the other.

**S2**. The benchmarks considered by the authors are quite diverse, including multiple domains. Moreover, the results differ between the different domains, showing the importance of a diverse benchmark.

**S3**. The combination SWA+SAM seems to be quite useful for practitioners potentially, as it works consistently well across a  broad set of benchmarks, while SWA and SAM are somewhat less consistent.

**S3**. The authors perform loss surface analysis experiments to provide intuition behind the results.

**W1**. In section 3.1 it is not clear how exactly the solutions $\theta^{NF}$ are obtained for each optimizer. My guess is that for SWA the authors used the same optimization trajectory but without averaging to obtain  $\theta^{NF}$, while for SAM they used an independent run? In this case, the relation between $(\theta^{NF}, \theta^{SWA})$ and the relation between $(\theta^{NF}, \theta^{SAM})$ will be quite different. In particular, I would expect the distance spanned by the horizontal access in Figure 2 to be much larger for SAM compared to SWA, making the interpretation of the plots more nuanced.

**W2**. In lines 226-227 the authors mention that for SWA they simply average the iterates of the baseline optimization run, without modifying the learning rate schedule. While this is a valid approach, the authors of SWA emphasize the importance of using a non-standard learning rate schedule (see e.g. [the official blogpost](https://pytorch.org/blog/pytorch-1.6-now-includes-stochastic-weight-averaging/#is-this-just-averaged-sgd)). The deviation from the standard SWA recipy makes the benchmarking results a bit harder to interpret.
questions: **Q1**. Could you please comment on the base optimizer used in each data domain (Vision, NLP, GRL)? Could the SWA poor performance on NLP tasks compared to Vision be explained by the different choice of base optimizer?

**Q2**. Could you pleas comment on W1? How is  $\theta^{NF}$ computed for both SWA and SAM? What is $\|\theta^{NF} - \theta^{SWA}\|$ and $\|\theta^{NF} - \theta^{SAM}\|$ in Fig. 2?

**Q3**. Could you pleas comment on W3? Why did you choose not to use a cyclical or constant learning rate schedule for SWA? Would it be possible to evaluate SWA with a constant learning rate schedule after epoch $E$ for a subset of your benchmarks to understand whether the choice of the learning rate schedule affected your results?
limitations: The authors adequately discuss the limitations of their work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper analyses the properties of two flat minima optimizers, Stochastic Weight Averaging (SWA) and Sharpness-Aware Minimization (SAM). The nature of the minima reached by the two methods is studied through linear interpolations and dominant Hessian eigenvalues. SWA and SAM are benchmarked across a wide range of tasks in Natural language understanding, Graph Representation Learning, and Image Classification. Based on these results, a set of observations about SWA and SAM are described.
strengths_and_weaknesses: Strengths:
1. The analysis of the minima found by SAM and SWA has not been studied earlier. The observations are quite interesting and valuable to the community.
2. The authors perform a detailed comparison of SAM and SWA across a wide range of benchmarks. All the results are reported with error bars.
3. Overall, the paper is well written, with novel observations clearly stated and explained with supportive evidence.

Weaknesses:
1. This work has not studied the role of the base optimizer. The authors have compared SAM and SWA across multiple tasks and architectures. However, some methods use SGD, while others use Adam as the base optimizer. The loss landscape and generalization have been compared with SGD and Adam [1]. The comparison of SGD vs. Adam for the same task with SWA and SAM, respectively, would help disentangle the impact of the base optimizer.

2. Regarding Observation 3 (L166): "We visualize 2D plots moving along random directions (not shown here due to space)". This plot must be added to the Appendix along with the observations. Otherwise, the saddle point hypothesis is not backed up with evidence. Additionally, it would be good to plot the Eigen Spectral Density [2] and show the presence of significant negative eigenvalues, which can confirm the presence of saddle point.

[1] Zhou, P., Feng, J., Ma, C., Xiong, C., Hoi, S. C. H., et al. Towards theoretically understanding why sgd generalizes better than adam in deep learning. Advances in Neural Information Processing Systems, 33:21285–21296, 2020.

[2] Ghorbani, Behrooz, Shankar Krishnan, and Ying Xiao. "An investigation into neural net optimization via hessian eigenvalue density." International Conference on Machine Learning. PMLR, 2019.
questions: 1. Including observations relating to the robustness of the optimizer to its hyperparameters would also add practical value to the work. The role of the hyperparameters of the flat-optimizer in reaching certain minima can also be studied. For example, Does a large $\rho$ lead to a completely different basin than a small $\rho$?
2. It would be clearer if the authors mentioned the tasks/models where a pretrained model is used. For example, it is mentioned in the Appendix that the CIFAR-10 ViT results used a pretrained ImageNet ViT. The characteristics of the flat optimizers might be very different if a model is trained from scratch with the flat optimizer instead of being finetuned with the flat optimizer.
3. It is mentioned that one of the reasons why flat optimizers fail is because the train and test minimizers are not correlated. Can this hypothesis be more thoroughly evaluated by plotting the linear interpolation plots on an OOD test set?. For example, for the CIFAR-10 dataset, one could evaluate on CIFAR-10.1/ CINIC-10 dataset.
4. Question 3 (d) of the checklist has been answered: "Yes, See Appendix." But the details regarding the compute are not available in the Appendix.
5. Why does Algorithm 1 returns $\theta^{SWA+SAM}$? Should it not return  $\theta^{SWA}$
limitations: The limitations of the work have been described in detail.
review_title: null

Meta-review: metareview: This paper is an empirical investigation comparing two popular optimization techniques, namely SAM and SWA. Authors compare the performance of these two methods over a wide range of tasks and architectures. They also inspect and compare the properties of minima found by these methods. Given the existing interest in the ML community to understand these methods, reviewers are in agreement that insights from these paper are valuable and impactful enough to accept this paper. 
confidence: Certain
Decision: Accept