Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a sharpness-aware minimization method that adds almost no computational overhead. The authors propose to use the training loss differences (more precisely, the KL divergence of output predictions) to proxy the sharpness term. The advantage of using this new metric would be that it only requires additional memory but no additional computations. They also propose an adjusted method that addresses the memory problem of the first method. The effectiveness of the proposed methods is then shown experimentally for Imagenet and CIFAR datasets.
strengths_and_weaknesses: Strengths:

1- The paper is very well-written. 

2- The limitation of the initial method in terms of memory has been properly addressed in the follow-up method.

3- The method is simple and efficient to use.

4- The experiments use state-of-the-art settings.

Weaknesses:

1- The method doesn't always outperform SAM in terms of accuracy. This is not so much a weakness because the method is mainly intended to improve computational overhead.

2- The experiments are only for image-classification tasks.

3- The method introduces a few hyper-parameters.
questions: 1- It is mentioned that before epoch E_start the outputs are not stable. What would happen if from the first epoch this method is used? Does this instability affect the performance?

2- How should the hyper-parameters (tau, E, and lambda) be selected for your method? Have you studied various hyper-parameters? It is mentioned that a fixed value is used for these hyper-parameters.

3- In Figure 4b, why does MESA have higher sharpness at the beginning of training but lower sharpness at the end of training compared to SAF? Also, from the visualization, it appears that MESA has the flattest landscape.

limitations: The limitation of the method with regards to memory is mentioned and properly addressed.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper proposes efficient implementations of sharpness-aware training. Specifically, this paper proposed Sharpness-Aware Training for Free (SAF) which requires cached prediction for previous checkpoints, and Memory-Efficient Sharpness-Aware (MESA) training which requires two forward propagation and one back propagation for each optimization step to achieve sharpness-aware training. While these approaches require relatively light computation costs compared to the original SAM, they achieved competitive accuracy in various tasks including CIFAR-10/100 and ImageNet-1k.
strengths_and_weaknesses: * Originality: This paper provides a novel idea to estimate the sharpness by leveraging the trajectory of weights (Eq. 7). Based on this idea, they proposed time-efficient (SAF) and memory-efficient (MESA) implementations of sharpness-aware training [+].

* Clarity: While Fig. 3 is intuitive to understand the key idea of this paper [+], it is still unclear to me that the equation in L164 and the second approximation in eq 7. Also, it is unclear that \Phi_t is defined on what parameter of the optimization step [-].

* Significance: I found that there exists a gap between reported SAM accuracies in previous studies and this paper. In Table 1 of this paper, the authors reported SAM accuracy as 76.9/78.6/79.3 for ResNet-50/101/152. However, the original SAM paper reported their accuracy as 77.5/79.8/80.8 with 100 epoch training (See Table 2 of [1]). I think this gap is critical since the accuracy of SAF 77.8/79.3/80.0, the minor improvements only exist for ResNet-50, and MESA 77.5/79.1/80.0, there are no improvements for SAM. Also, the ViT-S/32 results in Table 1 also make some confusion: In Table 2 of [2], they reported the accuracy of ViT-S/32-SAM as 70.5, which is severely better performing than the 68.9 in this paper. The authors should clarify this for a fair comparison.

[1] Foret, Pierre, et al. "Sharpness-aware minimization for efficiently improving generalization." arXiv preprint arXiv:2010.01412 (2020).

[2] Chen, Xiangning, Cho-Jui Hsieh, and Boqing Gong. "When vision transformers outperform ResNets without pre-training or strong data augmentations." arXiv preprint arXiv:2106.01548 (2021).
questions: Q1. In Fig. 2 and Fig. 5, the authors provide visualization of the loss landscape for various sharpness-aware training. Although the results showed that SAF/MESA found flatter local minima than SAM, one cannot find the clear intuition behind these empirical results. Also, while these visualizations tell us that MESA found flatter local minima than SAF for WRN28-10/CIFAR-100, the test accuracy of SAF is better than MESA. Therefore, it can be interpreted as flatter local minima considered in this paper would not result in better test accuracy in practice. Authors should clarify this paradox to remove the confusion of readers.

Q2. As mentioned in the original SAM paper, the SAM optimizer can find local minima with better test accuracy with longer training (See Table 2 of [1]). Can SAF and MESA also utilize the longer training without overfitting? I think this property would be important to the scalability of SAF & MESA.
limitations: While this paper provides a novel interpretation of sharpness by leveraging the trajectory. several points are needed to improve the clarity and significance of the paper.

1. It would be recommended to provide the detailed derivations of L163-L167. While the intuition behind these lines is clear, further clarity can be attained by providing the entire derivation of equations to readers.

2. It should be clarified why SAM accuracy in Table 1 is severely lower than the original SAM paper [1]. Since they did not report any confidence interval in Table 1, this accuracy gap can change the interpretation of experimental results in Section 4.1. 

3. It would be recommended to add the results of SAF/MESA for longer training and stronger augmentation. Since many SOTA models in computer vision [3,4] exploit longer training and stronger augmentation, the scalability of SAF/MESA should be validated with those settings.

[3] https://pytorch.org/blog/how-to-train-state-of-the-art-models-using-torchvision-latest-primitives

[4] Liu, Zhuang, et al. "A convnet for the 2020s." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

** After discussion phase ** 

Thanks to the clarification of the authors, I resolved my misunderstanding of this paper. However, I think these discussion points should be reflected in the main paper: Their key trick using $\arg \min$ is not specified in their derivation of the main paper. Also, further clarification on differences between main references (e.g., label smoothing, number of epochs) should be clarified in their main paper since the gap between their performances is sufficiently large to change the interpretation of experimental results.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper focuses on the computation cost perspective of Sharpness-Aware Optimization (SAM), which has 2x cost compared with SGD. 
The first proposed algorithm, SAM for Free (SAF), derives an alternative/surrogate optimization objective which can uses the past optimization trajectory to estimate the sharpness loss. The SAF algorithm has almost the same computation cost as SGD, while inducing some extra memory usage. To mitigate the memory cost, the authors propose the second algorithm, Memory-Efficient Sharpness-Aware Training (MESA), which reduces the memory usage significantly at the tradeoff of 15% more computation cost. The empirical results of SAF and MESA are pretty good compared with SAM and SGD on popular benchmarks (CIFAR and ImageNet), indicating that the the proposed algorithms indeed reduce the computation costs of SAM while maintaining the performance.
strengths_and_weaknesses: Strengths:

1. The SAF algorithm is proposed in a principled way: The authors studied the SAM loss carefully, and obtained some interesting observations (Sec. 3.1), naturally leading to an elegant algorithm

2. The authors indeed make the algorithm practical: Even though the general idea (Sec. 3.1) is elegant, it may still have some limitations when being used as a practical optimizer. The authors tackled some numerical challenge in Sec. 3.2, and moreover, they mitigated the memory cost issues by using the trick of exponential moving average (EMA), leading to the memory-efficient version (MESA). I really appreciate the efforts the authors spent on making the algorithm practically useful.

3. The empirical results are good: The proposed algorithms, SAF and MESA, have slightly better performance than SAM with ~1/2 computation cost. Also, the extra memory cost of MESA is indeed negligible compared with SAF.

Weakness: 

1. Is the replacement of cross-entropy with KL (in Sec. 3.2) really necessary? I think cross-entropy is still the best choice for classification, so I guess the replacement may have some negative impacts. I think it's better to have an ablation study on this replacement.
questions: 1. Is the reduction of computation cost from 2x to 1x SGD really meaningful? 
Theoretically speaking, 1x and 2x costs are about the same. Empirically, I know there are many factors that affecting the computation cost of training, and I'm not sure if 50% cost reduction is very meaningful practically. 

2.  See Weakness-1 above. That's one of my questions.
limitations: See the Questions section above.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes two methods for sharpness aware minimization (SAM) without the cost of SAM, as SAM doubles training cost. The methods are 1) SAF which adds a distillation loss from the predictions of prior epochs and 2) MESA which adds a distillation loss from the exponential moving average of the weights. For resnets on imagenet, SAF and MESA match or outperform other methods including SAM, ESAM, and Vanilla SGD.
strengths_and_weaknesses: Strengths:
- The paper tackles and interesting and practical problem, which is how to make training better without the added cost of methods like SAM.
- For ResNets on ImageNet, the empirical performance when compared to baselines is good. Moreover, this is without the cost.

Weaknesses:
- I do not understand how MESA is only 15% more expensive than standard training. If I am understanding correctly, MESA requires a forward pass through both the EMA weights and the standard weights. Doesn't this require two forward passes where there was previously only one?
- It seems like free should not be in the name of SAF because there is an additional memory overhead (minor)
- I did not understand very well how SAF and MESA relate to SAM. In particular, Equation 8 is not well motivated and I did not see where the KL came from.
- I have a few concerns with the ResNet baselines: 1) The ResNet baseline for SAM is below that reported in the SAM paper -- could this perhaps be because perturbations are synced? 2) Modern ResNets, e.g. with techniques from the timm library which is the library used in this paper can get to 80% accuracy (https://arxiv.org/pdf/2110.00476.pdf). Are the methods presented in this paper additive with these modern approaches?
- 
questions: I understand that there is not space to answer all of these questions.
- Why is the ResNet baseline for SAM much lower than that reported in the SAM paper? 
- Why is MESA only 15% more expensive than standard training
- The best ResNet50 in the paper is < 78%. However, modern day ResNets (e.g., from https://arxiv.org/pdf/2110.00476.pdf in the popular timm library, which is used in this paper) can get ~80%. Are the methods in this paper additive with more modern approaches.
- Why is GSAM much better than all other methods for the ViT model (minor).
limitations: The paper claims to address this in Section 3.3 but a revision could benefit from an explicit limitations section (minor).
review_title: null

Meta-review: metareview: This paper proposes a novel optimization method, called SAF, for reaching flat minima. The main claim is that the proposed method does not suffer from computational overhead of SAM-like methods which is typically 2x SGD. The proposed method is based on a novel loss that minimizes the KL-divergence between the output of the network with previous weights and current weights. This allows avoiding the extra computational overhead of SAM. Authors show that their method can achieve better empirical results in a compute-constrained regime. 

Reviewers are in agreement about the novelty of the proposed method and that the paper is well-written and easy to follow. The empirical results also show a clear advantage over other methods in a compute-constrained regime. The main concern about accepting the paper is due to some mismatch between reported numbers in the paper and that of original SAM paper as well as lack of clarity on some experimental details. I am leaning towards acceptance given the advantages mentioned above but I strongly recommend authors (and want to see this implemented for camera-ready) to at the very least make the following changes (as well as the ones proposed by reviewers) to adhere to clarity and reproducibility standards of publications in computer science and increase the impact of their paper:

1- In Table 1 (and perhaps Figure 1?), add ALL reported results for SAM including but not limited to a) The results reported in the original SAM paper. b) The results reported by other papers running SAM themselves c) The results of running SAM by authors.

2- In Table 1 (and perhaps everywhere), always make it extra easy for the reader to know if the number is taken from another paper or it is a the result you reproduced yourself.

3- Add all experimental details (including augmentation techniques used and what hyper-parameters are tuned) for all experiments. When there are mismatches that makes the comparison more difficult, this becomes even more important and allows the reader to make-up their mind about where the improvement might be coming from.

4- If you have considerations about original SAM paper’s numbers not being reproduced by other papers, you can add it as a footnote or discussion but it is still important to report them.
confidence: Less certain
Decision: Accept