Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes to make the recent Sharpness-Aware Minimization (SAM), which tries to find flatter minima (which tend to generalize better) more efficient. Vanilla SAM incurs a 2x compute as 2 gradient computations (one for the perturbation and one at the perturbation) are needed at each training step. The paper proposes sparsifying SAM (SSAM) by only applying SAM on some fraction $s$ of the model weights. Which $s$ fraction is determined either in a cheap greedy fashion (SSAM-D) or via Fisher Info (SSAM-F). Non-convex analysis is done showing that SSAM converges at the same rate as SAM. SSAM is shown to perform nearly as well as SAM on benchmark tasks like CIFAR10, CIFAR100, and ImageNet using a ResNet model of modest size.

strengths_and_weaknesses: Strengths:
* The paper is well-motivated -- SAM is a powerful method but its 2x performance hit may hinder its widespread adoption and this paper shows how it can be sped up with little loss in performance.
* Paper is clear, easy-to-follow, and mostly free of typos.
* Theoretical analysis supports the proposed method.

Weaknesses:
* Although SSAM does in theory reduce SAM's FLOPS, it's unclear whether it would actually help in practice. Due to the nature of how gradients are computed in practice, computing gradients for a sparse subset of weights is often not much faster than computing the full set, especially when the subset is 50% and the weights could be from any of the layers. While the paper shows theoretical FLOPS, it does not have any numbers on actual wall-clock speed, etc, though I do understand that reporting wall-clock speed has some issues of its own.
* Key ablations missing. What happens if you invert the mask? For SSAM-D, instead of the dropping the flattest weights from the mask, drop the sharpest weights. Furthermore, what if a random subset is dropped?
* What's the influence of m-sharpness (doing m perturbations and averaging the gradients at the perturbations -- see original paper) on SSAM? 
* Another strategy to reduce compute is using a 20% subset of the minibatch for the ascent gradient calculation (e.g. to find the perturbation). This strategy reduces the FLOPs significantly. How does SSAM compare to SAM when this approximation is used? Reference: https://arxiv.org/abs/2110.08529.
* The claim that SSAM can outperform SAM is weak: looking at Table 1, cifar100 95% and 98% have 0.40% gap for SSAM-F, which is larger than SSAM-F 50%'s improvement over SAM, suggesting the claimed improvement may just be noise.
* Tables start sparsity at 50%. Interested in seeing 1%, 5%, 10%,.. etc. At low sparsities it should be nearly identical to SAM.







questions: * Ablations would greatly strengthen the paper.
* More experiments / datasets would also help.
limitations: Yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The problem addressed in this study is that the indiscriminate perturbation manipulated by SAM on all parameters is suboptimal. The authors give two interesting observations that: there is little difference between the updating gradients of SGD and SAM in most dimensions; and there exist certain parameters that will still be flatter in some dimensions even if SAM is not used. 
Based on the observations, they propose an efficient and effective training scheme to obtain sparse perturbations by crafting a binary mask. This paper employs the Fisher Information and dynamic sparse training to select top-K most significant and relative parameters to craft a sparse binary mask for perturbation. 
The authors verify the effectiveness of their proposed training scheme on CIFAR10/CIFAR100 and ImageNet-1K datasets. The experimental results demonstrate that even with very large sparsity, both SSAM-F and SSAM-D can still obtain competitive performance against SAM. The authors also conduct visualization of landscape and Hessian spectra to show better generalization of SSAM.

strengths_and_weaknesses: Strengths: 

1. The motivation is transparent and derived from the experimental observations. The authors accurately identify the indiscriminate perturbation in previous sharpness-related works. The motivational analysis and experiments are persuasive to me. 
2. The written is straightforward and easy to understand. In addition, the landscape visualization is also informative from a motivational standpoint. 

Weakness: 
1. The experimental settings are not sufficient as compared with SAM paper. It would be better to have further experiments and a few ablation studies to confirm the model-agnostic characteristic of this technique. 
2. There is a misunderstanding between the definitions of sparsity in line 176. The sparsity s should be the percentage of parameters to be perturbed, i.e., s=1 means all the parameters should be perturbated. However, s=1 in the table 1 means no parameters will be perturbated. Could the authors check for the definition? 
3. It would be more convincing if the authors could have more analysis of the experimental outcomes. In the section titled "Questions," several inquiries regarding experiments are presented. 


questions: 1. The results shown only from sparsity 50% and only for ResNet model. Could the authors have more experimental results on different sparsity and models with different architectures? 

2. Why does it appear that employing fewer perturbation in Table 1 is unable to enhance the performance, if according to your purpose, that you can find certain parameters that can make loss flatter without perturbation? 

3. As stated in Section 3.2, “merely about 5%parameter space is sharp while the rest is flat”. Do you believe you have correctly identified the 5% parameters that do not require perturbation application through your methods and that experimental results demonstrate this? 

I am willing to increase my rating if my questions could be my addressed properly. 

limitations: The authors haven’t addressed the limitations and potential negative societal impact of this work. It would be better to have discussion on SSAM’s effectiveness of more different datasets and model structures, as well as the potential future work of SSAM.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper rethinks the indiscriminate perturbations of SAM on all parameters and claims that not all parameters are needed to be perturbed. The conclusion is supported by two observations --- (i) little difference between SGD and SAM gradients in most dimensions and (ii) more flatter without SAM in some dimensions. Inspired by these observations, this paper proposes to improve the effectiveness and efficiency of SAM via a sparse perturbation scheme (SSAM). Two realizations of SSAM are provided, namely SSAM-Fisher and SSAM-Dynamic. The convergence of SSAM is also discussed. Extensive experiments, including visual and numerical results, demonstrate the effectiveness and efficiency of SSAM.
strengths_and_weaknesses: Strength:

- The paper is well-written and organized.  
- Interesting observations on the flatness of SGD and SAM. It is surprising that the difference between SGD and SAM gradients is little and flat dimensions will be flatter without SAM
- It is quite natural to apply fisher information and dynamic strategy to encourage sparsity
- The effectiveness of SSAM is comparable to SAM; the computational cost is clearly reduced.

Weakness:
- I am very interested in why SAM makes the originally flat gradients worse?
- The ArgTopK operation selects $k$ parameters to be perturbed. How sensitive is the performance of SSAM with respect to $k$ in a certain task? How sensitive is the value of $k$ in different tasks (if training SOTA classifiers)?


questions: Please refer to the weakness part
limitations: N.A.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper presents a sparse version of sharpness-aware minimization (SAM). The proposed SSAM does not apply perturbation to all the model parameters, where the criteria is based on the Fisherman Information or dynamically generating a mask through training. The authors claim improved generalization and training speed over SAM.
strengths_and_weaknesses: - Strengths:
    - The idea and observation that not all the model weights need to be perturbed in SAM is generally novel. The authors also provide empirical justification that the difference between $g_{SAM}$ and $g_{SGD}$ is not very significant for lots of weights.
    - I like the provided training curves as they show that SSAM behaves very alike to SAM in terms of the training dynamics.


- Weaknesses:
    - I don't quite understand how the FLOPs reduction are calculated in Table 1, 2, 3. From the algorithm, SSAM still requires twice full back-propagation. The only difference is multiplying the mask with the perturbations. Can the authors explain the FLOPs and it would be better to show the clock time of the training process.
    - I don't quite understand why option 2 is faster than option 1 when generating the mask (the first sentence in Section 4.3). From my understanding, they both rely on the gradient scale and require twice gradient computations.
    - Can the authors further demonstrate that the weight they select to perturb coincide with those appear to the right half in Figure 1? Just want to confirm that the masking strategy is align with the gradient information.
    - A random masking baseline should be included to show the effectiveness of SSAM.
questions: Please see the weaknesses for questions.
limitations: The authors haven't covered the limitations. Potential limitations are limited results on other architectures besides ConvNets like Transformers.
review_title: null

Meta-review: metareview: This paper proposes a new scheme to improve computational efficiency of the SAM optimizer. The original SAM requires to asses the loss value at a perturbed point. The perturbation lives in the full parameter space. This paper shows that computing the perturbation in every direction is not necessary. By only selecting a sparse subset of the parameters to undergo a perturbation, the authors are able to maintain the original generalization performance of SAM while significantly reducing the number of FLOPS. The paper also presents a convergence analysis for the proposed scheme.
There was an active discussion between authors and reviewers, and authors provided very very thorough response to the questions and issues raised by the reviewers. As a result of this, multiple reviewers increased the original score. At the end, 3 out of 4 reviews are on the positive side, and the remaining one is a borderline reject. 
In concordance with the majority of the reviewers, I believe improving the computational cost of SAM is of huge practical interest, and this paper is a step toward in direction. I recommend accept. As the thorough reply from the authors contains interesting details, I strongly suggest that authors include these details to their submission (maybe to the supplementary material?)

confidence: Certain
Decision: Accept