Critical Influence of Overparameterization on Sharpness-aware Minimization

Authors: Sungbin Shin, Dongyeop Lee, Maksym Andriushchenko, Namhoon Lee
Primary area: optimization
Keywords: sharpness-aware minimization, overparameterization

Abstract:
Training overparameterized neural networks often yields solutions with varying generalization capabilities, even when achieving similar training losses. Recent evidence indicates a strong correlation between the sharpness of a minimum and its generalization error, leading to increased interest in optimization methods that explicitly seek flatter minima for improved generalization. Despite its contemporary relevance to overparameterization, however, this sharpness-aware minimization (SAM) strategy has not been studied much yet as to exactly how it is affected by overparameterization. In this work, we analyze SAM under varying degrees of overparameterization, presenting both empirical and theoretical findings that reveal its critical influence on SAM's effectiveness. First, we conduct extensive numerical experiments across diverse domains and show that SAM consistently improves with overparameterization. Next, we attribute this phenomenon to the interplay between the enlarged solution space and increased implicit bias resulting from overparameterization. Furthermore, we show that this effect is particularly pronounced in practical settings involving label noise and sparsity, and yet, sufficient regularization is necessary. Last but not least, we provide other theoretical insights into how overparameterization helps SAM achieve minima with more uniform Hessian moments compared to SGD, and much faster convergence at a linear rate.

Decision: reject
Reviewer scores: 6, 5, 6, 6

Reviews and discussion:
<thread>
creation date:
1730234226204

thread_type:
official_review

<thread_object>
creator: reviewer_l6ky

<content>
summary:
This work attempts to disclose the critical influence of overparamatrisation on  Sharpness-Aware Minimization  (SAM) from experimental and theoretical perspectives. It starts with an extensive evaluation to display a  highly consistent view showing the generalization benefit of SAM increases with overparametrisation. Without such overparametrisation  SAM may not work. This leads authors to consider the benefit in terms of more solutions and implicit bias. Finally, they developed theoretical advantages and advantages of overparametrization for SAM on linear stability, convergence and generalization.

strengths:
Nice empirical and theoretical work on overparametrisation on sharpness-aware minimization. It is interesting and useful for ICLR communauty

weaknesses:
See questions below.

questions:
(1) Please clarify the large curve surrounding the "real" curve in Fig . 1 (and possibly in other Figs). Specify if the shaded regions represent confidence intervals, standard deviations or soe other measures of uncertainty. 
(2) How can SAM improvements be compared to SGD improvements via overparametrisation? Hoa SAM and SGD performance changes with increasing model size (may be you can add provide (in Annex) plots showing relative improvement over a baseline for both optimizers across different nodel sizes)
(3) Min Max optimization has some limitations (High complexity; Oscillatory behavior, sensitivity to initialization, etc.), how these problems are addressed here ? Can equation (3) (or some of its properties) help mitigate these issues ? Did you observed these specific issues in your experiments and if so, how have you addressed them ?
(4) Can we use Elliptic Loss Regularisation for SAM instead of Min Max? Please comment on whether Elliptic Loss Regularization could be a viable alternative to the min-max formulation in SAM, and what potential advantages or disadvantages it might have in this context.
(5) Is there a sort of "optimal" number of parameters for which the computation price is acceptable ?Did you  observed any diminishing returns in performance improvement as model size increased. Please provide guidance on balancing computational cost with performance gains when using SAM in practice
</content>
</thread_object>

</thread>
<thread>
creation date:
1730362582135

thread_type:
official_review

<thread_object>
creator: reviewer_8b2s

<content>
summary:
This paper claimed that SAM benefits from the over-parameterization. They find that SAM could find different (e.g. simpler) solutions than SGD in the over-parameterization regime. In addition, they find that SAM is robust to label noise and sparse parameterization. They also include some theoretical analyses, thought these analyses are irrelevant to their main findings.

strengths:
1. The authors picked up an interesting perspective to investigate the mechanism of SAM. This research topic is significant.
2. The experiments are extensive, across various datasets and architectures.
3. The writing structure is clear, and the paper is easy to follow.

weaknesses:
Firstly, for the empirical findings,

1. It is unclear that whether over-parameterization consistently helps SAM find more generalizable solutions as the model parameters scale up. As shown in Fig.1 and Fig. 7 (I think the Fig. 7 is clearer), for MLP on MNIST, ResNet-18 on CIFAR-10, LSTM on SST2 and CNN on Atari, when model get more over-parameterized, the generalization improvement does not always increase; sometimes, a noticeable decline is observed. Moreover, as noted by the authors, when there is no sufficient regularization, SAM might not benefit from the over-parameterization (as shown in Fig. 5). In addition, I notice that the authors conducted experiments for ViT on CIFAR-10 in Fig.5 but not in Fig. 1. I wish the authors could include the results for ViT on CIFAR-10 in Fig.1.
2. Increasing depth of models yields different conclusions than increasing width. In Fig. 1, authors increase the number of neurons in each layer, leading to a seemingly consistent conclusion that over-parameterization helps SAM find generalizable solutions. However, in Fig. 17, when increasing the number of layers, SAM might not benefit from the over-parameterization (ResNet on CIFAR-10). Clearly, the increase of both depth and width demonstrates the over-parameterization, however, the conclusion differs. 
3. It is not surprising that the marginal improvement of SAM over SGD increases with higher label noise. In Fig. 12, in the over-parameterization regime, the performance of SGD drops significantly with higher label noise rate; however, SAM remains good generalization ability. Indeed, it is well known that SAM is robust to label noise, while SGD not. Thus, it is not surprising for us to observe such phenomenon. Also, it seems that the authors over-claimed that over-parameterization secures the robustness of SAM to label noise. Indeed, the observed increase of margin improvement of SAM over SGD with larger model parameters is primarily due to the pronounced decline in SGD's performance as the model parameters scale up under high label noise conditions.
4. The conclusion on the effect of sparsity contradicts with the main argument that SAM benefits from the over-parameterization. In Fig. 5, it is observed that the generalization improvement tends to increase as the model becomes sparser. However, increased sparsity indicates that the model is becoming more under-parameterized. The authors are suggested to explain such contradiction.
5. In Fig. 2 and 3, the authors compare GD and SAM, however, a comparison between SGD and SAM is more preferred.
6. In section 4.2, “implicit bias of SAM increase with over-parameterization” is over-stated. In this section, the authors demonstrated that we could use larger perturbation radius when scaling up the model parameters, however, this doesn’t indicate that a larger scale of model parameters induces stronger implicit bias towards flatter minima. 

In short, SAM doesn’t always benefit from the over-parameterization in practice. A more thorough justification is needed to clarify whether over-parameterization is advantageous for SAM and under what specific conditions it may provide benefits. Indeed, it is quite expectable that SAM would benefit from over-parameterization sometimes. Because in the under-parameterization regime, the model capacity is low (say only one solution in the solution space), thus both SGD and SAM achieve similar solutions. Once scaling up the parameters, the solution space is getting larger, thereby SAM could differ from SGD and find more generalizable solutions. However, SAM might not always from the over-parameterization. 

Second, for the theoretical analysis,

1. As noted by the authors, the theoretical analysis cannot support the findings in Section 3 and 4, and thus the significance of the theoretical analysis is limited. The section 6 is more like a collection of possible theoretical analyses that could be done for SAM. 
2. Indeed, there is no clear theoretical support for the main discovery of this paper, as noted by the authors in the limitations part in Section 7. This is another critical issue of this paper.

Overall, the major issue of this paper is the novelty. The empirical findings are largely anticipated, and the theoretical analysis closely follows prior works without introducing any new proof techniques.  Also, some statements are clearly over-claimed. The contribution of this paper is marginal, and thus I lean to rejection.


----------------
As I cannot respond in official comments further, I put my responses here:

Based on your clarification on the ViT experiment, I decide to increase my score from 5 to 6. However, I strongly recommend the authors to revise the paper from the following aspects:

- **Overstated claims**: Some statements, like "over-parameterization consistently improves the generalization benefit of SAM" in the introduction part, are over-stated, which might be misleading for the readers. Some failure experiments such as ViT, increasing depth of networks, are encouraged to put earlier in the main text, along with throughout discussions. *I firmly believe that adhering to strictness is not a bad thing for a scientific paper*.
- **Experiments on Absolute validation metrics**: Experiments using the absolute validation metrics are encouraged to put into main text along with the experiments using "Acc improvements".
- **Current theory cannot directly support your main finding**: Still, currently, your theory cannot directly support your finding. I am not diminishing your contribution, but clearly those complementary theories in current paper cannot persuade me to highly evaluate your paper. I wish the authors could consider my suggestions and improve the paper with more related theoretical analyses. 

BTW, I decide to keep my confidence score low.

questions:
N/A
</content>
</thread_object>

</thread>
<thread>
creation date:
1730455309666

thread_type:
official_review

<thread_object>
creator: reviewer_18yl

<content>
summary:
This paper studies the critical influence of overparameterization on SAM.

strengths:
The experiments in this paper are insightful, showing that overparameterization can increase the performance gap between SAM and SGD, while also highlighting the role of factors such as label noise, sparsity, weight decay, and early stopping in this phenomenon.

weaknesses:
- It is widely recognized that overparameterization improves the generalization performance of SGD (e.g., ResNet-152 outperforms ResNet-18 on Cifar10 using SGD). And it is unsurprising that overparameterization can also improve the generalization performance of SAM. However, the non-trivial aspect is that overparameterization increases *the gap between SAM and SGD*, as shown in Figure 1. To avoid trivialization, reconsidering the title and abstract might be beneficial.

- My primary concern is that the theoretical analysis can not support the main claim/findings (e.g., Figure 1). 
  - The linear stability analysis in Section 6.1 can not demonstrate how overparameterization affects SAM (i.e., that “greater overparameterization leads SAM to find flatter minima”).
  - In the convergence analysis in Section 6.2: (i) it lacks relevance to generalization, the core focus of the paper; (ii) it does not clarify how the degree of overparameterization influences the convergence speed within the interpolation regime; (iii) in the interpolation regime, the exponential convergence result also holds for SGD (Bassily et al., 2018). The proof for SAM appears to be a minor extension from SGD, which treats SAM as SGD + small permutation.
  - The theoretical analysis in Section 6.1/6.2 focus on a variant of SAM (SAM without normalization) for simplification, rather than the original SAM. While not a major issue, the formulation should be clearly presented in Section 6 to prevent potential reader confusion.

questions:
Could the authors provide adequate theoretical support for the main findings? e.g., an analysis on diagonal linear networks.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730760254946

thread_type:
official_review

<thread_object>
creator: reviewer_bqqz

<content>
summary:
The authors perform experiments to 
measure the effect of overparameterization in SAM for a diverse set of tasks (Section 3). The goal is to observe how overparameterization
affects SAM under various conditions, e.g.,  label noise, sparsity, and regularization. 
Furthermore, they prove that stable minima of SAM are flatter and have more uniform Hessian moments (if compared with SGD), and stochastic SAM can also converge at a linear. The overall contribution is that they empirically and theoretically proved that overparameterization critically affects SAM.

strengths:
- interesting and well-motivated problem
 - very well written

weaknesses:
- discussion on higher moments of Hessian is missing

questions:
This is an interesting paper. I have a question: how does the convergence of higher-order moments of Hessian in your result compare with the other approaches in the literature, e.g., [1]? Can you provide a literature review on the previous works considering higher-order moments of Hessian to define flatness?

[1] Tahmasebi, Behrooz, et al. "A Universal Class of Sharpness-Aware Minimization Algorithms." Forty-first International Conference on Machine Learning.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper examines the impact of overparameterization on sharpness-aware minimization, focusing on factors such as label noise, sparsity, and regularization. The reviewers found the topic to be well-motivated and the experiments extensive. However, major concerns were raised regarding the empirical and theoretical grounding of key claims, which were at times perceived as overstated or predictable. Despite the significant discussions the paper has generated, the initial evaluations remained lukewarm. The authors are encouraged to incorporate the important feedback given by the knowledgeable reviewers.