When Do Flat Minima Optimizers Work?
Authors: Jean Kaddour
Linqing Liu
Ricardo Silva
Matt Kusner
Venue: NeurIPS 2022 Accept (2022)
Keywords: deep learning
optimization
flatness

Abstract: Recently, flat-minima optimizers, which seek to find parameters in low-loss neighborhoods, have been shown to improve a neural network's generalization performance over stochastic and adaptive gradient-based optimizers. Two methods have received significant attention due to their scalability: 1. Stochastic Weight Averaging (SWA), and 2. Sharpness-Aware Minimization (SAM). However, there has been limited investigation into their properties and no systematic benchmarking of them across different domains. We fill this gap here by comparing the loss surfaces of the models trained with each method and through broad benchmarking across computer vision, natural language processing, and graph representation learning tasks. We discover several surprising findings from these results, which we hope will help researchers further improve deep learning optimizers, and practitioners identify the right optimizer for their problem.

When Do Flat Minima Optimizers Work?
Jean Kaddour"
Linging Liu" Centre for Artificial Intelligence University College London
Centre for Artificial Intelligence University College London
Ricardo Silva
Matt J. Kusner
Department of Statistical Science University College London
Centre for Artificial Intelligence University College London
Abstract
Recently,fat-miinee flat-minima optimizers, which seek find parameters in low-loss neigh- borhoods, have been shown to improve a neural network's generalization perfor- mance over stochastic and adaptive gradient- based optimizers. Two methods have received significant attention due to their scalability: 1. Stochastic Weight Aver- aging (SWA), and 2. Sharpness-Ar Aware Minimization (SAM). However, there has been limited investigation into their properties and no systematic benchmarking of them across different domains. We fill this gap here by comparing the loss surfaces of the models trained with each method and through broad benchmarking across computer vision, natural language processing, and graph representation learning tasks. We discover several surprising findings from these results, which we hope will help researchers further improve deep learning optimizers, and practitioners identify the right optimizer for their problem.
1 Introduction
Stochastic gradient descent (SGD) methods are central to neural network optimization (6. Recently, class of algorithms has focused on biasing SGD methods towards so-called "flat" minima, which are located in large weight space regions with very similar low loss values [43]. Theoretical and empirical studies 21
55 49 12 postulate that such flatter regions generalize better than sharper minima, c.g., due to the flat minimizer's 's robustness against loss function shifts between train and test data, as illustrated Fig. Two popular flat-minima optimization approaches are: 1.
Stochastic Weight Averaging (SWA) [48] and 2. Charpness-Awart Minimization (SAM) 22
While both strategies aim find flatter minima, they operate much differently. On the one hand, SWA based on the intuition that, near a flat minimum, gradients are smaller, leaving many iterates that flat region. Therefore, averaging iterates will produce solution that pulled towards these flatter regions, see Fig. top. On the other hand, SAM minimizes the maximum loss around neighborhood of the current iterate. This way, a region around the iterate designed to have uniformly low loss: see Fig. [ bottom. Crucially, SAM requires an additional forward/backwaar pass for each parameter update, making it more expensive than SWA.
of SWA and SAM in some domains, we are unaware of systematic comparison between them that would help practitioners choose the right optimizer for their problem and researchers to develop better optimizers. The SWA [48] paper was published in 2018, and the SAM [22 paper in 2021; however, the SAM paper, and its most noticeable
Despite the successes 76
follow-ups [65 12 103 do not compare against SWA. Further, there is very limited overlap in
"Equal contribution, correspondence to fiea.kaddouur.inging...o
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

terms of the model architecture and dataset used in the experiments among both papers. which are likely further confounded by other differences in the training procedures (e.g. data augmentations. hyper-parameters, etc.).
Contributions
1. In-depth comparison of minima found by SWA and SAM: We visualize linear interpolations between differ- ent models and quantify the minimizers' flatnesses. This analysis yields 4 insights, e.g., despite SAM finding flatter solutions than SWA as quantified by Hessian eigenvalues, they can be close to sharp directions, a phenomenon that has been overlooked in the previous SAM literature. Averaging SAM iterates leads to the flattest among all minima.
SWA
C
SAM
Train
2. Rigorous comparison of SWA and SAM's performance over 42 tasks: We empirically compare the optimizers with rigorous model selection procedure on broad range of tasks across different domains (CV, NL., and GRL), model types (MIPs, CNNs, Transformers) and tasks (classifica- tion, self-supervised learning, open dommnnomain question answer- ing, natural language understanding, and node/graph/link property prediction). We discuss 9 findings, e.g., that both dataset and architecture impact their effectiveness, that for NLP tasks, SAM improves over SWA in most cases, and that the converse holds for GRL tasks. When flat-minima optimizers do not help, we notice clear discrepancies be- tween the shapes of loss and accuracy curves. To assist future work, we open-source the code for all pipelines and hyper-parameters to reproduce the results.
Test
C
θ
Figure 1: The mechanics behind SWA and SAM, whose solution is denoted by + and x. respec- tively. SWA produces a solution O that is pulled towards flatter re- gions, while SAM approximates sharpness within the parameters' neighborhood (arrows).
Background and Related Work
2.1 Stochastic Gradient Descent (SGD)
The classic optimization framework of machine learning is empirical risk minimization
C (
Σ (x; N
where IRd vector of parameters, fI1..... N& } is a training set of inputs xn E RD, and f(:;; 0) is loss function quantifying the performance of parameters
SGD samples minibatch S C (1,..., N} of size (S] < N from the training set and updates the parameters through
1
O: ng (0:), where (0)
Vi (0;
I+
B
SEB
for length specified by the learning rate.
2.2 Stochastic Weight Averaging (SWA)
The idea of averaging weights dates back accelerating the convergence speed of SGD [78. 51 SWA's motivation is based on the following observation about SGD's behavior when training neural networks: it often traverses regions of the weight space that correspond high-performing models but rarely reaches the central points of this optimal set. Averaging the parameter values over iterations moves the solution closer to the centroid of this space of points.
The SWA update rule is the cumulative moving average
OSWA
OSGD
SWA
(3)
1+1

---

Algorithm 1 Stochastic Weight Averaging [8
Algorithm 2 Sharpness-Aware Minimization [22
Input:
Loss function L. training budget in number of iterations training dataset D == - { mini-batch size [B], averaging start epoch E, averaging frequency v, (scheduled) learning rate 7/, initial weights 0g. for A + 1,...., b do
Input: Loss function L. training budget in
number of iterations b, training dataset D
U { mini-batch size [8], neighborhood
radius p. (scheduled) learning rate 7, initial
weights 0g.
for < .,..., do
Sample mini-batch B from D
Sample a mini-batch B from D Compute worst-case
Compute gradient t- V (0:)
perturbation
Update parameters 0--1 0, 1l
V
7 (TC(O)
if 2 E and mod (k; D) = then
0-11 = (O:WA. + 0SWA) (1+1)
Compute gradient g VL (OSAMM+ Update parameters 0 05AM 179 end for return @SAM
end if end for return S
where ( is the number of distinct parameters averaged far and & is the SGD iteration number
SWA has two paperpparamttes: the update frequency v and starting epoch E. When using constant
learning rate, Izmailov et al. [48 suggests updating the parameters once after each epoch, i.e., [ and starting at as 0.757, where the training budget required to train the model until convergence with conventional SGD training.
He et al. [39 argue that SWA may always improve generalization, regardless of the loss function's 's geometry. Kaddour [5]] show that averaging a specific range of weights can speed up training convergence. Cha et al. g argue that tuning and E carefully is necessary to make it work effectively in domain generalization (DG) tasks. Besides DG tasks, a list of tuned hyper-parameters based on a fair model selection procedure across different architectures and tasks has been missing in the literature. To the best of our knowledge, Cha et al. (8) is the only study that compares SWA and SAM over the same experiments, but it focuses domain generalization tasks which we, therefore, leave out in this work.
2.3 Sharpness-Awary Minimization (SAM)
While SWA is implicitly biased towards flat minima, SAM explicitly approximates the flatness around parameters 0 to guide the parameter update. It first computes the worst-case perturbation that maximizes the loss within a given neighborhood then minimizes the loss w.r.t. the perturbed weights 0 + e. Formally, SAM finds by solving the minimax problem:
min max L( + +),,
0 |ellašp
where p > 0 is a hyperparameter.
To find the worst-case perturbation E efficiently in practice, Foret et 22 approximates Eq. via a first-order Taylor expansion of L(e + e) w.r.t. around O, obtaining
VoL(O)
arg max T VoC(O) R [ellaSp
(5)
[Vo((0)
In words, is simply the scaled gradient of the loss function w.r. to the current parameters 6. Given é, the altered gradient used to update the current 0₂ (in place of g(o,)) is
Va max L(O + <) & VoL(O))o+e. [|c] S
Due to Eq. 5
SAM's computational overhead consists of an additional forward and backward pass per parameter update step compared to SWA and non-flat optimizers.
PSWA parameters constant between averaging steps.

---

SAM's performance strongly depends on the neighborhood radius p. For example, Chen et al.
Wu et [93] show that p should be set to values outside the originally considered ranges by Foret et al. [22] Analogously to Sec. 2.2 this lack of coherence among hyper-paramete tuning protocols the SAM literature makes tricky to determine SAM's comparative effectiveness.
[2].
2.4 Other Flat-Minima Optimizers
There are several extensions of SWA 36 g and SAM 66
103
101
For simplicity, we do not
consider them in this work. Besides SWA and SAM, other flar-minima optimizers include e.g., However, due to their computational cost and/or lack of performance gains, we do not include them in this work. Chaudhari al. [9] requires [5, 20] forward and backward passes per parameter update. Sankar et al. (84) similarly requires [5, 10] forward and backward passes to estimate the Hessian trace and 6 of experiments yield minimal improvement of S 0.27%, see Table I in Sankar et al. [84). In contrast, SWA and SAM have been shown to increase performance by multiple percentage points in while requiring fewer computational resources.
some cases g 12
3 How do minima found by SWA and SAM differ?
In this section, we investigate SWA and SAM solutions in two prototypical deep learning tasks, where these optimizers improve over the baseline. Our goal is to understand better their geometric properties (instead of their generalization performance, which is the focus of Sec.
First, we investigate the behavior of the loss landscape along the line between non-flat and flat solutions (Sec. (.T). Previous studies successfully used such linear interpolations to gain novel insights, c.g., for training dynamics [32 [25] regularization [69 28). and network pruning [26). Second, motivated by findings in Sec. 3.1 we average SAM iterates and visualize interpolations between averaged and non-averaged solutions (Sec. 3.2). Interestingly, the averaged SAM solution less susceptible to asymmetric directions. Third, we compare quantitative measurements of all solutions' flatnesses (Sec. (.3). Here, we compute dominant Hessian eigenvalues, as commonly used the flat minima literature 98 [2 [22]. Lastly, in Appendix [A. I we further compute CKA [6] and cosine similarities between SWA/SAM's network output logits.
We choose the following widely used for evaluation in flat-minima optimizer papers, and (ii) a novel, challenging Python code summarization task, on which state-of-the-ard models achieve only around 16% Fl score on the test set (which is higher than its commonly achieved accuracy on the more challenging training set), and that has not been explored yet in the flat-minima literature. Specifically, for (i), we investigate the loss/accuracy surfaces of a WideResNe128- 10 999 model on CIFAR- 100 (33 (baseline non-flat Graph Isomorphism Network [95] model on OGB-Codee Code2 |5 (baseline optimizer: Adam [56]).
disparate learning settings: (i) well-known image classification task,
optimizer: SGD with momentum (SGD-M)
83 For (ii), we use the theoretically-groumee
All optimizers start from the same initialization. We denote the minimizer produced by the non-flat methods (SGD-M and Adam) by @NF and the flat ones by gSWA and GSAM
3.1 What is between non-flat and flat solutions?
We start by comparing the similarity of flat and con-flat minimizers through linear interpolations. This analysis allows us to understand they are in the same basin and how close they are to a region of sharply-increasing loss, where we expect loss/accuracy to differ widely between train and test. Further, for each of our four observations, we recommend a future work direction.
To linearly interpolate between two sets of parameters 0 and 0', parameterize the line connecting these two by choosing a scalar parameter a and defining the weighted average ((a) = (1-a)0 o If there exists no high-loss barrier between two networks 0, O' along the linear interpolation, we say that they are located in the same basin, i.e., {0,60 0') E Ω. 775 102]. basin an area in the parameter space where the loss function has relatively low values. Due to NN non-linearities, the linear combination the weights of two accurate models does not necessarily define an accurate model. Hence, we generally expect high-loss barriers along the linear interpolation path.
While there are alternative distance measures that could used compare two networks, they typically either (a) do not offer clear interpretations, pointed out by Frankle et al. (26), (b) yield

---

trivial network connectivity results, such as non-linear low-loss paths, which can be found for any
two network minimizers 20 27 23
SWA
SAM
Loss
Accuracy/ /F1
Loss
Accuracy/ FF
GIN 0n Codt2 WIN on CIFAII0
2000
und
(8)
-00 00
-05
-04
80
6.20
C13
Figure 2: Training (blue) and test (red) losses (-) and accuracies (.... of linear interpolations
(aa) = (1 - a) + a0' (for E (,1) between SWA (+) and SAM (x) solutions (a 0.0) and non-flat flat baseline solutions (+, a = 1.0).
Obs. 1: (0SWA, ONF, E S2NF. @SWA and @NF are in the same basin, as can be seen in Figures 2 and De. Additionally, gNt is near the periphery of sharp increase in loss, can be seen when moving in the direction from ØSWA to g (i.e., a > 1). Conversely, OSWA finds flat regions that change slowly in the loss. This bias of SWA to flatter loss beneficially transfers to the accuracy landscape too: Figures 2p and 2 show the accuracy/F1 score rapidly dropping off approaching and beyond @Nr Interestingly, Figures and 2 see that for Code2, for < 0, there exist solutions with even better training loss/accuracy but worse test loss/accuracy. However, @SWA is close to the test accuracy maximizer along this interpolation. Future work may inspect why the cross entropy loss function used for GIN/Code2 seems less well correlated with its accuracy compared to WRN/CIFAR 100.
Obs. 2: SAAM E Ω * S2NP, 0SAM and NNF are not in the same basin: Figures 2 and 2 show that there is high loss barrier between them, respectively. Figures 2 and 2 show that 0 @SXM and even nearby points in parameter space achieve higher accuracies/FI scores (i.e., generalize better) than ONE and points around it. This is an interesting result because we expect different basins to produce qualitatively different predictions, exhibit different performances [46] 66]. Grewal & Bui 34 successfully combine models yielded by different optimizers, and we think future work should study ensembling SAM and non-SAM solutions.
of the motivations behind combining models, even if they
Obs. 3: SAM finds saddle point. Figure 2g shows GIN being located in sharp training loss minimum whose loss is much higher than 0 NT," Yet, its test loss is slightly higher, and its Fl score better. We visualize 2D plots moving along random directions (not shown here due space) to confirm that GIN is saddle point (Appendix A.2). p. A common pathology among curvature- based methods is that they attract saddle points [6]. Since SAM takes some form of curvature into account, too, we believe that future work should investigate SAM's propensity to find saddle points and potential remedies.
Obs. 4: gSAM is closer to sharper directions than OSWA, as can be seen by Lune Canc(0SAA)) (OSAM(0.))))) R 2 Cune (- while Cune (OSWA (0.1)) ze Lune LuneeeSnii (OSWA(- (-0.1)). where C(-)ure refers both training and test loss functions. A possible explanation for SAM being closer to sharp sides is that while it finds different basins than SGD/SWA by smoothing the loss surface (as illustrated in Fig. D. within a local basin, it may oscillate around the minimizer similarly as SGD. One cause for this can be that SAM's hypersphere is larger than SAM's radius p. If that holds, then given small enough learning rate, we expect it to oscillate around 6* E S2AAM (the smaller the learning rate, the less likely escapes the basin due to that stochasticity). Two possible remedies are: (1) adapt/schedule p. or (2) average SAM iterates to bias its solution towards the flatter side. (1) has been explored by We try (2) in the next subsection. Future work may study SAM's basin escape time, e.g..
103 [OI].
using convolutions (58) or stochastic differential equations [102].
5

---

3.2 What happens if we average SAM iterates?
Based on observation "QSAM is closer to sharper directions than gswA", averaging SAM iterates may further improve generalization, referred to as (WASAM). The reason that while SAM finds perter-performin basins, within the basin, its final iterate may still be near a side that increases sharply in the loss.
Sharpness- Minimization
WRN on CIFAR100
GIN on Code2
Loss
Accuracy
Loss
5003
(b
1003
Figure 3: Training (blue) / test (red) losses (-) accuracies (....) between non-flat baseline
SWA (+), SAM (x) ++ WASAM (+).
Starting with the first of the two previously analyzed settings (WRN/CIFARIO0), Figures Ba and 3p show that @SWA+SAM (marker: *) achieves the lowest test loss and highest test accuracy, respectively, WRN
What stands out in comparison to the previous plots is OWN 's (x) proximity to sharp sides, surpris-
ingly similar to WRN (+) here and in Figures 2 and 2e. As we hoped, OSWA+SAM is indeed closer to a flatter region, as can be seen by Lune (OWA-A SSMM (-0.)) Laore (OSWAASAM (0.2)).
In GIN/OGB- 2.GGe, one unanticipated finding is that @SWA+SAM escapes the (previously discussed) GIN
saddle point of O appearing here maximum in Figure 3p. A likely reason is that SAM traversed nearby flatter regions before arriving at the saddle point, especially if is a non-strict saddle. In terms of Fl score, Figure 30 shows that while GIN (+) and GIN perform about equally well, the @SWA improves over both.
flatter region found by GIN
3.3 How "flat" are the found minima?
We now quantify the flatnesses of all four optimizers over both tasks by computing the median of the dominant Hessian eigenvalue across all training set batches using the Power Iteration algorithm 74 (98. This metric measures the worst-case loss landscape curvature. We choose this metric as is
very commonly used in the minima flatness literature, e.g., [9] 12 22
62 85
Table shows that SAM leads to flatter minima than SWA in both cases. Interestingly Amax (RR) x (ONE
2.5 Amax ((SSWA gSAM)). while Amax WRN) 5.75. (OSWA+SAM), indicating room for im- WRN
provement in terms of flatness for both SWA and SAM. The relative differences are less dramatic for GIN/Code2, although surprisingly Amax (ONE- (IN) Amax GIN In sum, averaging SAM iterates leads to the flattest minima and best-performing minima in both cases (see Sec.
Table 1: Median Amax of Hessian over all training set batches. Task
Baseline
SWA
SAM
WASAM
WRN on CIFARI00
673
265
237
117
GIN on Code2
16.65
16.79
11.31
9.96
4 How do SWA and SAM perform on a broad set of experiments?
As we point out in the introduction, there is almost no overlap and consistency regarding reported SWA and SAM results in the literature. This section addresses this gap. For example, Bahri al. 44. Chen et al. 12 illustrate that the flat minima found by SAM improve generalization on Transformer [90 architectures compared to non-flat optimizers, but they do not compare against SWA. Hence, it is unclear if the computationally cheaper SWA may provide better or similar performance.
We compare flat minimizers SWA, SAM, and averaged SWA iterates (WASAM) over the non-flat minimizers across a range of different tasks the domains of computer vision, natural language processing, and graph representation learning. We average all runs at least three times across random

---

Table CV test results: Supervised Classification (SC), and Self-Supervised Learning (SSL) tasks.
Task
Model
Baseline
SWA
SAM
WASAM
WRN-28-10
16.7
1.05
01.34
+0.251...
96
+0,2220014
00.42
+0.41-0.02
CIFARIO
VIT-B-16
98, 98.95
-0.01100s
+0.07
+0.10
Mixcr-B-16
40.02
40.19
+0.22
WRN-28-10
10.93
1.020000
+ 1.82ac.14
+22.4_0.114
80.80
1.864000
2.333
+2.60-0.00
CIFARIO0
VTT-B-16
92.77
-0.12
+ 00.19.0.00
+0.13_o.07
Mixer-B-16
0.45
1..2
0.997
MoCo
89. 30.25.0.om
-0.03e0.1
-0.25+0.05
-0.17=0.10
SimCLR
88 20.0
-0.00
0.05
-0.1320.04
CIFARIO
SSL:
SimSiam
89.86 89.86.2.22
+ 0.
0.07
+ +0.11:0.0
90.32 24
-000
0.09
0.144.0.0
BYOL
0.70
+0.14jo.os
+ +0.21
SwaV
87.28
2.09
0.07.0.12
0.012.0.0
MoCo
81.74+0.18
0.97
0.91
+ 1.40::0100
SimCLR
63.28
2.955.0
0.1884624
+1.07=0.18
ImageNette
SSL:
SimSiam
61.77g0.14
+ +0.20
+ 0.
+ +0.181020
17.4910a6
+0.2010.1
10277
0.666.057
BYOL
84.16g0.4
+ +0.76
+0.15ge.5
+0.31go.1
SwaV
88.1610m1
+ +1.01_0a7
+0.081e.1
1..03
seeds (more often for experiments with higher variability, see details in Appendix (B) and we report the corresponding standard error. We bold the best pest-performing approach and any approach whose average performance plus standard error overlaps it.
Hyper-parameters. For all architectures and datasets, we set hyperparameters shared by all methods (e.g., learning rate) mostly to values cited in prior work [ As explained in Secs. 2.2 and 2.3] the effectiveness of flat-minima optimizers is highly sensitive to their additional hyper- parameters. We select hyper-parameters using grid search over held-out validation set. Specifically, for SWA we follow Izmailov et al. [48] and hold the update frequency v constant to once per epoch and tune the start time E' {0.57, 0.67,, I O. 757,, 0.9T") (T' is the number of baseline training epochs). Izmailov et al. [48] argue that cyclical learning rate starting from E helps to encourage exploration the basin. For the sake of simplicity, we average the iterates of the baseline directly but include even earlier starting times (i.e., 0.5T, 0.67). For SAM, we tune its neighborhood size p E {0.01, 0.02, 0.05, 0.1, 0.2), as in previous work [22
Appendix B contains the values of all hyyer-parametee and additional training details (including pub- lic model checkpoints, hardware infrastructure, software libraries, etc.) to ensure full reproducibility alongside open-sourcing our code.
4.1 Computer Vision
Supervised Classification (SC). We evaluate the CNN architectures WideResNets [99] with 28 layers and width 10, and PyramidNet (PN) with 110 layers and widening factor 272 (88 as well as Vision Transformer (VIT) [19] and MLP-Mixer [87] on CIFAR 10, 100} [63] All experiments use
basic data augmentations: horizontal flip, padding by four pixels, random crop, and cutout [7]
Self-Supervised Learning (SSL). We consider the following methods on CIFARIO and Ima- geNette") Momentum Contrast 41 a Simple framework for Contrastive Learning (SimCLR) [0) Simple Siamese representation learning (SimSiam) IT. Barlow Twins [T00.. Bootstrap your own Latent (B (BYO [35, and Swapping Assignments between multiple Views of the same image (SwAV) 77. All SSL methods use ResNet- 18 tions, we use k-nearest--eighhee
[40] as backbone network. To test the frozen representa- classification with memory bank (94]. We choose k = 200 and temperature T = 0.1 to reweight similarities. Compared to learning a linear model on top of the representations, this evaluation procedure is more robust to hyperparameter changes [59]
'Soeetmess with minor modifications, e.g., adjusting per-device batch sizes be compatible with our GPU infrastructure.

---

Figure 4: (a) NLP test results: Open-Domain Question Answering and Natural Language Understand- ing (GLUE) including paraphrase, sentiment analysis, and textual entailment. (b) GRL test results: Node Property Prediction (NPP), Graph Property Prediction (GPP), Link Property Prediction (LPP).
Task
Madel
Easelino
SWA
SAM
WASAM
1ask
Model
Boseline
SWA
SAM
WASAM
NQ
HD
10,30
-020
+0.45
NPP:
SAGIE
77.79
-0.17
-0.02
TrivisQ4
FiD
67.71
+ =0
+0.89
- -0.92
Protcin
DGCN
85. 55.42,017
-0.33.0.05
-0.14
-U.08
COLA
00.41 10a3
+000
+1.87,100
11.34
NPP:
SAGE
DGCN
78.02
73.88
+0.80
-0.44
+0.18
+000
.m.53
SST
RoBERTa
-0
0.11.
GPP:
+0.73.011
+0.00...
...55.0.15
MRPC
RobuRTa
50.34
+008asve
+0.70.0.4
a0.8
Colc2
GIN
15.7
+0.83
RoBERTa
50,40 40m
+0.00.00
+0.85
GPP:
GIN
28 10
...00.0.s
0038.014
0.33.0.15
QOP
Rolitiiaa
+0.01amm
-0 06
Molpcba
DOCN
25.05,031
.1.90.0.29
-015,
MNLI
RoBERTs
57.41.mm
+0.88
+0.30.am
40.35
LPP:
Bioks
CP
Comptex
849420m
BL0Ruace
0.07.0.0
0.14::.03
10001000
-002.am
+ 0.08....z
+ 0..12.0.0
ONLI
RoRERTs
92. a0
-008
+0.00
+0.1 11
LPP:
GCN
72.22..
-0.0..03
+ 11500..13
RTB
RoBERTA
10.00
10.70.006
0.46.002
Chuiion2
SAGE
81.95uass
+ +115
-0.11aa
+0.56
(a)
(h)
4.2 Natural Language Processing
We consider the task of open domain question answering (ODQA) using a T5-based model Fusion- In-Decoder (FiD)
44 We evaluate FiD-base on the test sets of Natural Questions (NQ) [64] and TriviaQA 150 We also consider natural language understanding tasks included in the GLUE benchmark 191, which cover acceptability, sentiment, paraphrase, similarity, and inference. We [72 for each task individually and report the results the GLUE dev set.
fine-tune RoBERTa-base
4.3 Graph Representation Learning
We use subset of the Open Graph Benchmark (OGB) datasets 45 The tasks are node property
prediction (NPP), graph property prediction (GPP), and link property prediction (LPP). For each task, we use two of the following GNN architectures and matrix factorization methods: GCN [57], DeeperGCN (DGCN) (68. SAGE [77 GIN 193, ComplEx [89]. and CP [66]. We use popular training schemes, such as virtual nodes, cluster sampling 14, or relation prediction as auxiliary training objective [13). The reported metries are ROC-AUC for Proteins, Accuracy for Products, Fl
score for Code2, Average precision for Molpcba, and Mean Reciprocal Rank for Biokg/Citation2.
4.4 Findings
We use G(.) to describe the generalization aeuneey/T//ROCAUU//PPMM of all optimizers fNon-flat baseline(NF), SWA, SAM, WASAM).
Datasets matter. For example, for node property prediction (Proteins), we see that no flat optimizer improves over the baseline optimizer, however, for (Products), flat-minima optimizers on the same architectures significantly improve over the baseline. We further explore the impac of different data augmentation strategies in Appendix A.77
2. Architectures matter, e.g., there is vast difference across model architectures for link property prediction on the Citation2 dataset: using GNN with GCN layers, SAM achieves statistically significant boost of > 1.30% and SWA slightly hurts the performance. When we replace the GCN layers with SAGE layers (and fix everything else), we see a boost of > 1.15% for SWA, while SAM hurts the performance by - 0.31%.
3. SWA underperforms on NLP tasks. SAM achieves the best performance in 7/10 experiments on NLP tasks, consistent with the findings of Bahri et al. 44, which show that SAM can boost performance across a wide range of NLP tasks. However, SWA never performs best, only improving the results in 1/10 cases, and even hurting the performance 4 tasks. Surprisingly, g (WWAAMM) > (SSAM) for (SST, QNLI) while SWA decreases performance in these cases.
4. SWA beats SAM on GRL tasks. G(SWA) > (NF) in 10/12 experiments, while (SAM)) > (NF) only in 4. We examine why SAM under-performs in Appendix AA.4
5. SWA does not work well with Transformers. SWA often does not improve and sometimes hurts performances, as can be seen the ViT results in Table 2 and NLP results in Fig. 4a In contrast,
SAM has some positive effects in these settings. We explore this further in Appendix A.3
8

---

6. SWA and SAM improve SSL task performance. This is non-trivial as the theoretical motivation behind finding flat minima is linked supervised learning losses 43
77 2T. Concurrently to
our work, Ramesh et al. 82 report that SAM helps for contrastive CLIP
79 models too.
Flat optimizers do not strictly improve over non-flat optimizers. The non-flat optimizer is nearly always the best for NPP Proteins and SSL methods on CIFARIO. We investigate the NPP Proteins solutions in the next subsection and recommend a more thorough investigation of the landscapes of SSL objectives for future work.
Flat-minima optimizers offer asymmetrie payoffs: at worst, they decreased performance by -0.3%, at best, they increased it by 2.60%.
9. Averaging SAM iterates often improves over SWA or SAM alone. ((WASAM) > mSWA)( (SAM)) in 39/42 cases. We hypothesize that asymmetric payoff's are the reason: when either SWA or SAM does not improve over the baseline (as discussed above), it does not hurt (much) either, hence WASAM more robust across all tasks.
4.5 Why do flat-minima optimizers fail?
Here, we audit one of the cases, where neither @SWA nor 0SAM improves over @NF (this happens in out of 42 cases): training GraphSAGE (37 model CCB-Proteiss: a protein-protein interaction graph where the goal is to predict the presence of protein functions (multi-label binary classification) 45
OSWA performs noticeably worse; OSAM performs about equally well.
Adam to SWA
Adam to SAM
Loss
ROC-AUC
Loss
ROC-AUC
on
623
(a)
(d)
6.20
Figure 5: GraphSAGE on OGB-Proteine Adam's (+) solution performs about equally well as SAM
(x), and better than SWA (+).
Fig. 5 shows two linear interpolations: between @ (ADAM) and (1) @SWA (Figures 5 and and 5p), and
(2) O (SAM (Figures 5c and (D). In contrast to success cases in Fig. 2 here: (a) for both SWA and SAM, the training loss miniizerr is very uncorrelated with the test loss minimizer; (b) SAM and
ADAM seem to be contained in the same test loss/accuracy basin. More analyses can be found the Appendix.
Limitations and Future Work
First, some of the fixed, shared hyperparameter values used from previous works may harm the effect of flat optimizers. The ideal experimental design includes tuning all hyperparameters independently for the non-flat optimizer, SWA, SAM, and WASAM. However, this forces the number of required runs to grow exponentially in unique hyperparameters and quickly renders this benchmark infeasible.
Second, despite our best efforts to evaluate the optimizers on a broad range of benchmark tasks, there are still plenty of unexplored domains; especially some of which are known to be sensitive careful optimization. For example, bi-level optimization problems (44] are common in generative modeling
31 42)
deep reinforcement learning 160 44 meta-learning [8 (52). causal machine learning
53
We are unaware of an investigation of flat minima optimization for such problems.
Third, in general, we believe fruitful directions of research include (a) optimizers that explicitly find basins where training loss flatness more directly corresponds higher hold-out accuracy, (b) post-processing methods for existing optimization runs to move into flatter regions of these basins
(2) (c) loss functions whose contours more tightly align with accuracy contours, (d) the study of flat-minima hyperparameter interactions (e.g., learning rate and neighborhood radius in SAM) (see Appendices A.5 and A.6 for first results), (e) analyses of flat minima optimization on convergence
speed [51].

---

Our benchmark results point to which tasks would most benefit from improving these future work directions: graph learning tasks would benefit from improvements in (a), as SAM never among the best-pes rest-performing method, and language tasks would benefit if (b) is improved, as SWA is never among the best performing method).
Conclusion
We investigated when flat minima optimizers work by conducting fair comparison of two popular flat- minima optimizers. We examined the behavior of SWA/SAM by analyzing their loss landscapes on two representative deep learning tasks. Our next step was evaluate their generalization performance on a broad and diverse set of tasks (in data, learning settings, and model architectures). Based on this benchmarking, Finally, when SWA/SAM did not improve over baselines, common assumptions seemed broken (i.e., train-to-test loss minimizers
identified findings, of which some directly guide future work directions.
correlated).
Acknowledgements
We are very grateful to Kilian Q. Weinberger and Gao Huang for initial discussions and intuition, Pontus Stenetorp for NLP experimental design advice, and Oscar Key for feedback on the draft. JK and LL acknowledge support by the Engineering and Physical Sciences Research Council with grant number EP/S021566/11. This research was supported through Azure resources provided by The Alan Turing Institute and credits awarded by Google Cloud.
References
[1] Aghajanyan, A., Shrivastava, A., Gupta, A., Goyal. N., Zettlemoyer, L., and Gupta, S. Better fine-tuning by reducing representational collapse. In International Conference on Learning Representations, 2020.
[2] Andriushchenko, M. and Flammarion, N. Towards understanding sharpness-aware minimiza- tion. In International Conference on Machine Learning, pp. 639-668. PMLR, 2022.
[3] Athiwaratkun, B., Finzi, M., Izmailov, P., and Wilson, A. G. There are many consistent explanations of unlabeled data: Why you should average, 2019.
[4] Bahri, D., Mobahi, H., and Tay, Y. Sharpness-aware minimization improves language model generalization, 2021.
[5] Bisla, D., Wang, J., and Choromanska, A. Low-pass filtering sgd for recovering flat optima in the deep learning optimization landscape, 2022. URL http://s//axiiv.or///bbbb 08025
[6] Bottou, I.., Curtis, F. F.,, and Nocedal, J. Optimization methods for large-scale machine learning. Siam Review, 60(2):223-311, 2018.
[7] Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., and Joulin, A. Unsupervised learning of visual features by contrasting cluster assignments. In Larochelle, H., Ranzato, M., Hadsell, R.. Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems 33: Annual Conference
Neural Information Processing Systems 2020, NeurIPS paper 2020/ hwt.//hashhaaaa 70f ebd2b99622999fe hwwh/70ffb6266erfffo..
2020, December 6- 2020, virtual, 2020. URL https:: wtw.://dd/....coccooooo...mo
nwwripe.rcccmm cc/
Abstraactbbbtttt.. html
[8] Cha, J.. Chun, S., Lee, K., Cho. H.-C., Park, S.. Lee, Y. and Park, S. SWAD: Domain generalization by seeking flat minima. In Beygelzimer. A., Dauphin. Y., Liang, P., and Vaughan, W. (eds.), Advances in Neural Information Processing Systems, 2021. URL https://openreviev wttp:ooumm/idd-2iii.. JYJUU
[9] Chaudhari, P., Choromanska, A., Soatto, S., LeCun, Y., Baldassi, C., Borgs, C., Chayes, J. T., Sagun, I.., and Zecchina, Entropy-sgd: Biasing gradient descent into wide valleys. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France,
10

---

April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https / //openreview mott/forrur
[10]
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. E. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Muchine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Leurning Reseurch, pp. 1597-1607. PMLR, 2020. http:httt::ppceeeeeeeee mlr. www.sprsssvvv.... http//v11/cccccc c0einn00chennn_ppc html
[11] Chen, X. and He, K. Exploring simple siamese representation learning. Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 15750-15758, 2021.
[12] Chen, X., Hsich, C.-J., and Gong, B. When vision transformers outperform resnets without pre-training or strong data augmentations, 2021.
[13] Chen, Y., Minervini, P., Riedel, S., and Stenetorp, P. Relation prediction as an auxiliary training objective for improving multi-relational graph representations. In 3rd Conference on Automated Knowledge Base Construction, 2021. URL /tt/httpss//peereieee.... www.w...ee.orrm/
1d=Qa3uS3H7-Le
[14] Chiang, W., Liu, X., Si, Li, Y., Bengio, S., and Hsieh, C. Cluster-gen: An efficient algorithm for training deep and large graph convolutional networks. In Teredesai, A., Kumar, V., Li, Y., Rosales, R., Terzi, E., and Karypis, G. (eds.), Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2019, Anchorage, AK. USA, August 4-8, 2019, pp. 257-266. ACM, 2019. doi: 10.1145/32925003330095. URL attps://don oooooro////1 10. 1144///222200033309900
[15] Cubuk, E. D., Zoph, B., Mane, D., Vasudevan, V., and Le. Q. V. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 113-123, 2019.
[16] Dauphin, Y. N., Pascanu, R., Gülgehre, Ç., Cho, K., Ganguli, S., and Bengio, Y. Iden- tifying and attacking the saddle point problem high-dimensional non-convex optimiza- tion. In Ghahramani, 7., Welling, M., Cortes, C.,, Lawrence, N. D., and Weinberger, K. Q. (eds.), Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pp. 2933-2941, 2014. URL httppe:/ neurips cc/ sccpappr//// 2014/hash/ 17 TTe2aa00eec604005555555506663333 hat.ae2044c.06bb-bbaattaa..t.tte html
/
[17] Devries, T. and Taylor, G. W. Improved regularization of convolutional neural networks with cutout. CoRR, abs/1708.044. 04052, 2017. URL http:://axxi/// org/ abs /1708 045522
[18] Dong, Z., Yao, Z., Gholami, A., Mahoney, M. W., and Keutzer, K. HAWQ: hessian aware quantization of neural networks with mixed- preecisin... In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 November 2, 2019, pp. 293-302 IEEE, 2019. doi: 10.1 N..119//CCC.22119000 10. 111111190000 2019.00038
Uttptttss:///oi. org
[19] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., De- hghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL httpttt//opeereeiiiii net Sirruuiddd-YicbNNTTT
[20] Draxler, F., Veschgini, K., Salmhofer, M., and Hamprecht, F. A. Essentially no barriers in neural network energy landscape. In Dy, J. and Krause, A. (eds.), Proceedings of the 35th International Conference on Machine Learning, ICMI. 2018, Stockholmsmässonn Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 1308- 1317. PMLR, 2018. http:tt::/// httt:::/pproceeiingggn http://inmmmii....e////////
mlr wttp:es///0/rraaleee1ea.. 2tt00//raxeee111ee html
[21] Dziugaite, G. K. and Roy, D. M. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. In Elidan, G., Kersting, K., and Ihler, A. T. (eds.), Proceedings of the Thirty-Thine Third Conference on Uncertainty
11

---

in Artificial Intelligence, UAI 2017, Sydney, Australia, August 11-15, 2017. AUAT Press, 2017. URL http://aa.gg../////////ooo/nnsrs
[22] Foret, P., Kleiner, A., Mobahi, H., and Neyshabur, B. Sharpness-aware minimization for efficiently improving generalization. In 9th International Conference on Learning Repre- sentations. ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https://openrevii
[23] Fort, S. and Jastrzebski, S. Large scale structure of neural network loss landscapes. In Wallach, H. M., Larochelle, H., Beygelzimer, A., d' Alché-Buc, F., Fox, E. B., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver BC, Canada, pp. 6706-67 14, 2019, URL htts://t://p www.eedddinnn...uiiis...... cc/ppeer////p
[24] Franceschi, L., Frasconi, P., Salzo, S., Grazzi, R., and Pontil, M. Bilevel programming for hyperparameter optimization and meta-learning. In International Conference Machine Learning, pp. 1568-1577. PMLR, 2018.
[25] Frankle, J. Revisiting "qualitatively characterizing neural network optimization problems". CoRR, abs/2012.068990 2020. URL https: www.ariwwv.....oo...arrrreeeaeee wtggabss/2122 06898
[26] Frankle, J., Dziugaite, G. K., Roy, D. M., and Carbin, M. Linear mode connectivity and the lottery ticket hypothesis. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 3259-3269. PMLR, 2020. BtLLtttp//pprceeiiiii mlr. .pres// html
[27] Garipov, T., Izmailov, P., Podoprikhin, D., Vetrov, D. P., and Wilson, A. G. Loss surfaces, mode connectivity, and fast ensembling of dnns. In Advances in Neural Information Processing Systems, 2018
[28] Geiping, J., Goldblum, M., Pope, P. E., Moeller, M., and Goldstein, T. Stochastic training is not necessary for generalization. CoRR, abs/2109. 14119, 2021. https:////aar org/ abs/ /209 14119
[29]
Ghorbani, B., Krishnan, S., and Xiao, Y. An investigation into neural net optimization via hessian eigenvalue density. In Chaudhuri, K. and Salakhutdinov, R. (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2232-22411 PMLR, 09-15 Jun 2019. URL https::
mlr. wtr.press/veee v97 www.:////vvg///obbbbabaaa http//hhobbaaiiii1 html
[30] Golub, G. H. and Welsch, J. H. Calculation of gauss quadrature rules. Mathematics of computation, 23(106):221-33,, 1969.
[31] Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial networks. Communications of the ACM, 63(1)): 139-144, 2020.
[32] Goodfellow, I. J. and Vinyals, O. Qualitatively characterizing neural network optimization problems. In Bengio, Y. and LeCun, (eds.), 3rd International Conference on Learning Rep- resentations, ICLR 2015, San Diego, CA. USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL Sttt:tt:/// //appiv////xvvv./// org. org/ abs/ 1412. 6544
[33] Gotmare, A., Keskar, N. S., Xiong, C., and Socher, R. A closer look at deep learning heuristics: Learning rate restarts, warmup and distillation. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL /ttp://////p www.openneevii.......... net meetfoormmme fitimmid=rr11410 witrriddrrr110@@g.
[34] Grewal, Y. and Bui, T. D. Diversity is all you need to improve bayesian model averaging. In Bayesian Deep Learning Workshop at Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, 2021.
12

---

[35] Grill, J., Strub, F., Altché, F., Tallec. C., Richemond, P. H., Buchatskaya, E., Doersch, C., Pires, B. A., Guo, Z., Azar, M. G., Piot, B., Kavukcuoglu, K., Munos, R., and Valko, M. Bootstrap your own latent- A new approach to self-supervised learning. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. htLhhttpss//rrcceddiia
neur neurips abstract.html
(apapper2022//110 httppp/prro000//oaoaaaaaaaaaaea0 (ttaaaa0000000001002011006666660
[36] Guo, H., Jin. J., and Liu, B. Stochastic weight averaging revisited. CoRR, abs/2201.00519, 2022. URL. htttps://arxtw.oorg/bbb///// wtt.:/ogg/abb./...oaeaea///
[37] Hamilton, W. L., Ying. R., and Leskovec, J. Inductive representation learning on large graphs. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 1025-1035, 2017.
[38] Han, D., Kim, J., and Kim, J. Deep pyramidal residual networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu. HI, USA, July 21-26, 2017, pp. 6307-6315 IEEE Computer Society, 2017. doi: 10.1 10.1109/CVPR.2017..600 URL
attps://doo org/oorgr//1 10. 1 119///PPP 2017. 668
[39] He, H., Huang, G., and Yuan, Y. Asymmetric valleys: Beyond sharp and flat local minima. In Wallach, H. M., Larochelle, H.. Beygelzimer, A., d' A'cch-B-Buu F., Fox, E. B., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NewrlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 2569-2560, 2019. UtL.hsttps::/proceeedii www.diinnnnnnnuuseeeeeeeee www.iir....... ct/ pecpppppr//019 44abbbaa299bb4444443774744662222
wwteerbbbbttttateoee html html
[40] He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Puttern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 770-778. IEEE Computer Society, 2016. doi: 10.1109/CVPR
20.6.90. URL. https:tps:// /doii org/10. 10. 2109/CVPP 2016.00169 6.90
[41] He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. B. Momentum contrast for unsupervised visual representation learning. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp. 9726-9735. Computer Vision Foundation / IEEE, 2020. doi: 10. 1. RBBp/C/PRR2200.220000009 sttpp://dol org/10. 1109/ CVPR42600..0.... 2020.00975
URL
[42] Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[43] Hochreiter, S. and Schmidhuber, J. Flat minima. Neural computation, 9(1::1442, 1997.
[44] Hong, M., Wai, H.-T., Wang, Z., and Yang, Z. two-timescale framework for bilevel optimiza- tion: Complexity analysis and application to actor-critic. arXiv preprint arXiv:2007. 05/ 170, 2020.
[45] Hu, W., Fey, M., Zitnik, M., Dong, Y., Ren, H., Liu, B., Catasta, M., and Leskovec, J. Open graph benchmark: Datasets for machine learning graphs. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL https: /ttp::/e////oooeiiingg. neurips. cc/ paper 2020//2000//00 http///booo111115bbe20
wow.ooo-ffdooooooo html
[46] Huang, G., Li, Y., Pleiss, G., Liu, Z., Hoperoft, J. E., and Weinberger, K. Q. Snapshot ensembles: Train 1, get M for free. In 5th International Conference on Learning Repre- sentations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL
[47] Izacard, G. and Grave, É. Leveraging passage retrieval with generative models for open domain question answering. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 874-880, 2021.
13

---

[48] Izmailov. P., Podoprikhin, D., Garipov, T., Vetrov, D. P., and Wilson, A. G. Averaging weights leads to wider optima and better generalization. In Globerson, A. and Silva. (eds.). Proceedings of the Thirty-Fourth Fourth Conference on Uncertainty in Artificial Intelligence, UAI 2018, Monterey, California, USA, August 6- 10, 2018, pp. 876-885 AUAI Press, 2018. URL http: http://aaii..r. .rg. www.oogg/ga...ooooooooo 2018/ http://ppoceeeecc//oeeaaoooo www.sgg//ppppps/paa.aaaeeee 313. pdf
[49] Jiang, Y., Neyshabur, B., Mobahi, H., Krishnan, D., and Bengio, S. Fantastic generalization measures and where to find them. In Sth International Conference Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https /poenrevier seaet/oorrmiiddSlgPPPP.
[50] Joshi, M., Choi, E., Weld, D. S., and Zettlemoyer, L. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp- 1601-1611, 2017.
[51] Kaddour, J. Stop wasting my time! saving days of imagenet and bert training with latest weight averaging. arXiv preprint arXiv:2209.1435 2022. URL 2209. 14981
org/bb//
[52] Kaddour, J., Saemundsson, S., and Deisenroth (he/him), M. Probabilistic active meta- learning- In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Ad- vances in Neural Information Processing Systems, volume 33. pp. 20813-20822. Curran Associates, Inc., 2020. URL https::/ http://ooooonnnnn..rr..i.peopeep nwwrine cc http:/aappr///// 202pr//220//flee/ effol779bbb4ee22a/41028666666666 Paper pdf
[53] Kaddour, J., Zhu, Y., Liu, Q., Kusner, M., and Silva, R. Causal effect inference for structured treatments. In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, W. (eds.), Advances in Neural Information Processing Systems, 2021. URL [tLpttss///pppeeeeiiee wttp:evwwwwwwwwefooeee net/forum? id=0v9EPJGc10
[54] Kaddour, Lynch, A., Liu, Q. Kusner, M. J., and Silva, R. Causal machine learning: A survey and open problems. arXiv preprint arXiv:2206. 012200.154777 15475, 2022. URL https://arxii org/abs/220 15475
[55] Keskar, N. S., Mudigere, D., Nocedal, J., Smelyanskiy, M., and Tang, P. T. P. On large- batch training for deep learning: Generalization gap and sharp minima. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL htttps://operreieee net/ forum?id= Berum?d-HHoyRRYye
[56] Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In Bengio, Y. and LeCun, Y. (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL attp://arxx org/ abs/ 11121111..00 6980
[57] Kipf. N. and Welling, M. Semi semi-supervised classification with graph convolutional networks. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL. https //openreview net.:oorummi---SS/aaaaaaa
[58] Kleinberg, B., Li, Y., and Yuan, Y. An alternative view: When does sgd escape local minima? In International Conference on Machine Learning, pp. 2698-2707. PMLR, 2018.
[59] Kolesnikov, A., Zhai, X., and Beyer, L. Revisiting self-supervised visual representation learn- ing. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 1920-1929, 2019.
[60] Konda, V. and Tsitsiklis, Actor-critic algorithms. Advances neural information processing systems, 12, 1999.
14

---

[61] Kornblith, S., Norouzi, M., Lee, H., and Hinton, G. E. Similarity of neural network repre- sentations revisited. In Chaudhuri, and Salakhutdinov, R. (eds.), Proceedings of the 36th International Conference on Machine Learning, ICMI. 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 3519-3529. PMLR, 2019. http:htttp//ppoceednng mlr press/ http///oobbiitthhheee html
[62] Krishnapriyan, A. S., Gholami, A., Zhe, S., Kirby, R. M., and Mahoney, M. W. Characterizing possible failure modes in physics-informed neural networks. In Ran- zato, M., Beygelzimer, A., Dauphin, Y. N., Liang, P., and Vaughan, J. W. (eds.), Ad- vances in Neural Information Processing Systems 34: Annual Conference on Neural In- formation Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp. ctppscc/pppeee
26548-26560, 2021. URL https: /tttp::/pproceedinggg www.aiiinnnnn.n.uuinee neurips. cc pappppp/p////aaa/s/eee TEE86600GF11000cccc11172222222 -tt.://22272551--saaattttt hhttl
[63] Krizhevsky, A. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
[64] Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452 466,, 2019.
[65] Kwon, J., Kim, J., Park, H., and Choi, I. K. Asam: Adaptive sharpness-aware minimization for scale-invariant learning of deep neural networks. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 5905-5914, PMLR, 18-24 Jul 2021. URL https: httpss//poceedings .ml.. prw.s / w139//www22bbbbb. html
[66] Lacroix, T., Usunier, N., and Obozinski, G. Canonical tensor decomposition for knowledge base completion. In Dy, J. G. and Krause, A. (eds.), Proceedings of the 35th International Conference on Machine Learning. ICML 2018, Stockholmssäässan Stockholm, Sweden, July 10-15, 2018. volume 80 of Proceedings of Machine Learning Research, pp. 2869 2878. PMLR, html
2018. URL RtLLhttp:// hattp://proceedingg.
mlr,
[67] Lakshminarayanan, B., Pritzel, A., and Blundell, C. Simple and scalable predictive uncertainty estimation using deep ensembles. In Guyon, I., von Luxburg. U., Ben- gio, S., Wallach, H. M., Fergus, R., Vishwanathan, S. V. N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neu- ral Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 6413,6413, 2017. URL hitps: httpp//peeingsseeeeeee www.ieeripp.. www.iiss..pppeeeeeee wwwpurrss.ccpppper//eeeeee wccppappr//017//aaeeea 05522444771222511177fff8aaaa8883
Abstract.
[68] Li, G., Xiong, C., Thabet, A., and Ghanem, B. Deepergen: All you need to train deeper gens, 2020.
[69] Li, H., Xu, Z., Taylor, G., Studer, C., and Goldstein, T. Visualizing the loss landscape of neural nets. In Bengio. S., Wallach, H. M., Larochelle, H., Grauman, K., Cesa- Bianchi, N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 6391-64011 2018. URL htt.htttss/pprrccedingg cc/ / /ttp//////................000000 htt.:///ee...oooo.o.o.oooo.aaaaa
neurips
[70] Li, 7., Cui, 7.,, Wu, S., Zhang, X., and Wang. I.. Fi-gnn: Modeling feature interactions via graph neural networks for ctr prediction. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pp. 539-548, 2019.
[71] Liu, Q., Nickel, M., and Kiela, D. Hyperbolic graph neural networks. Advances in Neural Information Processing Systems, 32, 2019.
[72] Liu, Y., Out, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L.,, and Stoyanov, V. Roberta: robustly optimized bert pretraining approach. arXiv preprint arXiv: 11692, 2019.
15

---

[73] Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. In International Con- ference on Learning Representations, 2019. httpattps://pprrriieeeeetttttoot
Bkg6RiCqY7
[74] Mises, R. and Pollaczek-Geinine Geiringer, H. Praktische verfahren der gleichungsauflösung. ZAMM- Journal of Applied Mathematics and Mechamics/Zeitschrffft für Angewandte Mathematik und Mechanik, 91)):88-77, 1929.
[75] Neyshabur, B., Sedghi, H., and Zhang, C. What is being transferred in trans- fer learning? In Larochelle, H., Ranzato, M.. Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems 33: Annual Confer- ence on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12. 2020, virtual. 2020. URL https://prooceednngss het.ssenurrs... cc/ htte//aper//ae/ happer/2220/haai// 0607f4c705595b911 2tt.0/999959911144444/2277227660 Abw.eaadtttttttttttttt.....
[76] Nikishin, E., Izmailov, Athiwaratkun, B., Podoprikhin, D., Garipov, T., Shvechikov, P., Vetrov, D., and Wilson, A. G. Improving stability in deep reinforcement learning with weight averaging. In Uncertainty in artificial intelligence workshop on uncertainty Deep learning. 2018.
[77] Petzka, H., Kamp. M., Adilova, L., Sminchisescu, C., and Boley, M. Relative flatness and generalization. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021. URL
[78] Polyak, B. T. and Juditsky, A. B. Acceleration of stochastic approximation by averaging. SIAM journal on control and optimizaaiion 30(4):838-855, 1992.
[79] Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A.. Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning transferable visual models from natural language supervision. In Meila, M. and Zhang. T. (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, PMLR, 2021.
volume 139 of Proceedings of Machine Learning Research, pp. mlr wwwpppreseeeeee hwtp://11/rrraaa222222 ord21a html
URL
[80] Rahman, M. K. Training sensitivity in graph isomorphism network. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pp. 2181-2184, 2020.
[81] Rajeswaran, A., Finn, C., Kakade, S. M., and Levine, S. Meta-learning with implicit gradients. Advances in neural information processing systems, 32, 2019.
[82] Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
[83] Rumelhart, D. E., Hinton, G. E., and Williams, R. J. Learning Representations by Back- Propagating Errors. pp. 696-699, MIT Press, Cambridge, MA, USA, 1988. ISBN 0262010976.
[84] Sankar, A. R., Khasbage, Y., Vigneswaran, R., and Balasubramanian, N. A deeper look at the hessian eigenspectrum of deep neural networks and its applications to regularization. arXiv preprint arXiv:2012.03880 2020.
[85] Stutz, D., Hein, M., and Schiele, B. Relating adversarially robust generalization to flat minima. In 2021 IEEE/CVF International Conference Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pp. 7787-7797, IEEE, 2021. doi: 10. 109/CCC/984922...01... 00771. URL https://:///ot org/ 10.
2021. 2021.00771 00771
[86] Susmelj, I., Heller, M., Wirth, P., Prescott, J., and et al., M. E. Lightly. GütHub. Note: 2020.
http://gghuub.coom/iiggtyyye
[87]
Tolstikhin, I. O., Houlsby, N., Kolesnikov, A., Beyer, L., Zhai, X., Unterthiner, T., Yung, J., Steiner, A., Keysers, D., Uszkoreit, J., Lucic, M., and Dosovitskiy, A. Mlp-mixer: An all-mlp architecture for vision. CoRR, abs/2105.0166. 2021. URIhtttpa http:://axxvvoo../. abs/ 22050 01601
16

---

[88] Touvron, H., Cord, M., Sablayrolles, A., Synnaeve, G., and Jégou, H. Going deeper with image transformers. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pp. 32 42.42.1 IEFF, 2021, doi: 10. 1 109/ 0WW.44992222210000112 URL https: //dp:. org/ 10. 1w111110///I99222 200.::2222221000011100
[89] Trouillon, T., Welbi, J., Riedel, S., Gaussier. É., and Bouchard, G. Complex embeddings for simple link prediction. In Balcan, M. and Weinberger. K. Q. (eds.), Proceedings of the 33nd International Conference
Machine Learning, ICML. 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pp. 2071-2080. JMLR.org, 2016. htLptttp: http:tttpprocccciinngg mlr www.pppwsssseeeeee http://///////oooooononoo hhmml
[90] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L.., Gomez, A. N., Kaiser, L.,, and Polosukhin, I. Attention is all you need. In Advances in neural information processing systems, pp. 5998 6008, 2017.
[91] Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 353-333 2018.
[92] Wolf, T., Chaumond, J., Debut, L.,, Sanh, V., Delangue, C., Moi, A., Cistac, P., Funtowicz, M., Davison, J., Shleifer, S., et al. Transformers: State-of-the-ar natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38-45, 2020.
[93] Wu, Y., Bojehevski, A., and Huang. H. Adversarial weight perturbation improves gener- alization in graph neural networks, 2022. URL hUr6K4D9F7P
[94] Wu, Z., Xiong, Y., Yu, S. X., and Lin, D. Unsupervised feature learning via non- parametric instance discrimination. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. 3733- 3742. Computer Vision Foundation IEEE Computer Society, 2018. doi: 10. 1 109/CVPR. 2018.00393. URL /tpp:///peee thecvf cww/ wwweeeettcvrreereeeeeeee Vmssuervisod_FFaaueee__eaam.aaa 201________11_pppppp_pp___ html
[95] Xu, K., Hu, W., Leskovec, J., and Jegelka, S. How powerful are graph neural networks? In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA. USA, May 6-9, 2019. OpenReview.net, 2019. URL httptt/t////oeeeeeeeee wwwp:/ewwwwww.wweeeeeeeeeeeeeeee wwt.:eeuntffiiiooo ryGs61A5Km
[96] Yang, Y., Hodgkinson. L., Theisen, R., Zou, J., Gonzalez, J. E., Ramchandran, K., and Ma- honey, M. W. Taxonomizing local versus global structure in neural network loss landscapes. In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neu- ral Information Processing Systems, 2021. URL htttps://penreviiee net/ wwt.:eeeetoooaeeeeee /tfomoormmi id= P6bUrLREcne]
[97] Yao, Z., Gholami, A., Keutzer, K., and Mahoney, M. LARGE BATCH SIZE TRAINING OF NEURAL NETWORKS WITH ADVERSARIAL TRAINING AND SECOND-ORDER INFORMATION, 2019. URL https:// net. formmrommmoooooooo H1
[98] Yao, Z., Gholami, A., Keutzer, K., and Mahoney, M. W. Pyhessian: Neural networks through the lens of the hessian. In Wu, X., Jermaine, C., Xiong, L., Hu, X., Kotevska, O., Lu, S., Xu, W., Aluru, S., Zhai, C., Al-Masri, E., Chen, Z., and Saltz, J. (eds.), 2020 IEEE International Conference on Big Data (IEEE BigData 2020), Atlania, GA, USA, December 10-13, 2020, pp. 581-590. IEFE, 2020. doi: 10. 109//9999 2tt.555022222220093778881 https:///oi..... org/ 1109/ BigData50022. 2020 9378171
URL
[99] Zagoruyko, S. and Komodakis, N. Wide residual networks. In Wilson, R. C., Hancock, E. R., and Smith, W. A. P. (eds.), Proceedings of the British Muchine Vision Conference 2016, BMVC 2016, York, UK, September 19-22, 2016. BMVA Press, 2016. URL http: html
//www. bmva. http://www//211//ppper//p///pppp
17

---

[100] Zbontar, J., Jing, L., Misra, I,, LeCun, Y., and Deny, S. Barlow twins: Self-supervised learning via redundancy reduction. Meila, M. and Zhang, (eds.), Proceedings of the 38th International Conference on Machine Learning, ICMI. 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 12310-12320, PMIR,, 2021. URIhhttp. httt::/pproceedaaie
mlr. www.mmmrr...///..mt..m..
wwwppe/ssssseeeppeeppe v139 2wta3//2oonaar211 html
[101] Zhao, Y., Zhang, H., and Hu, X. Ss-sam: Stochastic scheduled sharpness aware minimization for efficiently training deep neural networks. arXiv preprint arXiv:2203.0990. 2022.
share
[102] Zhou, P., Feng, J., Ma. C., Xiong, C., Hoi, C. H., et al. Towards theoretically understanding why sgd generalizes better than adam in deep learning. Advances in Neural Information Processing Systems, 33:21285 21296, 2020.
[103] Zhuang, J., Gong, B., Yuan, L., Cui, Y. Adam, H., Dvomek, N. C.,, sekhar tatikonda, Duncan,
J. and Liu, T. Surrogate gap minimization improves sharpness-aware training. In International Conference on Learning Representations, 2022. URL htttt://pppeerrereeii wtwp:/rrwwiiie.eeefruuuuu net/forum? id=edOMMAnhLa-
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Sec. 5
(c) Did you discuss any potential negative societal impacts of your work? [No] We consider this work to be an investigation into optimization algorithms that are central to modern ML systems. As these systems can be used in vastly different ways, we believe that it intractable to isolate the societal impact of novel insights into optimization.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] Supplemental material
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See Sec. 4
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] See Appendix.
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
18

---

(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
19


Reviews:
[{"review_id": "v_U-ONp-PI4", "reviewer": "Reviewer_J6KH", "summary": "This paper compares two well-known flat-minima-seeking methods, stochastic weight averaging (SWA) and sharpness aware minimization (SAM). Based on the geometric loss landscape analysis, the paper suggests combining SWA and SAM. Through experiments conducted on a wide range of tasks, the paper attempts to address when and why each method fails. ", "questions": "- Do SWA solutions and non-flat baseline solutions share optimization trajectories? If so, observation 1 seems to be a naïve consequence.\n- In Section 3, is Adam optimizer used for SWA and SAM in GIN/Code2 task? Or, only for non-flat solutions?\n- Regarding line 296, I cannot find an explanation in the Appendix as to why SWA and transformers do not work well together.\n", "limitations": "The paper provided limitations along with future research directions in Section 5. \n", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The paper tackles a significant and practically-relevant problem supported by a fair amount of experiments conducted across various tasks and domains. There is a need for comparative studies of flat-minima-seeking algorithms.\n- Overall, the paper is well-written and easy to follow.\n\n**Weaknesses**\n\n*(W1) Robustness to various settings:*\n- In Section 3, it is unclear whether the findings in Figure 2 can be generalized to other settings. For instance, is the result robust to varying hyperparameters (e.g. SAM's radius) or even to different random seeds? Considering that this is an analysis paper, the robustness of the findings should be highlighted.\n- Although the authors stated the scope of the work in Section 2.4, it is questionable whether the results would hold under variants of SWA and SAM (e.g., Adaptive SAM, Fast Geometric Ensemble).\n\n*(W2) Credibility of linear interpolation analysis:*\n- Regarding observations 3 and 4 in Section 3.1, one may consider asymmetric valleys [1]. While SAM solutions are closer to the sharper basin in certain directions, it is not guaranteed to be true in all directions. \n- I cannot find a 2D plot in the Appendix, which was mentioned in line 168. \n- It remains unclear how SAM+SWA solutions escape the saddle point in Fig 3.\n\n*(W3) Novelty of proposed algorithm:*\n- The proposed way of mixing SWA and SAM outperforms baselines in half of the proposed cases. In this regard, it lacks novelty. \n- Further, the motivation, i.e., observation 4, needs to be enhanced in line with the previous comment (W2). For example, I wonder if dominant Hessian eigenvalue analysis would hold under different settings since it is well-known that such analysis is not translation invariant w.r.t. neural network parameters.\n\n*(W4) Lack of interpretation:*\n- While the findings in Section 4.4 are interesting, it is mostly devoted to when SWA and SAM fail, not why. In order to conduct future research based on the paper’s observations, well-reasoned interpretations or hypotheses are crucial. \n- As in A.2, a study concerning global flatness and well-connectedness or other distance metrics (e.g., CKA) would be worthwhile [2]. \n\n**References**\\\n[1] He, Haowei, Gao Huang, and Yang Yuan. \"Asymmetric valleys: Beyond sharp and flat local minima.\" Advances in neural information processing systems 32 (2019). \\\n[2] Yang, Yaoqing, et al. \"Taxonomizing local versus global structure in neural network loss landscapes.\" Advances in Neural Information Processing Systems 34 (2021): 18722-18733.\n\n------------------------------------------------\n(Post-rebuttal) I have updated my score to 5. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "rZ96QJi7Ww", "reviewer": "Reviewer_ah85", "summary": "The authors perform a detailed comparison of stochastic weight averaging (SWA) and sharpness-aware minimization (SAM), first with loss landscape analysis and then with a broad set of benchmarks. The authors find several surprising results, and in particular show that combining the two methods, it is often possible to achieve improvements over both of them, leading to consistently good performance.", "questions": "**Q1**. Could you please comment on the base optimizer used in each data domain (Vision, NLP, GRL)? Could the SWA poor performance on NLP tasks compared to Vision be explained by the different choice of base optimizer?\n\n**Q2**. Could you pleas comment on W1? How is  $\\theta^{NF}$ computed for both SWA and SAM? What is $\\|\\theta^{NF} - \\theta^{SWA}\\|$ and $\\|\\theta^{NF} - \\theta^{SAM}\\|$ in Fig. 2?\n\n**Q3**. Could you pleas comment on W3? Why did you choose not to use a cyclical or constant learning rate schedule for SWA? Would it be possible to evaluate SWA with a constant learning rate schedule after epoch $E$ for a subset of your benchmarks to understand whether the choice of the learning rate schedule affected your results?", "limitations": "The authors adequately discuss the limitations of their work.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**S** = strength, **W** = weakness, **Q** = question.\n\n**S1**. The authors perform a detailed benchmark study that was missing from the literature. It is interesting to see that SAM and SWA are in fact quite complimentary, without one method clearly dominating the other.\n\n**S2**. The benchmarks considered by the authors are quite diverse, including multiple domains. Moreover, the results differ between the different domains, showing the importance of a diverse benchmark.\n\n**S3**. The combination SWA+SAM seems to be quite useful for practitioners potentially, as it works consistently well across a  broad set of benchmarks, while SWA and SAM are somewhat less consistent.\n\n**S3**. The authors perform loss surface analysis experiments to provide intuition behind the results.\n\n**W1**. In section 3.1 it is not clear how exactly the solutions $\\theta^{NF}$ are obtained for each optimizer. My guess is that for SWA the authors used the same optimization trajectory but without averaging to obtain  $\\theta^{NF}$, while for SAM they used an independent run? In this case, the relation between $(\\theta^{NF}, \\theta^{SWA})$ and the relation between $(\\theta^{NF}, \\theta^{SAM})$ will be quite different. In particular, I would expect the distance spanned by the horizontal access in Figure 2 to be much larger for SAM compared to SWA, making the interpretation of the plots more nuanced.\n\n**W2**. In lines 226-227 the authors mention that for SWA they simply average the iterates of the baseline optimization run, without modifying the learning rate schedule. While this is a valid approach, the authors of SWA emphasize the importance of using a non-standard learning rate schedule (see e.g. [the official blogpost](https://pytorch.org/blog/pytorch-1.6-now-includes-stochastic-weight-averaging/#is-this-just-averaged-sgd)). The deviation from the standard SWA recipy makes the benchmarking results a bit harder to interpret.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "kte0H1UOkKY", "reviewer": "Reviewer_ssYQ", "summary": "This paper analyses the properties of two flat minima optimizers, Stochastic Weight Averaging (SWA) and Sharpness-Aware Minimization (SAM). The nature of the minima reached by the two methods is studied through linear interpolations and dominant Hessian eigenvalues. SWA and SAM are benchmarked across a wide range of tasks in Natural language understanding, Graph Representation Learning, and Image Classification. Based on these results, a set of observations about SWA and SAM are described.", "questions": "1. Including observations relating to the robustness of the optimizer to its hyperparameters would also add practical value to the work. The role of the hyperparameters of the flat-optimizer in reaching certain minima can also be studied. For example, Does a large $\\rho$ lead to a completely different basin than a small $\\rho$?\n2. It would be clearer if the authors mentioned the tasks/models where a pretrained model is used. For example, it is mentioned in the Appendix that the CIFAR-10 ViT results used a pretrained ImageNet ViT. The characteristics of the flat optimizers might be very different if a model is trained from scratch with the flat optimizer instead of being finetuned with the flat optimizer.\n3. It is mentioned that one of the reasons why flat optimizers fail is because the train and test minimizers are not correlated. Can this hypothesis be more thoroughly evaluated by plotting the linear interpolation plots on an OOD test set?. For example, for the CIFAR-10 dataset, one could evaluate on CIFAR-10.1/ CINIC-10 dataset.\n4. Question 3 (d) of the checklist has been answered: \"Yes, See Appendix.\" But the details regarding the compute are not available in the Appendix.\n5. Why does Algorithm 1 returns $\\theta^{SWA+SAM}$? Should it not return  $\\theta^{SWA}$", "limitations": "The limitations of the work have been described in detail.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The analysis of the minima found by SAM and SWA has not been studied earlier. The observations are quite interesting and valuable to the community.\n2. The authors perform a detailed comparison of SAM and SWA across a wide range of benchmarks. All the results are reported with error bars.\n3. Overall, the paper is well written, with novel observations clearly stated and explained with supportive evidence.\n\nWeaknesses:\n1. This work has not studied the role of the base optimizer. The authors have compared SAM and SWA across multiple tasks and architectures. However, some methods use SGD, while others use Adam as the base optimizer. The loss landscape and generalization have been compared with SGD and Adam [1]. The comparison of SGD vs. Adam for the same task with SWA and SAM, respectively, would help disentangle the impact of the base optimizer.\n\n2. Regarding Observation 3 (L166): \"We visualize 2D plots moving along random directions (not shown here due to space)\". This plot must be added to the Appendix along with the observations. Otherwise, the saddle point hypothesis is not backed up with evidence. Additionally, it would be good to plot the Eigen Spectral Density [2] and show the presence of significant negative eigenvalues, which can confirm the presence of saddle point.\n\n[1] Zhou, P., Feng, J., Ma, C., Xiong, C., Hoi, S. C. H., et al. Towards theoretically understanding why sgd generalizes better than adam in deep learning. Advances in Neural Information Processing Systems, 33:21285–21296, 2020.\n\n[2] Ghorbani, Behrooz, Shankar Krishnan, and Ying Xiao. \"An investigation into neural net optimization via hessian eigenvalue density.\" International Conference on Machine Learning. PMLR, 2019.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "XxEezBwjHX", "reviewer": "Reviewer_Twme", "summary": "This paper focuses on two famous flat-minima optimizers, SAM and SWA. They provide lots of experiment results to analyze the solution of these two optimizers and find that they focus on different aspects. Finally, they try to combine them to get better performance than SAM or SWA and conduct some experiments in a variety of tasks to help us better understand these two algorithms.", "questions": "1. In the experiments, whether the distributed setting is used. The main reason is that we find this setting is very important for SAM, which may also affect the analysis of SAM.\n\n2. We know that data augmentation can improve the generalization of neural networks. When using data augmentation, the improvement of SAM will be reduced. What is the relationship between data augmentation, SAM, and SWA.\n\n", "limitations": "The paper provides some experiment analysis to illustrate the properties of SAM and SWA. \n\n1. The contribution of this paper. Combining SAM and SWA is not novel for me.\n\n2. The performance of SWA+SAM is not very great for me. In some tasks, the performance are very close between SAM+SWA and SAM/SWA and in some experiments SAM+SWA will hurt the performance.\n\n3. The experiment results of ViT training on ImageNet. I think this experiment is very important for SAM since SAM can achieve about 5% improvement on ImageNet.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. This paper provides lots of experiment results in various tasks to analyze SAM and SWA, which help us better understand them.\n\n2. The proposed method is very clear and simple to reproduce.\n\n3. SWA+SAM can achieve better performance in some tasks.\n\n4. The figures and writing are very easy to understand.\n\nWeakness:\n\n1. Although the paper provides lots of experiment results, some important experiments are not included in this paper. For example, some related work illustrate that SAM can achieve a huge improvement in ViT training on ImageNet (5% improvement). I think this is very important for people to better understand the relationship between SAM and SWA. \n\n2. Some comparison with current state of the art methods, such as GSAM.\n\n3. The accuracy of SAM+SWA is not better than vanilla SAM in some experiments. In addition, in some tasks the performance are very close. I think that will also limit the contribution of this paper.\n\n4. About the novelty of this paper. Although the paper focus on a very important problem, but the proposed method try to combine current two methods, SAM and SWA, which will also weak the contribution of this paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept