Lookaround Optimizer: $k$ steps around, 1 step average
Authors: Jiangtao Zhang
Shunyu Liu
Jie Song
Tongtian Zhu
Zhengqi Xu
Mingli Song
Venue: NeurIPS 2023 poster (2023)
Keywords: Deep Learning
Computer Vision
Mode Connectivity
Weight Average

Abstract: Weight Average (WA) is an active research topic due to its simplicity in ensembling deep networks and the effectiveness in promoting generalization. Existing weight average approaches, however, are often carried out along only one training trajectory in a post-hoc manner (i.e., the weights are averaged after the entire training process is finished), which significantly degrades the diversity between networks and thus impairs the effectiveness. In this paper, inspired by weight average, we propose Lookaround, a straightforward yet effective SGD-based optimizer leading to flatter minima with better generalization. Specifically, Lookaround iterates two steps during the whole training period: the around step and the average step. In each iteration, 1) the around step starts from a common point and trains multiple networks simultaneously, each on transformed data by a different data augmentation, and 2) the average step averages these trained networks to get the averaged network, which serves as the starting point for the next iteration. The around step improves the functionality diversity while the average step guarantees the weight locality of these networks during the whole training, which is essential for WA to work. We theoretically explain the superiority of Lookaround by convergence analysis, and make extensive experiments to evaluate Lookaround on popular benchmarks including CIFAR and ImageNet with both CNNs and ViTs, demonstrating clear superiority over state-of-the-arts. Our code is available at https://github.com/Ardcy/Lookaround.

Lookaround Optimizer: k steps around, 1 step average
Jiangtao Zhang', Shunyu L Song'.t, Tongtian Zhu', Zhengqi Xu', Mingli Song 'Zhejiang University, "Hanzzhou City University
{zhjgtao, liushunyu, sjie, raiden, xuzhengqi, brooksong)0zj edu.
Abstract
Weight Average (WA) is an active research topic due to its simplicity in ensembling deep networks and the effectiveness in promoting generalization. Existing weight average approaches, however, are often carried out along only one training trajec- tory in a post-hoc manner (i.e., the weights are averaged after entire training process is finished), which significantly degrades the diversity between networks and thus impairs the effectiveness. this paper, inspired by weight average, we propose Lookaround, straightforward yet effective SGD-based optimizer leading to flatter minima with better generalization. Specifically, Lookaround iterates two steps during the whole training period: the around step and the average step. In each iteration, 1) the around step starts from a common point and trains multiple networks simultaneously, each transformed data by different data augmenta- tion, and 2) the average step averages these trained networks to get the averaged network, which serves the starting point for the next iteration. The around step improves the functionality diversity while the average step guarantees the weight locality of these networks during the whole training, which essential for WA to work. We theoretically explain the superiority of Lookaround by con- vergence analysis, and make extensive experiments evaluate Lookaround on popular benchmarks including CIFAR and ImageNet with both CNNs and ViTs, demonstrating clear superiority over sate-of-the-and Our code available bttpp:///iituut wwt.aa...oom/aaaaa woom///rrcy/Looarouud
Introduction
Recent research on the geometry of loss landscapes in deep neural networks has demonstrated Linear Mode Connectivity (LMC): two neural networks, trained similarly on the same data starting from some common initialization, are linearly connected to each other across a path with near-constant loss [33 9. LMC reveals that neural network loss minima are not isolated points the parameter space, but essentially forms connected manifold g 35 It has recently been attracting increasing attention from research communities, attributed to its great potential into motivating new tools to more efficiently reuse trained networks. A simple but effective strategy is Weight Average (WA), which directly averages the network weights of multiple trained [46] or sampled along the training trajectories for flatter minima and thus better generalization.
[9 networks
WA has become popular approach in various fields for its efficiency in parameters and effectiveness in performance. For example, Izmailov et al. [19] show that Stochastic Weight Averaging (SWA) of multiple points along the trajectory of SGD, with cyclical constant learning rate, leads to better generalization than conventional training. Following that, Cha et al. 3 propose Stochastic Weight Averuging Densely (SWAD), i.e., averaging weights for every iteration in a region low validation loss to capture flatter minima and thus achieve superior out-of-doman Recently, Wortsman et al. [46) propose averaging weights of multiple fine-tuned models with various hyperparameters, coined Model Soups, to improve accuracy without increasing inference time.
generalizability.
Corresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Generally speaking, WA guides the model points around the loss basin into the interior of the loss basin to find flatter solution, which is shown to approximate ensembling the predictions [19].
3.50
3.50
Ø
e
ginit
3.35
3.35
moaan
Albeit striking results achieved in some cases, WA easily breaks down owing the diversity- locality conflicts. In one end, when WA is car- ried out within only one optimization trajectory after the training converges (as SWA does), the function diversity of the sampled checkpoints whose weights are averaged is limited, impair- ing the benefits of WA ensembling. independently, WA can result in completely invalid model due to the large barrier between different local minima (violating the locality principle that solutions should be in the same loss basin and closed to each other in parameter space). Prior WA approaches are either limited in function diversity or easily violating the locality principle, hindering its effectiveness and application in practice.
3.20
3.20
0
@²
8
3.05
Figure 1: Test set loss landscape. (Left) Around step for diversity. (Right) Average step for locality.
the other end, if WA is conducted between models trained
In this work, we propose Lookaround, a straightforward yet effective SGD-based optimizer to balance the both. Unlike prior works where WA often carried out along only one training trajectory post-hoc manner, Lookaround adopts an iterative weight average strategy throughout the whole training period constantly make the diversity-localtty compromise. Specifically, each iteration, I Lookaround consists of two steps: the around step and the average step, as shown in Figure [ The around step starts from a common point and trains multiple networks simultaneously. each on transformed data by different data augmentation, resulting in higher function diversity between trained networks. The average step averages these trained networks to get the averaged network, which serves as the starting point for the next iteration, guarantees the weight locality of these networks during the whole training. Lookaround iteratively repeats the two steps constantly, leading the averaged model to flatter minima and thus enjoying better generalization. We theoretically explain the superiority of Lookaround by expected risk and convergence analysis, and make extensive experiments to evaluate Lookaround popular benchmarks including CIFAR and ImageNet with both CNNs and ViTs, demonstrating clear superiority over state-of-th -fthe-of-the-arts.
The contributions of this work can be summarized as follows.
We propose Lookaround, a SGD-based optimizer that enjoys strategy. To seek flat minima within the loss basins, Lookaround iteratively averages the trained networks starting from common checkpoint with various data augmentation.
exploration
We theoretically analyze the convergence of different competitive optimizers in the quadratic case, and prove that Lookaround show lower limiting expected risk and faster convergence.
Extensive experiments conducted on the CIFAR and ImageNet benchmarks demonstrate that various CNNs and ViTs equipped with the proposed Lookahead optimizer can yield results superior to the state-of-the-ar counterparts, especially the multi-mon motell ensemble methods.
2 Related Work
Mode Connectivity. Mode connectivity is intriguing phenomenon in deep learning where independently trained neural networks can have their minima connected by low-loss path on the energy loss landscape 212 [4]. Some early works [12 g find that different independently trained networks can be connected by curves or multiple polylines in low-loss basins. Based on this, some later works 35 I show that when training from pretrained weights, the model stays in the same basin of the loss landscape, and different instances of such models can be connected by a straight line. Such findings motivates weight averaging, which averages the weights of different models obtain higher performance. Recently, weight averaging has been widely used in the fields of neural language processing (21 32 39] and multi-objective optimization [33] We aim to model connectivity to give our network access to lower positions in the loss basin. In this paper, we propose simple yet effective method to demonstrate the practical significance of this study.
Weight Averaging. While weight averaging shares some similarities with the aggregation opera- tions in Graph Neural Networks (2 45] 23 22 its primary application is distinct. In the context of

---

neural architectures, weight averaging typically represents overall averaging of weights across different models with the same architecture, diverging from the node-based aggregation in GNNs, His- torically, the general idea of weight averaging can trace its roots back to convex optimization [4] 38
And with the breakthrough of loss landscape visualization technology [2 28
many applications
have applied this idea to neural networks [12
49 34 The concept of weight averaging has been 19
integrated into single-traeectory training [22
49 and has seen extensive application in distributed setups 129 31 (0). Specifically, [99 found that the models trained with SGD optimizer often fell on the edge flat basin. Based on that, they propose the SWA method, which uses one-time weight averaging of multiple network points after the entire training process is finished, leading the network down to lower locations in the basin. The later work 449 proposes an optimizer that continuously uses weight averaging to update model weight to improve performance and robustness. Averaging weight during training is more effective than the averaging after training. Unlike single-trajectory optimization strategy, the recent Model Soups method [46] is inspired by mode connectivity, which starts from a typical pretrained weight and averages final fine-tuned models to improve generalizabil- ity. These methods inspire to employ weight averaging method with multi-data augmentations to improve convergence and generalizability.
Ensemble. The ensemble learning is a traditional technology that combines multiple model outputs to achieve better robustness and performance 16
[ 2 16 27]. Ensemble methods usually need to train different models to reduce variance and improve the prediction effect. However, in recent years, some methods [8] 12] obtain different model checkpoints to conduct the ensemble in a single trajectory of model training, which significantly reduces the training time. Note that these methods all require separate inference through each model, which adds to the calculation cost. In contrast, method does not require additional inference calculations.
3 Method
In this section, we present our opti- mization method Lookaround and pro- vide an in-depth analysis of its proper- ties. In Section [3.1. we provide a de- tailed description of the Lookaround optimizer. In Section 3.2, we ana- lyze the expected risk of Lookaround, then thoroughly study gence at various levels and compare with the existing optimizer Looka- head [49] and SGD [40]. Intuitively, our proposed optimizer seeks flat min- ima along the training trajectory by iteratively looking for the diverse model candidates around the trained points, hence called Lookaround. The pseudo-code of Lookaround is pro- vided in Algorithm
Algorithm Lookaround Optimizer.
Require: Initial parameters do, objective function C, data augmentation list AUG of size d, synchronization period k, optimizer A. dataset 2, numbers of training epochs E. for epoch = 1 to E do Synchronize parameters for 1,2, d do
conver-
Or.j.0 - Φt-1
end for # Around Step: Independent model training. for i 1, 2,..... do
sample minibatch of data B ~ D
for j 1, d do
0r,j. - J + A(C, O.,, AUG,
end for end for
# Average Step: Weight averaging.
Compute average weight ,,,,,k 1 Dd 01.j Perform update d; 02,,, end for
3.1 Lookaround Optimizer
In this subsection, we introduce the de- tails of Lookaround. In Lookaround, the training process is divided into multiple intervals of length k. with each interval consisting of an "around step" and an "average step". Let Φ denote the initial weights. At th round of Lookaround, the weights are updated from Φ,-1 to ob according to the around step and the average step. The resulting weights d; then serve the starting point for the subsequent round.
return parameters
Around Step. Given the optimizer A, the objective function L, and the stochastic mini-batch of data B from train dataset D. In the around step, d different models are independently trained under d
3

---

different data augmentations AUG = (AUG1..... AUGa) for k batch steps, as follows:
0t.jk it.j + A(C, AUG
(I)
Thanks to the data augmentations, each model is trained in period k to diverse location in the loss landscape scattered around @t-1. The Around Step allows optimizing the region surrounding each model iterate, which helps the search of flatter minima. Moreover, note that as the d different models are trained independently, we can use parallel computing to speed up the whole training process.
Average Step. In the average step, simply average Re the weights of each independently trained model to obtain Φ for the next around step as follows:
0 0
t=1
It has been observed that models with the same initialization and trained on different data augmen- tations exhibit similar loss basins (35), with the models scattered around the edges of these basins. Therefore, incorporating simple averaging technique into the training process may facilitate the convergence of the models towards lower-loss regions. In Appendix A we demonstrate how this technique effectively guides the model to the interior of the loss basin.
3.2
Theoretical Analysis
3.2.1 Noisy Quadratic Analysis
The quadratic noise function has been commonly adopted as an effective base model to analyze optimization algorithms [42 49 47 255, where the noise incorporates stochasticity introduced by mini-batch sampling. In this subsection, we analyze the steady-state risk of our proposed optimizer on the quadratic noise function to gain insights into the performance of Lookaround, and compare the result with those obtained using SGD and Lookahead.
The quadratic noise model is defined as Č(0) [0 -) c). Following Zhang et al. [49], we assume that the noise vector с ~ NN(0", I)) both A and Σ are diagonal matrices, and that the optimal solution 6* respectively. As the i-th element of the noise vector e. satisfies [[[]] of the iterates O, can be written as follows:
We denote a, and of as the i-th elements on the diagonal of and Σ,
(0T)² + of. the expected loss
1 Σ (BLO,)) VIO.ol of),
((0.) E((()
where or,; represents the 8th item of the parameter 0, The limiting risk of SGD, Lookahead and Lookaround are compared in the following Proposition by unwrapping [[O,] and v Equation 3 Proposition (Steady-state risk). Let < < 1/L be the learning rate satisfying L max; ag One can obtain that, in the noisy quadratic setup. the variance of the iterates obtained by SGD, Lookahead (49] and Lookaround converge to the following matrix:
72 2
(4)
I- AA)
a [I (I-7A)24
V Lookahead
V
(5)
a²(1 (I ) + 2a(1 aI (I (1-7A)
*1, (E(U,)
a2 (I 7 2k + 20 aI) (I *))
v Lookaround
V
Lookahead:
(6)
a (dI (d 1) 7 7
<1, if d33 and (
respectively, where a denotes the average weight factor of models with varying trajectory points, as described in [4).
Proposition I limplies that the steady-state variance matrix of Lookaround is element- -iie smaller than those of SGD and Lookahead if > and a E [1/2, 1). Moreover, Proposition [ shows that increasing the number of data augmentation methods d yields smaller limiting variance and thus lower expected loss (see equation 3) which guarantees good generalizability in real-world scenarios. The proof is deferred to Appendix

---

3.2.2 Convergence on Deterministic Quadratic function
In this section, we analyze the convergence of Lookaround and Lookahead in the noise-free quadratic setting, which is important for studying the convergence of optimization algorithms [3. 37
Convergence rate characterizes the speed in quadratic function at which the independent variable converges the optimal solution, satisfying 1 0*1| < p² [10 0- In order calculate the convergence rate, we model the optimization process. and treating this function as linear dynamical system allows us to calculate the convergence rate, as in [30]. The value of the convergence rate will be determined by the cigenvalues of the dynamic transition equation, and we leave the calculation of this part to show in the Appendix [8.2
Given the emphasized significance of the condition number in paper 37 we conducted extensive experiments under series of conditional numbers.
We evaluated the convergence rate of Lookaround, Looka- head, and Classical Momentum (CM) under different con- dition numbers from 10' to 10", which are shown in Fig- ure 2 The blue dashed line represents the optimal conver- gence rate that an algorithm can achieve in the absence of any constraints. A value closer to the blue dashed line indicates a faster convergence rate. CM can achieve the optimal convergence rate critical condition num- ber [30]. To the left of this critical point, there exist com- plex eigenvalues that correspond to oscillations. In this regime, CM exhibits a flat convergence rate, which is known "under-damped" behavior [30. We show that Lookaround and Lookahead are faster than CM in the under-damped regime. Lookaround converges faster and is more stable than Lookahead in the low condition num- ber case, and realizes comparable performance with Lookahead the high condition number case. In subsequent experiments, we show that Lookaround achieves fast and stable convergence in training deep neural networks on popular benchmarks.
Op hpttmal
1-Comerreene 2tta
10-1
CM Looknhead
Leasarsund
10-2
10-3
30"
10
10P
10a
105
105
3:00
Condition Number
Figure 2: Convergence rate on quadrat- ics of varying condition number. We fix the step k = 20 for Lookahead and Lookaround, and fix the CM factor 3 0.99.
Experiments
In Section 4.1 and Section 4.2 we present experimental verification on different tasks, including both random initialization and finetuning tasks on both ViTs and CNNs architectures to validate the effectiveness of the Lookaround optimizer. In Section 4.4] we compare the proposed method with ensemble learning methods that require multiple models. In Section 4.5 we conduct ablation experiments on different components of the Lookaround method to explore their contributions. In Section 4.6 we analyze the parameter robustness of Lookaround and further investigate its performance in the micro-domain.
Within a single epoch, both the proposed Lookaround and the competitors undergo training on an identical times the data augmentations. With such setup, guarantee consistency in the data volume utilized by each method, thereby ensuring fair comparisons in terms of computation.
4.1 Random Initialization
Random initialization is standard model initialization method in deep learning. In this subsection, we verify the performance of our algorithm on various networks with randomly initialized weights based on CIFAR and ImageNet datasets.
4.1.1 CIFAR 10 and CIFAR 100
We conduct our experiments on CIFARIO [6] and CIFAR100 [26 datasets. Both CIFAR10 and CIFAR 100 datasets have 60,000 images, 50,000 of which are used for training and 10,000 for validation. We use SGDM (36] as our baseline, and we compare our method with SWA [1).
SWAD 3. Lookahead (4). We validate different methods on multiple network architectures such as
VGG19 43 ResNet50 17 ResNet101 [7]. ResNet152 [17). ResNeXt50 (8) and all methods are
5

---

Table 1: Test set accuracy under training procedure with random initialized models. In this table, all models are trained for same amount of time at x2 32 resolution. (To provide more comprehensive view of the data, we substitut the Top5 metric for CIFARTO with the NLL loss.)
VGG19
ResNet50
ResNet101
ResNet152
ResNext50
Method
Top1
NLI
Top1
NLL
Top1
NLI
Top1
NLI
Top1
NIIL
SGDM
93.92
0.26
95.96
0.16
96.16
0.16
96.28
0.16
95.72
0.17
SWA
94.89
0.21
96.42
0.14
96,34
0.14
96.82
0.13
96.09
0.16
CIFARIO
SWAD
93.23
0.29
95.39
0.19
94.49
0.22
95.00
0.20
93.89
0.26
Lookahead
94.72
0.23
96.38
0.16
96.61
0.15
96.46
0.15
96.47
0.15
Ours
94.44
0.25
96.59
0.14
96.73
0.14
97.02
0.14
96.70
0.13
VGG19
ResNet50
ResNet101
ResNet152
ResNext50
Method
Top1
Top5
Top1
Top5
Top1
Top5
Top1
Top5
Top1
Top5
SGDM
73.84
91.09
79.61
95.21
79.91
95.54
80.16
95.49
79.10
94.86
SWA
73.62
90.89
80.17
95.56
80.53
95.39
80.86
95.43
79.14
94.96
CIFAR100
SWAD
73.37
89.93
80.19
95.46
79.92
95.09
80.00
95.36
79.27
95.17
Lookahead
74.02
90.68
80.06
95.24
81.14
95.84
81.36
95.77
80.12
95.26
Ours
74.29
91.15
81.60
95.99
81.97
96.15
82.22
96.40
81.14
96.12
VCG19
ResNet50
ResNeXt50
4
SCDM
SGDM Lookshead Lookaround
SCDM
Train Loes
Lookabead Lookaround
Lookahoad
3
Leokarvand
2
2
2
1
0
500
100
150
200
50
100
150
200
0
50
100
150
200
Top-1 Accuracye%)
80
70
70
70
60
60
60
SCDM
SGDM
SCDM
50
Lookabsad
50
Lookshead
50
Lookahead
Lookaround
Lookaround
Lookareand
40
kanda
40
150
50
100 Epach
150
200
100
150
200
Epoch
Epoch
Figure 3: Training loss and Top-1 accuracy curves of various networks CIFAR100 under different optimization methods.
performed on well-validated parameters. For each epoch in all experiments, we use the same mixed data for training using three kinds of data augmentation: random horizontal flip, random vertical flip, and RandAugment (5). This ensures that the comparison between different methods is fair. For the training process, we use discount factor of size 0.2 decay the initial learning rate 0.1 at 60, 120, 160 epochs. Please refer to Appendix I. for more specific details.
We present our results in Figure 3 and Table I which indicates that method achieves stable perfor- mance improvement on different networks. Compared to the SGDM optimizer, both Lookaround and Lookahead demonstrate faster convergence rates and higher accuracy. However, what's noteworthy is that, with training losses similar to Lookahead, Lookaround not only achieves a higher accuracy but also has better stability. This suggests that within the same training duration, we have found more optimal optimization path. Meanwhile, although the SWA and SWAD methods adopt the same optimization process the SGDM, they achieve higher performance via multiple samplings in the training trajectory combined with an average step. In contrast to SWA and SWAD, the multi-averagin Lookaround can achieve consistent and superior performance across different networks.
4.1.2 ImageNet
The ImageNet dataset widely used to test the model performance. has training set of 1.28 million images and validation set of 50,000 images with 1,000 classes. We train the model

---

BesNet50
BesNet50 e150
V/F-B/16
VIF
SGDM
Topa AccuracyrSs
1op- Accurscy's
Train Loos
Looksahead
Trais Looa
Looksshead
8
Lookaround
Lockaround
5
SCDM
Adam
Lookshead
Lookerround
Lookeround
0
10
20
300
40
50
10
20
30
40
50
10
20
30
40
50
10
20
30 Epoch
40
Epoch
Epach
Epoch
Figure Training loss Top-1 accuracy curves of ResNet50 (left) and ViT-B/16 (right) on CIFAR100 under different optimization methods.
Table 2: Top-1 accuracy of ResNet50 Ima- geNet under different optimization methods.
on the training set and observe the model perfor- mance the validation set. Based on standard hy- perparameter settings, we train for 100 epochs and use multi-step scheduler to adjust the learning rate on 30, 60 and 90 epochs with a discount factor of size 0.1. We present our results in Table 2 We get similar conclusions on the ImageNet as on CIFAR: higher test set accuracy and stronger generalization performance, which coincides with the theoretical results in the previous section.
OPTIMIZER
Top-1
Top-5
SGDM
75.97
92.89
SWA
76.78
93.18
LOOKAHEAD
76.52
93.11
LOOKAROUND
77.32
93.29
4.2 Finetuning
The transfer learning technique, which involves pretraining a model on large related dataset and then finetuning on smaller target dataset of interest, has become widely used and powerful approach for creating high-quality models across various tasks.
To explore the scalability of our Lookaround method, we apply our method to scenarios that require pre-training, including CNN or ViT (7] architecture. Unlike the previous experiments, we adopt more appropriate hyperparameters for training in the ViT experiments, including the Adam optimizer and cosine annealing scheduler structure. Our results are shown in Table 3Jad and Figure 4 Under the ResNet architecture, our proposed Lookaround method remains stable and performs better, while the SGDM method shows greater fluctuations.
In contrast, the Lookaround methods based weight averaging do not produce such phenomenon, which partly demonstrates the hypothesis that average weight can reduce the convergence variance. In our comparison experiments with the other optimizations, we found that the results of VIT-B/I6 under different optimizers were better than those reported in the original paper [Z]. This is mainly due use of more data augmentation during the training process, aiming for fair comparison with Lookaround. This result also indicates that Lookaround can still achieve better performance under improved baseline conditions.
Table 3: The test set accuracy and the NLL loss under training procedure with pretrained models. In this table, all models are fine-tuned for the same amount of time 224 224 resolution. The results of Adam using ViT-B/16 are taken from reference (7].
CIFARIO
CIFAR100
Backbone
Method
Top-1
NLL
#param.
Top-1
NLL
#param.
SGDM
97.55
0.109
23.52M
84.50
0.692
23.71M
SWA
97.48
0.105
23.52M
84.81
0.668
23.71M
ResNet50
Lookahead
97.65
0.103
23,52M
84.78
0.676
23.71M
Ours
97.82
0.099
23.52M
85.20
0.658
23.71M
Adam
98.13 13
85.65M
87.13
85.72M
Adam
98.34
0.060
85,65M
91.55
0.298
85,72M
VIT-B/16
SWA
98.47
0.049
85.65M
91.32
0.304
85.72M
Lookahead
98.51
0.047
85.65M
91.76
0.280
85.72M
Ours
98.71
0.041
85.65M
92.21
0.267
85.72M

---

Table 4: The set accuracy under SAM and Lookaround optimization using ResNet50 or ViT-B/16. The results of SAM using ResNet50 are taken from 77
Backbone
Method
resolution
pretrain
CIFARIO
CIFARI00
SAMT
32
79.10
ResNet50
Lookaround
32
96.59
81.60
Lookaround SAM
32
96.38
79.79
SAM
224
V
97.65
85.97
ResNet50
Lookaround
224
V
97.82
85.20
Lookaround SAM
224
V
97.88
86.21
SAM
224
V
98.51
92.39
ViT-B/16
Lookaround
224
V
98.71
92.21
Lookaround SAM
224
V
98.67
92.54
4.3
Compared with Sharpness-Awary Minimization
We draw a comparison between Lookaround and Sharpness-Awan Minimization (SAM) [0], an algorithm designed to improve generalization by steering model parameters towards flatter loss regions The comparative results are presented Table
The results seem to indicate that, under the default parameters of SAM, this method is more suitable for higher resolutions, while Lookaround achieves the best performance at lower resolutions. For the finetuning, neither SAM nor Lookaround consistently emerges the top choice when used indi- vidually. However, when combined, they often enhance model performance significantly. Thus, for higher accuracy, recommend using both the SAM and Lookaround optimizers concurrently. Both are orthogonal to each other. By combining the two methods, we can achieve superior performance.
4.4 Compared with Ensemble Method
Ensemble methods traditionally used to further improve inference performance by combining the outputs of multiple different models. We compare our Lookaround method with classical Ensemble methods (Logit Ensemble and Snapshot Ensemble (18), and the results shown in Table 5
The performance of the single model obtained by our training exceeds the performance of the ensemble under multiple models. Moreover, the model obtained by method is not a simple superposition of Ensemble models. We take the single model obtained by us with the three models obtained by Logit Ensemble Logit Ensemble, and get a more significant performance improvement. This indi- cates that the model we obtained is orthogonal to the three models obtained by Logit Ensemble and can be combined produce better performance.
Table 5: Performance compared with the ensemble meth- ods (without pretrained weights) on CIFARI00 dataset. "Ours Logit Ensemble" means that we take the model obtained by our method and the three models in Logit En- semble to Logit Ensemble.
Method
ResNet50
ResNet101
ResNet152
Logit Ensemble
81.16
81,92
81.89
SnapShot Ensemble
79.88
80.63
80.23
Ours (Lookaround)
81.60
81.97
82.22
Ours Logit Ensemble
83.43
83.35
83.53
4.5 Ablation Study
Table 6: Ablation Study of Data Augmentation (DA) and Weight Averaging (WA) in the proposed Lookaround using ResNet50.
In this subsection, we investigate impact different compo- nents of Lookaround on its per- formance, including data Aug- mentation methods and weight averaging. As shown in 6
Dataset
DA
WA
Ace (%)
Dataset
DA
WA
Acc (%)
95.84
78.81
95,96
79.61
CIFARIO
CIFAR100
95.79
79.49
96,59
81.60
'Intriguingly, SAM is proved exhibit dynamics akin decentralized weight averaging algorithm (51).

---

Random Initialization
Finetuning
Top-1 Accuracy (%)
80
81.0
70
MmM
SCDM
60
(Ours)Small k(5)
80.5
100
(Ours)
50
20 50 Steps(k)
100 200
5001000
25
50
100
125
150
175
200
Epoch
Figure 5: Top-1 accuracy of ResNet50 on CIFAR dataset with different steps k. (Left) Error bar of three different random seeds with random initialization weight. (Right) Top-1 accuracy of two different k values with pretrained weight.
10-0.1
Ir-0
65
72.5-
Top-1 Aor (N6
60
Ape Pa
70.0
Ace (%)
74
155
67.5
73-
65.0
62.5-
72.
60.0
100
150
100
150
100
150
Iteration
Iteration
Iteration
H-trajectory
Binet
trojectoyy
Ret
M trtjjjctoryy
Mean-ne
Figure 6: Accuracy curve of sub-model state update on the test set with different learning rates using ResNer50 on CIFARI00. At position 0 on the horizontal axis, H-net (trained by randomly horizontally flipped data) and R-net (trained by RandAugment data) are averaged Mean-net. Subsequently, during 160 weight iterations, H-net and R-net continue to be trained with their respective data, while Mean-net represents the model obtained by averaging the weights at each itreation.
when we consider Data Augmentation (DA) or Weight Averaging (WA) independently. their im- provement effects may be relatively small. However, when we combine them, the synergistic effect between them can lead to more significant overall improvement. Specifically, we found that the improvement effects of DA and WA show nonlinear trend in the interaction case, indicating certain synergistic effect between them. Under the interaction. Data augmentation guides the model into different loss basins, and Weight Averaging guides different models into lower loss basins, resulting in higher performance improvement and a qualitative leap.
4.6 Additional Analysis
Robustness of the Parameters. For the number of data augmentations, we
Table 7: Top-1 accuracy (%) of different data augmenta- tion (DA) number by using ResNer50 on CIFAR100 dataset.
train the model with different num- bers of data augmentations, and the results are shown in Table 7 The re- sults indicate that an increase in number of data augmentations results in improved network performance. In order to reduce the training time, only three data augmentation methods selected in this paper for experimental verification at different levels. Note that the methods compared with ours use the same incremental data set, so the comparison is fair.
of DA
2
3
Top-1 (%)
78.20
80.82
81.60
81.19
81.74
82.02
Top-5 (%)
94.50
95.19
95.99
95.65
95.85
96.02
For different choices k: the number of around steps, we select different intervals for full verification, and the results are shown in Figure5][Left (Left). Let us look at Large-k (391) versus Small-k(55) in Figure single epoch). We find that large steps
(Right) (391 is determined by the number of batches (green line) can achieve higher accuracy and stability in the early stage of training, while small steps can achieve higher accuracy in the later stage. In future research, we may explore a learning rate scheduler suitable for Lookaround to help it achieve the best performance.

---

We visualize the training trajectory accuracy of the sub-models in Lookaround in Figure 6 Under different learning rates, the model, after averaging the weights, can maintain a good baseline and continuously improve the performance of the model under the low variance, and the sub-models also obtain more beneficial training effects due to the weight averaging.
Conclusion
In this paper, present the Lookaround optimizer, SGD-based optimizer that employs diversity- in-locality exploration strategy to find solutions with high generalization ability in the loss basin. We theoretically prove that Lookaround obtain lower variance and faster convergence speed. Experimental results on CIFAR and ImageNet and across various network backbones show that, our proposed Lookaround significantly improves the training stability and achieves better generalization performance comparable to the single-trajectory WA optimizers and ensemble methods.
Linitation. Lookaround is limited by the additional cost of network searching using different data augmentation, resulting in longer training time proportional to the the number of trained networks. Notably, Lookaround incurs the same inference time other optimization methods. Considering the significantly superior performance, we believe the extra time consumption is worthwhile in many real- world scenarios where the training time time can be reduced due to the parallelizability of the "around step" mechanism in Lookaround.
concerned the inference time. Moreover, training
Acknowledgements
This work is supported by the National Key Research and Development Project (Grant No: 2022YFB2703101 100), National Natural Science Foundation of China (62106220, 61976186, U20B2066), and Ningbo Natural Science Foundation (2021J189) and the advanced computing resources provided by the Supercomputing Center of Hangzhou City University, We sincerely thank the anonymous reviewers for their valuable comments on the work.
10

---

References
[1] Eric Bauer and Ron Kohavi. An empirical comparison of voting classification algorithms: Bagging, boosting, and variants. Machine Learning, 1999.
[2] Leo Breiman. Bagging predictors. Machine Learning archive, 1996.
[3] Junbum Cha, Sanghyuk Chun, Kyungjac Lec, Han-Cheol Cho, Scunghyun Park, Yunsung Lec, and Sungrac Park. Swad: Domain generalization by seeking flat minima. Conference on Neural Information Processing Systems, 2021.
[4] Ekin D. Cubuk, Barret Zoph, Jonathon Shlens, and Quoe V. Le. Randaugment: Practical automated data augmentation with reduced search space. Conference on Neural Information Processing Systems, 2019.
[5] Ekin D. Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V. Le. Randaugment: Practical automated data augmentation with reduced search space. In JEEE/CVF Conference Computer Vision and Pattern Recognition Workshops, 2020.
[6] Thomas G. Dietterich. Ensemble methods in machine learning. mulriple classifier systems, 2000.
[7] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Min- derer, G. Heigold, and S. Gelly. An image is worth 16x words: Transformers for image recognition scale. In International Conference on Learning Representations, 2021.
[8] Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred A. Hamprecht, Essentially barriers in neural network energy landscape. International Conference on Machine Learning, 2018.
[9] R. Entezari, H. Sedghi, O. Saukh, and B. Neyshabur. The role of permutation invariance in linear mode connectivity of neural networks. International Conference Learning Representations, 2022.
[10] Pierre Foret, Ariel Kleiner. Hossein Mobahi, and Behnam Neyshabur. Sharpness- aware minimization for efficiently improving generalization. International Conference Learning Representations, 2021.
[11] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin. Linear mode connec- tivity and the lottery ticket hypothesis. International Conference Machine Learning, 2019.
[12] Timur Garipov, Pavel Izmailov, D. A. Podoprikhin, Dmitry Vetrov, and Andrew Gordon Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. Conference Neural Information Processing Systems, 2018.
[13] Gabriel Goh. Why momentum really works. Distill, 2017.
[14] A. Gotmare, N. S. Keskar, C. Xiong, and R. Socher. Using mode connectivity for loss landscape analysis. arXiv: Learning, 2018.
[15] Ishaan Gulrajani and David Lopez-Paz. Paz. In search of lost domain generalization. In International Conference Learning Repreventations, 2020.
[16] Trevor Hastie, Jerome H. Friedman, and Robert Tibshirani. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer Series in Statistics. 2001.
[17] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE/CVF Computer Vision and Pattern Recognition Conference, 2016.
[18] Gao Huang, Yixuan Li, Geoff Plciss, Zhuang Liu, John Hopcroft, and Kilian Q. Weinberger. Snapshot ensembles: Train 1, get M for free. In International Conference on Learning Representations, 2017.
[19] Pavel Izmailov, D. A. Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and hetter generalization. Conference on Uncertainty in Artificial Intelligence, 2018.
[20] Samyak Jain, Sravanti Addepalli, Pawan Kumar Sahu, Priyam Dey, and R Venkatesh Babu. Dart: Diversify- aggregate-repeat training improves generalization of neural networks. In IEEE/CVF Computer Vision and Pattern Recognition Conference, 2023.
[21] Sébastien Jean, Kyunghyun Cho, Roland Memisevie, and Yoshua Bengio. On using very large target vocabulary for neural machine translation. Annual Meeting of the Association for Computational Linguistics, 2014.
[22] Yongcheng Jing. Efficient Representation Learning With Graph Neural Networks. PhD thesis, 2023.

---

[23] Yongcheng Jing. Yiding Yang. Xinchao Wang. Mingli Song, and Dacheng Tao. Meta-aggregator: Learning to aggregate for 1-bit graph neural networks. In IEEE International Conference on Computer Vision, 2021.
[24] Thomas N Kipf and Max Welling. Semi supervisd classification with graph convolutional networks. In International Conference on Learning Representations. 2016.
[25] Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions, In International Conference on Machine Learning, 2017.
[26] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Handbook of Systemic Autoimmune Diseases, 2009.
[27] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Conference Neurul Information Processing Systems, 2017.
[28] Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. Conference on Neural Information Processing Systems, 2018.
[29] Xiangru Lian. Ce Zhang. Huan Zhang, Cho-Jui Hsieh, Wei Zhang, and Ji Liu, Can decentralized algorithms outperform centralized algorithms? a case study for decentralized parallel stochastic gradient descent. In Advances in Neural Information Processing Systems, 2017.
[30] James Lucas, Shengyang Sun, Richard Zemel, and Roger Grosse. Aggregated momentum: Stability through passive damping. In International Conference on Learning Representations, 2018.
[31] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Cettuumniiaaii-EEfcieenn Learning of Deep Networks from Decentralized Data. In International Confer- ence on Artificial Intelligence and Statistics, 2017.
[32] Stephen Merity, Nitish Shirish Keskar, and Richard Socher. Regularizing and optimizing Istm language models. Learning, 2017.
[33] Seyed Iman Mirzadeh, Mchrdad Farajtabar, Dilan Gorur, Razvan Pascanu, and Hassan Ghasemzadeh. Linear mode connectivity in multitask and continual learning. Learning, 2020.
[34] Giung Nam, Hyungi Lee, Byeongho Heo, and Juho Lee. Improving ensemble distillation with weight avearging and diversifying perturbation. In International Conference on Machine Learning, 2022.
[35] Bchnam Neyshabur, Hanic Sedghi, and Chiyuan Zhang. What is being transferred in transfer learning. Conference on Neural Information Processing Systems, 2020.
[36] Q. Ning. On the momentum term in gradient descent learning algorithms. Neural Networks, 1999.
[37] Brendan O' O'donoghue and Emmanuel Candes. Adaptive restart for accelerated gradient schemes. Founda- tions of computational mathematicx, 15:715-732, 2015.
[38] Boris T. Polyak and Anatoli Juditsky, Acceleration of stochastic approximation by averaging. Siam Journal on Control and Optimization, 1992.
[39] Yujia Qin, Cheng Qian, Jing Yi. Weize Chen, Yankai Lin, Xu Han, Zhiyuan Liu, Maosong Sun, and Jie Zhou. Exploring mode connectivity for pre-trained language models. In Conference on Empirical Methods in Natural Language Processing, 2022.
[40] Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics, pages 400-407, 1951.
[41] David Ruppert. Efficient estimations from slowly convergent robbins-monro process. Technical report, Cornell University Operations Research and Industrial Engineering, 1988.
[42] Tom Schaul, Sixin Zhang, and Yann LeCun. No more pesky learning rates. In International Conference on Machine Learning, 2013.
[43] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. IEEE/CVF Computer Vision and Pattern Recognition Conference, 2014.
[44] Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In International Conference Machine Learning, pages 2013.
12

---

[45] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In International Conference Learning Representations, 2018.
[46] Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo Lopes, Ari S. Morcos, Hongseok Namkoong. Ali Farhadi, Yair Carmon, Simon Komblith, and Ludwig Schmidt. Model soups: averaging weights multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning. 2022.
[47] Yuhuai Wu, Mengye Ren, Renjie Liao, and Roger Grosse. Understanding moort-horizon bias in stochastic meta meta-optimization. arXiv preprint 10X 2018.
[48] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transfor- mations for deep neural networks. JEEE/CVF Computer Vision and Puttern Recognition Conference, 2016.
[49] Michael R. Zhang, James Lucas, Jimmy Ba, and Geoffrey E. Hinton. Lookahead optimizer: k steps forward, 1 step back. Conference on Neural Information Processing Systems, 2019.
[50] Tongtian Zhu, Fengxiang He, Lan Zhang, Zhengyang Niu, Mingli Song, and Dacheng Tao. Topology-aware gencralization of decentralized SGD. In International Conference on Machine Learning, 2022.
[51] Tongtian Zhu, Fengxiang He, Kaixuan Chen, Mingli Song, and Dacheng Tao. Decentralized SGD and average-direction SAM are asymptotically equivalent. In International Conference Muchine Learning. 2023.
13

---

A Exploration in the Loss Landscape
To demonstrate the effectiveness of Lookaround during training, we set up an experiment and visualize the loss landscape of the models under different data augmentations in Figure By observing the loss landscape, we can gain clearer understanding of the role played by the weight averaging at different stages during the training process.
Training Process and Parameter Settings. We first train a ResNet50 on the CIFAR 100 dataset. The learning rate is initialized to 0.1 and decay at 60, 120, and 160 epochs using MultiStepLR scheduler with a decay factor 0.2. The batch size is set to 128, we use stochastic gradient descent with momentum to optimize the model and use random crops and random vertical flips augmentation to enhance the training datasets. We use model checkpoints at epochs 50, 110, and 150 as our three pretrained models( V-network). The three pretrained models correspond to learning rates of 0.1, 0.02 and 0.004, respectively. Using these pretrained models as the starting point, we finetune the model with 1, 10, 100, 1000, 10000 iterations under the corresponding learning rate and the setting of random horizontal flipping (H-network) or RandAugment (R-network).
Visualization Method. We use the visualization method in Garipov et al. 12 to plot the loss land- scape. In this method, the weights of the three models are flattened respectively as one-dimensional vectors We, Wh, Wy, and then two orthogonal vectors are calculated between the three vectors as the X-axis direction and the Y-axis direction: (wa, Wp). Then the normalized vectors ú = u//lul 0 e/]lul foam an orthonormal basis in the plane contain wy; Wh. Wy. Then a point with coordinates (2: y) in the plane would be given by P = Wy + x ú + y 0.
(w/, w,), V (w, W'y w/ wo)/ w',, w..l2
Discussion and Inspiring. Under different learning rates and different around steps k Lookaround has the tendency to lead the model trained on different data augmentation to the near-constant loss manifold. In such circumstances, the "average step" can lead the model into the center of the loss basin to get lower test loss. However, weight averaging does not necessarily work in all cases. For example, the network obtained after weight averaging gets a larger loss under a large learning rate with a large around step (e.g.. Ir 0.1, between different basins rather than in the center of a basin. Moreover, in the case of around step k 1, the weight averaging also does not achieve better performance. Nevertheless, such extreme cases do not prevent weight averaging from being useful tool to speed up the training process. The center of the basin in loss landscape, which requires multi-step gradient descent to reach, can be reached by only one weight averaging step. At other learning rates and around steps k, the models after weight averaging all result in lower loss than the individual model. Such phenomena encourage us to explore more methods to find more optimal solutions in the loss landscape in the future.
10000). In this case, the model is located on a peak
B Steady-state and Convergence Analysis of Lookaround
We use quadratic functions to analyze the steady-state and convergence analysis of Lookaround. First, we present the proof of Proposition 2
Proposition (Steady-state risk). Let < 7 < 1/L be the learning rate satisfying L = max; 0g.. One can obtain that in the noisy quadratic setup, the variances of SGD, Lookahead [49] and Lookaround converge to the following fixed matrix:
7.2 2
V =
(7)
I- I-
AA)
0.2 [I (I-7A)20
v Lookahead
V
(8)
a²(I (I A 2b + 2a(1 a)([ -(1 (I-7A)+)
a2 (I- AA) + 20 aI (I ) v
2 Lookaround
Lookahead-
(9)
2 (dI (d [I 7A)M
<1, if d33 and (12211
14

---

around steps &
around stegss
around steps 100
around stags 1000
around steps 10000
Rate 0.3
1.6
1.2
0.8
Learning Rate
1.6
1.2
0.8
Learting 0.004
1.6
122
Network obtained through "average step"
Figure 7: The test loss landscape of ResNet50 CIFARI00. The diamond block (V-network) represents the pretrained model trained with random vertical flipping. Then we can use random horizontal flipping and RandAugment to finetune V-network to get H-network and R-network.
respectively, where denotes the average weight factor of models with varying trajectory points, as described in 44
From Wu et al.
47
we can obtain the following dynamics of SGD with learning rate ?
[[ (I 7A) B[ v|0(++1] = (I (I->A)² V[0 + A2
Lemma 1,
The expectation and variance of lookaround have the following iterates:
(l [I -I- (I) (( (1),
(10)
([b d d AA) v[ d(I 1--II- -I-
(11)
Φi yields Φ by performing an around step and an average step.
Proof. The expected iterate follows from SGD:
E 1 = El Σ 00. 2 B(
d
Σ [I AA) BiO..oo (I (I- 7A)* E[o'
For the variance 0² 1+ we can break down into two parts as follows:
d V( = v1 Σ 0r..nl Σ
VlOc..el
Σ
de
itj.eeijed
15

---

The covariance of the different models can be calculated in the following way:
cov(Or,i, (r.jkk = E j. - E( [E(0c.j.k]
BB[[ - A - -I- >A (1- B|O'
(I- A
2k Vl
After permuting and regrouping again, obtain the iterate with respect to the variance.
v
1 dc
cov(Bl, O(.kk)
Σ d2
V[oei.kl
k-1
d2 (de -d) A 2k +
(I -I- (I-AA)=0 7 2
1=0
d 1 (1 - AA) 24 V1 +
A (I- (I-
d(I AA)²)
The proof is now complete.
Remark. From Equation T0 the expectation term for & Lookaround eventually converges to as does Lookahead and SGD.
From Zhang et al. 449 we have the following analysis about the variance fixed point of lookahead with learning rate and weight factor a, which represents the average weight factor of models with different trajectory points in the Lookahead optimizer, which is generally (0, 1):
a² (I 7A
Lookakead = a² - (I - A) + 2a(1 a) (I- (1))
SGD
(12)
We now derive the fixed point of the variance, proceed with the proof of Proposition 1:
aI- 7A) (1-) + 2a aI- (I-
*
V Lookaround
Lookahead
(2I (d 1) 7A)
Proof.
d-1 [I
7.2 A - (I 7A) 2kg
V Lookavound
d
AA)
V
Lookeroand oand
+
d(I - (I - 7)
1
J >2 (I 7 22
V Lookaround
d
(I A 2k
d(I (I (1- 72)
(I 1
2k]
d (d - 1) A
AA)
(I (I- A 2k
d (d 1)(I >A 2A VsGD.
According to Equation 12 we can deduce that
I I- (I A 22
V LLookaround
d (d [I AA)2 22 SGD
a² (1 (1- 2k
()(1-a)(1 (I 7A))
V Lookahead-
a² (d 1) 2k)
The proof is now complete.
B.1 Comparing the dynamics of Lookahead
We now proceed with the proof for the range of constraints variable Equation 9 When > and [0.5, 1), the Lookaround method can obtain a smaller variance than the Lookahead method:
16

---

Proof. Let B = (I --) due to 0 < y < 1/L, = max,a,, so we can have B=I. Substituting the matrix B into the expressions for the variance fixed point relation of Lookaround and Lookahead:
a² (I (A ) 20 a)
(d (d 1) - --) -I- V
V Lookaround
Lookakead
I B² + 2-21-00 B)
0
Lookaheed
d1 (d. 1)B²
2- B 2-2n
V Lookahead
d (d 1)B²
Let the coefficient matrix be denoted as C, when > 3, for each diagonal element Cj of C, we can scale the denominator of this expression follows:
2-a B
2-2a BBs
Cs
2B
Then, can derive the range of by restricting the right-hand side expression as follows:
2-a B
2-2a BB
1,
3 2B
As 0 & Ba & 1, we can multiply both sides of the inequality by the denominator, then rearrange and combine like terms to obtain the following form:
B 2 2a
40 < 0.
BBs
a
Skipping the detailed steps, we can obtain the case where a E [0.5, 1) and d > 3 (a < is subject to Lookahead's 's settings), the coefficient matrix C so the convergence speed of Lookaround is slower than Lookahead.
0.5 by solving the quadratic equation. Therefore, in
B.2 Deterministic quadratic convergence
Our method samples data from multiple data augmentation strategies, which can be analogously viewed averaging the sampling of historical trajectories during the convergence analysis of quadratic functions. Thus, for weight averaging, we select model points that align with each point in the k-step trajectory. From this perspective, we examine and compare the convergence rates of Lookaround and Lookahead.
We first show the state transition equation for the classical momentum method in quadratic functions:
Vt+1 fv. - Vof(O.) Bv. A6,,
(13)
0 0 + VV++ ,,v, + (I -I-740 A)),
(14)
Here, v stands for momentum term. We can generalize this matrix form:
Vr+1 Oc+1
A 1-7A 7
73
0
Thus, given the initial 0p. we can obtain the convergence rate with respect to 0 by the maximum eigenvalue of the matrix. Referring to Zhang et [49] and Lueas al. 30 we obtain the state transition matrix regarding our algorithm as follows:
0.0
2 1.0
0 0t-1,k
i 2
0,11,1
0,-2,1
where L, B and T denote the average weight matrix, the single-step transfer matrix and the position transformation matrix respectively:
k+1
1
L
0
0
0
17

---

(1+
8) - 7A
-81
I
B
0
:
0
AA
B1
31
0
T=
0
I 0
0
After specifying the appropriate parameters and performing matrix multiplication to obtain the state transition matrix, the convergence rate م can be obtained by calculating the largest eigenvalue of the matrix. Note that since this linear dynamical system corresponds to updates, we finally have to compute the &tth root of the eigenvalues recover the correct convergence bounds.
C Experimental Detail
C.1 Random Initialization
C.1.1 CIFARIO and CIFAR100
Data augmentation details. For the CIFARIO dataset, we use [RandomCrop + *] for data aug- mentation, and for the CIFAR100 dataset, use [RandomCrop + RandomRotation] for data augmentation. can be replaced by three different data augmentation methods of random horizontal flip, random vertical flip, RandAugment [4].
Training details. For the CIFAR1O and CIFAR100 datasets, we have applied some common settings. The initial learning rate is 0. and the batch size is 128. Additionally, warm-up phase of epoch is implemented. Subsequently, different learning rate schedulers are used based on the specific dataset. For the CIFAR 100 dataset, we utilize the MultiStepLR scheduler. The learning rate is decayed at the 60, 100, and 160 epochs, with decay factor of 0.2. For the CIFAR 10 dataset, we employ the CosineAnnea learning rate scheduler. comparison with other optimizers or optimization methods, we use the default settings of the other methods in the corresponding papers.
C.1.2 ImageNet.
For the ImageNet dataset, our data augmentation strategy involves [RandomResizedCrop + + RandomRotation) Here, can be substituted by one of the three data augmentation methods: random horizontal flip, random vertical flip, or RandAugment [4) We adopt the following training settings: an initial learning rate of 0.1 and batch size of 256. The model undergoes training for total of 100 epochs. Additionally, we make use of the MultiStepLR scheduler, which decays the learning rate by factor of 0.1 the 30th, 60th, and 90th epochs.
C.2 Finetuning
In this stage, all images are resized to 224 224 pixels to match the input size of the pretrained model. All models use the ImageNet- 1 pretrained weights soureed from the PyTorrch library. Throughout the finetuning process, we employed three data augmentation methods: random horizontal flip, RandAugment, and AutoAugment.
For the training of ResNet50, we performed grid search as shown in Table to determine the optimal learning rate and weight decay. Every training process lasted for 50 epochs with batch size
18

---

Table Hyperparameter search for finetuning on ResNet50.
Dataset
Epochs
Base LR
Weight Decay
CIFARIO
50
[0.001, 0.003, 0.01, 0.03) [0.001, 0.003, 0.01, 0.03)
(0.0001, 0.00005, 0.00001| (0.0001, 0.00005, 0.00001)
CIFAR100
50
Table 9: Comparison between different training strategies with ResNet18 on Stanford Cars dataset. Here "H" denotes random horizontal flip data augmentation. "R" denotes RandAugment data augmentation. "A" denotes AutoAugment data augmentation. "Logit Ensemble SWA" denotes logit ensemble approach that employs SWA models, which have been acquired through various data augmentation techniques. "T-Budget" and "I-Budget" respectively represent Training budget and
Inference budget.
Method
Data Augmentation Strategy
Top-1 Acc
Top-5 Acc
T-Budget
I-Budget
SGDM
"H" "H" "R" "R"
86.57
97.52
1x
1x
SWA
87.60
98.21
Ix
1x
SGDM
87.30
98.01
1x
1x
SWA
88.00
98.21
Ix
1x
SGDM
87.63
98.05
SWA
"A" "H" + "R"
88.11
98.15
IX
lx
Logit Ensemble
89.24
98.44
2X
2x
Lookaround
"H" + "R"
89.35
98.47
2x
1x
Logit Ensemble
"H" "R" "A"
90.19
98.67
3x
Lookaround
"H" "R"" + "AA
90.76
98.78
3X
1x
Logit Ens
"H"" + "R" "A" "H" + "R" "A"
90.39
98.80
3x
3x
Lookaroun+-SWA
91.04
98.86
3x
1x
set to 128. For the training ViT-B/16, we used the Adam optimizer, starting with learning rate of 0.00001. We set B1 to 0.9, 82 to 0.999, and applied weight decay of 0.01. To optimize memory usage, the batch size was set to 64.
C.3 Compared with Sharpness-Aware Minimization
We adopt the default hyperparameters from the origin paper of SAM 10 and combine them with the optimal hyperparameters found during random initialization or finetuning to conduct comparative experiments between SAM and Lookaround.
C.4 Compared with Ensemble Method
We compare Lookaround with Logit Ensemble and Snapshot Ensemble 18] In the setting of Logit Ensemble, we train multiple models separately using different data augmentation methods, and then average the outputs of these models for prediction. However, this approach requires more inference time. In the setting of Snapshot Ensemble, we use the CosineAnnealing waereAAnnelingaaRRestieee eermRsstarts learning rate scheduler collect four snapshots during the training process. Then, we average the outputs of these different snapshots for prediction. This approach also requires more inference time.
Besides, conducted full experimental validation of Logit Ensemble and SWA under dataset Stanford Cars, as shown in Table 9 Under the Stanford Cars dataset, Lookaround still performs better than Logit Ensemble. Additionally, adding SWA into the training process of Lookaround can further improve performance. Under the limit of single-model inference, Lookaround+SWl achieves Top-1 accuracy of 91.04% exceeding the optimal 88.1 11 % (SWA with AutoAugment).
C.5 Ablation Study
In ablation experiments. evaluate the individual contributions of two components: Data Augmentation (DA) and Weight Averaging (WA) to the effectiveness of the Lookaround. Below are the specific settings for different ablation experiments.
For the experiments without both DA and WA, three models were trained independently. each utilizing distinct data augmentation strategy, for duration of 200 epochs. After training, we identified
19

---

and reported the highest accuracy achieved by these models on the test set. In the experiments that employed only DA, we trained single model using all data, combining the three data augmentation strategies, without the use of weight averaging. For the experiments with only WA, three models were trained separately for 200 epochs each, employing different data augmentation strategies. Following this, we selected the model that showcased the best performance on the test set.
C.6 Additional Analysis
In the robustness experiments with the number of Data Augmentation methods, the six data augmen- tation methods are given as: KandomVerticclll ment, RandonPerspective, RandomEqualize. All the Augmentation methods are from the Py Torch library. When more data augmentation methods are used, the training time will be correspondingly increased in our method. Therefore, in this paper's main experiments, we only select three data augmentation methods for comparison to reduce the time consumption.
RandomHoorizotallFiip RandAugment, Aut
D Contemporaneous Work
After completing our work, we discovered similar work at CVPR 2023 within the same year. Inspired by similar motivations, the DART [20] method proposed by Samyak Jain et al. utilizes a similar training structure that includes different data augmentation training of multiple sub-models, with weight averaging occurring at intervals.
Methodology. DART starts weight averaging until the second half training process and adopts sparser weight averaging strategy. Meanwhile, the DART scheme uses the ERM [15] strategy to initialize the model. In contrast, Lookaround maintains a consistent and dense weight averaging strategy throughout its training, making it more frir-frienlly in real-world applications.
Theory. DART proves that the average while training is more robust from the perspective of noise feature. On the other hand, Lookaround proves that it can get lower expected loss under the setting of quadratic noise function.
Experiments. DART validates its effectiveness in domain generalization. Lookaround is tested in the scenarios of finetuming and training from scratch.
In conclusion, although sharing a similar idea, two methods differ substamtially in theoretical founda- tions and experimental approaches. Both methodologies are complementary and furnish significant insights for the broader research community.
And we compared Lookaround with DART with the same data augmenta- tion, the results shown in Table TO Our results were better than DART, the reason behind this may come from the fact that we used more frequent "average steps" during training and started "average steps" carlier in the cycle. More frequent "average steps" are more conducive to the training of Lookaround, referenced in the main body of the paper.
Table 10: Comparison with the DART method with ResNet18 on CIFARIO and CIFAR100. The results of DART! are taken from reference 20).
Method
CIFARIO
CIFARI00
Dant!
97.14
82.89
Lookaround
97.22
83.16
E
Relationship with Model Soups
Model Soups 46 framework for finetuning common pretrained model using different hy- perparameters and then averaging the weights of different finetuned models to improve model performance and generalization. In the finetuning settings, we conducted an in-depth comparison between Lookaround and Model Soups, with the related results presented in Table The "Update BN" suggestion originated from the reviewer's feedback. Given that we employed a CNN archi- tecture, the averaging of means and variances in the batch normalization (BN) layers of different models within Model Soups might lead to distortions, failing to accurately represent the statistical characteristics of the true hidden layer inputs. To address this, passed the averaged model through
20

---

Table 11: Comparisons between Lookaround, Model Soups on ResNet50. "M=18" represents 18 different hyperparameters. "M=3" represents that Lookaround is trained using different data augmentation techniques. 18 different configurations come from the combinations of three data augmentations (RandomHoriizontllFiin RandAugment, AutoAugment), three initial learning rates: {0.01, 0.003, 0.001} and label smooth or not. Uniform Soups represents that we use the top- 10 accuracy models for uniform soups (the accuracy of these models ranges from 83.91 % to 84.64%).
Method
Update BN
Top-1 I Acc
Top-5 Acc
Uniform Soups
82.93
97.44
Uniform Soups
84.53
98. 10
Uniform Soups*
84.60
97.81
Uniform Soups*
V
85.57
98.04
Greedy Soups
85.52
98.02
Greedy Soups
86.09
98.11
Lookaround (M=3)
85.20
97.15
the entire training set using forward propagation to individually update the BN layer's running mean and running var.
The experimental results indicate that the Lookaround method, employing only three types of data augmentation, still remains comparable performance to the Model Soups method, which utilizes 18 types of data augmentation. This further attests to Lookaround's generalization capability even under limited data augmentation conditions. Additionally, we observed that the Lookaround method is applicable to tasks with random initialization, while the Model Soups, due to its prerequisite of pretrained weights, cannot be employed in such scenarios.
F
Computational Complexity and Comparison
The Lookaround optimizer can be viewed as maintaining multiple backups of model weights during the optimization process. Each backup only trained under its corresponding data augmentation strategy (forward and backward propagation) and updates the parameters using weight averaging after multiple iterations.
In general, if the time for an iteration is defined as Ω and the time to perform a weight average is w, given that the dataset contains B batches, the standard time complexity SGDM to complete an epoch is O BQ)). The time complexity of Lookaround is OdBR B Given that w is much smaller than St, this can be approximated as (dd:B).
In our experiments (corresponding to Table 1 to 5 in the paper), to establish fair comparisons in computational among different approaches, we adopt the same augmentations for the competitors. Specifically, within a single epoch, both the proposed Lookaround and the competitors undergo training on an identical d times the data augmentations. With such a setup. we guarantee consistency in the data volume utilized by each method, thereby ensuring fair comparisons in terms of computation.
21


Reviews:
[{"review_id": "Vmifbn7PHv", "reviewer": "Reviewer_HsP8", "summary": "Flatness-aware optimizers have gained significant attention in the field of research for training deep neural networks that are robust. Weight Averaging (WA) is a popular approach to finding solutions within the flat regions of the loss surface. However, previous WA methods have two limitations. First, when WA is performed within a single optimization trajectory after training convergence, resulting in limited functional diversity among the averaged members. Second, when WA is performed across diverse modes, the averaged weights may negatively impact performance. To address these challenges, this paper introduces a new method called Lookaround Optimizer, which aims to overcome these limitations.", "questions": "Experiments\n\n- The response regarding concerns addressed in the Weakness section will be critical to the final decision.", "limitations": "Limitations are adequately addressed in the Limitation section.", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Originality\n\n- Proposed optimizer seems to be relatively simple and straightforward but still attractive.\n- This article presents a theoretical analysis regarding the variance of the steady state.\n\nClarity\n\n- The paper is written effectively, ensuring high accessibility for readers.\n- Methods are simple and easy to follow.", "weaknesses": "Experiments\n\n- Regarding the experiments conducted, the authors acknowledged in the Limitation section that their proposed method incurs additional training costs (I think the proposed optimizer requires nearly $d$ times more computational resources when averaging $d$ models). Therefore, for a fair comparison, it would be necessary for other baseline methods to undergo additional training epochs as well. However, it appears that the authors used the same number of training epochs for all the experiments.\n\n- The overall results of the experiments are rather counterintuitive. Particularly in Table 3, the reported accuracy of the Deep Ensemble method being lower than the accuracy of a single solution trained using the proposed method is quite unexpected. Similarly, in Table 1, the reported performance of flatness-aware optimizers such as SWA [1] and SAM [2] being lower than SGDM contradicts the knowledge and expectations of the research community.\n\nReferences\n\n[1] Pavel Izmailov, D. A. Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. Conference on Uncertainty in Artificial Intelligence, 2018.\n\n[2] Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. International Conference on Learning Representations, 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your comments on our work. We hope the following response will address your concerns.\n\n**Q1: Lookaround uses $d$ times more computation. How does Lookaround fairly compare to other methods?**\n\nWe agree that it is imperative to establish equitable comparisons in computational analysis among different approaches, which is precisely what we've accomplished. Given that the proposed Lookaround utilizes $d$ times the amount of data, we took care that in our primary experiments (corresponding to Table 1 to 4 in the paper), all other methods were also trained using $d$ times the data volume. Specifically, within a single epoch, both the proposed Lookaround and the competitors undergo training on an identical $d$ times the data augmentations. With such a setup, we guarantee consistency in the data volume utilized by each method, thereby ensuring fair comparisons in terms of computation. With the same amount of computation, Lookaround outperforms the competitors by a considerable margin, which demonstrates the superiority of Lookaround in efficiency.\n\nIn the limitation section, we mentioned that Lookaround is limited by the additional cost, considering that Lookaround usually needs more training time to converage than its competitors. However, if restricted to the same training buget, Lookaround still outperforms the others even it have not reached its own optimum. We will provide further clarity on this aspect in the revised paper.\n\n**Q2: In Table 3, the reported accuracy of the ensemble method being lower than the accuracy of a single solution trained using the proposed method is quite unexpected.**\n\nThanks for the comment. The proposed Lookaround, as previously mentioned, achieves remarkably superior performance with a single solution to the preceding ensemble method. In the case of Logit Ensemble, results are obtained by ensembling from *independently* trained models. The performance improvement through ensembling is solely derived from the diversity. On the contrary, Lookaround can be perceived as an ensemble (Recall that weight average approximates straightforward ensemble as proven in SWA) of *simultaneously and interdependently* trained models. During the training process, these models benefit from one another by weight averaging and converge towards flatter minima. The performance gain arises not only from the diversity (each model is trained on different augmentations), but also from the superiorioty of the base models. Hence, it is entirely rational that the proposed Lookaround surpasses simple ensembling methods, which should be considered an advantage rather than a disadvantage of our work.\n\n**Q3: Similarly, in Table 1, the reported performance of flatness-aware optimizers such as SWA and SAM being lower than SGDM contradicts the knowledge and expectations of the research community.**\n\nThanks for raising these concerns. Here we would like to make the following clarifications.\n- For the optimizer SWA, we would like to remind the reviewer that it outperforms SGDM in nearly all our experiments, as evidenced in Table 1, Table 2, and Table 4 in our paper. The only one exception arises in the case of the model VGG19 on CIFAR100, where SWA achieves slightly lower accurcy. These results, however, align quite consistently with the prevailing consensus within the research community.\n\n- For the optimizer SAM, we have carefully checked the experimental configurations and repeated the experiments three times. The results remain consistent across our repeated trials. We speculate that the heavy data augmentation employed in our experiments might be the predominant factor in the inferior performance of SAM. To ensure fair comparisons in computation, both Lookaround and the competitors, including SAM, are evaluated with heavy data augmentations (random horizontal flip+random vertical flip+RandAugment). The heavy data augmentations may serve as a potential distraction for SAM, as both techniques are proposed to enhance generalization. To verify this hypothesis, we conducted a sanity check of our code with only a single type of data augmentaion. The results are provided in Table S1. It can be seen that *SAM significantly ourperforms SGD with only one type of augmentation, which is still consistent with consensus of the research community*. \n    \n    Table S1: Comparison of different optimization methods for ResNet18 on various datasets. \"H\" denotes random horizontal flip data augmentation. \"R\" denotes RandAugment data augmentation.\n\n    |                  |  SGDM (H) |  SGDM \\(R\\) |  SAM (H)  |  SAM \\(R\\)  | Lookaround (R+H) | Lookaround+SAM (R+H)|\n    | :--------------: | :---: | :---: | :---: | :---: | :--------: | :------------: |\n    |     CIFAR100     | 75.72 | 76.14 | 76.84 | 77.08 |   77.20    |     77.55      |\n    |    Flowers102    | 95.93 | 96.54 | 97.01 | 97.35 |   97.65    |     97.75      |\n    | Stanford cars196 | 86.77 | 87.02 | 87.53 | 87.82 |   89.35    |     89.67      |\n\n\nFurthermore, the results listed in Table 1 and Table S1 implies a potential merit of the proposed Lookaround: it is orthogonal to SAM and heavy data augmentation, and can work compatibly with both, as validated by *ours* (Lookaround+heavy augmentation) in Table 1, and *Lookaround+SAM* in Table S1. We hope our response can successfully address your concerns and strengthen your confidence in our work."}, {"review_id": "P14Ryyx1gy", "reviewer": "Reviewer_vocv", "summary": "This article introduces the Lookaround Optimizer, a novel optimization algorithm for deep neural networks. The Lookaround Optimizer is based on the idea of lookaround, which involves maintaining two sets of weights. The first set of weights is updated using the standard gradient descent algorithm, while the second set of weights is updated using the averaged gradients of the first weight. The Lookaround Optimizer has been shown to improve the generalization performance of deep neural networks, and it outperforms other state-of-the-art optimization algorithms on a variety of benchmark datasets. The authors also provide theoretical analysis of the Lookaround Optimizer. Overall, the Lookaround Optimizer is a promising approach for improving the training of deep neural networks.", "questions": "See above", "limitations": "The authors have addressed the limitation.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Lookaound Optimizer performs better than other state-of-the-art optimization algorithms (Lookahead, SAM) on multiple benchmark datasets.\n- The proposed optimizer seems to be model-free and can be applied to various computer vision scenarios.\n- The paper is well-written and easy to understand.", "weaknesses": "- While the authors have demonstrated fascinating performance on benchmarks, the technical innovation of this paper is limited. Similar ideas that utilize multiple models have already in proposed in varioius area, especially in meta learning. \n- With that being said, while the authors have mentioned the connections to these related topics, they did not compare with some of them. For example, I think at least the authors should compare the performance of Lookaround Optimizer with Model Soups (i.e., train multiple copies of models using the same initialization, the same order of data, but with different augmentation), and other model merging techniques like Model Ratatouille. \n- The training time will be a bottleneck for applying this method as the authors  acknowledged. \n- Another limitation is that, the augmentation seems to be a must for this method. Therefore, it seems that this method is not applicable (at least not straightforward) when we try to apply it on other modality, such as language and speech.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments and suggestions. In the following, your comments are first stated and then followed by our point-by-point responses.\n\n**Q1:  Similar ideas that utilize multiple models have already in proposed in varioius area, especially in meta learning.**\n\nThanks for the feedback. Broadly speaking, the goal of meta-learning is to train a model on a diverse range of learning tasks, such that it can quickly solve new learning tasks. To the best of our knowledge, the closest work with the proposed Lookaround is perhaps the MAML-based approaches. However, Lookaround distinguishes itself significantly from MAML in at least the following two aspects:\n* **Problem Settings.** MAML is motivated by learning the common knowledge across tasks, such that new tasks can be quickly solved. In other words, MAML typically assumes the availability of massive tasks for training, with the ultimate goal of solving new tasks with only a few instances. Lookaround, like SGD, Lookhead, or some other optimizers, focuses on addressing general optimization problems.\n* **Methodologies.** MAML trains the initial parameters of the model to maximize its performance on a new task after updating the parameters through one or more gradient steps, utilizing a small amount of data from that particular task. In contrast, Lookaround simultaneously trains multiple models on different augmentations and periodically averages their weights to obtain a final model with enhanced generalization.\n\n**Q2: Compare the performance of Lookaround optimizer with Model Soups and Model Ratatouille.**\n\nWe greatly appreciate the reviewer's suggestion. Here we provide the experimental results, details and discussions as follows.\n\nTable S1: Comparisons between Lookaround, Model Soups, and Model Ratatouille on ResNet50. \"M=18\" represents 18 different hyperparameters. \"M=3\" represents that Lookaround is trained using 3 different data augmentation techniques.\n|   Optimizer    |        Method (M=18)        | CIFAR100  |\n| :------------: | :-------------------------: | :-------: |\n|      SGDM      |    Model Soups (Greedy)     |   82.07   |\n|      SGDM      |    Model Soups (Uniform)    |   1.00    |\n|     AdamW      |    Model Soups (Greedy)     |   79.18   |\n|     AdamW      |    Model Soups (Uniform)    |   78.28   |\n|     AdamW      | Model Ratatouille (Greedy)  |   80.31   |\n|     AdamW      | Model Ratatouille (Uniform) |   1.58    |\n|   Lookaround |      Lookaround (M=3)       | **83.62** |\n\n**Experimental Details**：18 different configurations come from the combinations of three data augmentations (RandomHorizontalFlip, RandAugment, AutoAugment), three initial learning rates: ($\\{0.2, 0.1, 0.05\\}$ for SGDM, $\\{5e^{-5},2e^{-5},e^{-5}\\}$ for AdamW) and label smooth or not. For Model Ratatouille, we choose Stanford cars196, Flowers102, and CIFAR10 as auxiliary domains, and CIFAR100 as the target domain for further fine-tuning.\n\n**Result Discussions**: From Table S1, it can be easily seen that Lookaround *outperforms Model Soups and Model Ratatouille* significantly. Another notable result is that both Model Soups and Model Ratatouille easily break down in uniform averaging strategies (Model Soups with SGDM achieves $1\\%$, and Model Ratatouille with AdamW achieves $1.58\\%$). In contrast, Lookaround periodically averages the weights during the entire training process, which effectively bypasses the pitfalls of weight averaging between independently trained models.\n\nWhile Lookaround outperforms Model Soups and Model Ratatouille, it is still necessary to note that *Lookaround is proposed for a quite different problem setting compared to the other two methods*. Model Soups and Model Ratatouille are designed for more efficient utilization of existing models, assuming that these models have already been pre-trained on specific datasets. Lookaround, on the other hand, serves as an optimizer for training deep models, devoid of any assumptions regarding the initialization of the model parameters. \n\n**Q3: The training time will be a bottleneck for applying Lookaround.**\n\nWe agree that it is imperative to establish equitable comparisons in computational analysis among different approaches, which is precisely what we've accomplished. Given that the proposed Lookaround utilizes $d$ times the amount of data, we took care that in our primary experiments, all other methods were also trained using $d$ times the data volume. Specifically, within a single epoch, both the proposed Lookaround and the competitors undergo training on an identical $d$ times the data augmentations. With such a setup, we guarantee consistency in the data volume utilized by each method, thereby ensuring fair comparisons in terms of computation. \n\nIn the limitation section, we mentioned that Lookaround is limited by the additional cost, considering that Lookaround usually needs more training time to converage than its competitors. However, if restricted to the same training budget, Lookaround still outperforms the others even it have not reached its own optimum. We will provide further clarity on this aspect in the revised paper.\n\n**Q4:  It seems that this method is not applicable on other modality.**\n\nWe wholeheartedly agree that exploring the application of Lookaround in other modalities, such as NLP, holds great promise. Indeed, there are existing methods of data augmentation in language tasks, such as EDA[1], TTA[2]. While it may not be a direct and straightforward adaptation, the underlying idea of Lookaround can be explored and applied in innovative ways to these modalities. We eagerly anticipate future advancements and the emergence of new approaches that harness the concept of Lookaround in diverse modalities.\n\n[1] Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks, EMNLP-IJCNLP.\n[2] Text AutoAugment: Learning Compositional Augmentation Policy for Text Classification, EMNLP."}, {"review_id": "JK0HxiIseK", "reviewer": "Reviewer_vFuf", "summary": "This paper proposes Lookaround, a new optimization method that incorporates weight averaging into the optimization process. The algorithm consists of two steps: 1) the around step launches several parallel runs of gradient descent led by different data augmentations, 2) the average step does weight averaging of the networks obtained by these parallel runs. These two steps are repeated along the whole training process, with the result of the average step being a starting point for the next around step. The proposed method is similar to Stochastic Weight Averaging and Model Soups, though it does the averaging during training rather than at the end of training. Lookaround shows strong results compared to the existing optimization methods (SGD with momentum, AdamW, Lookahead, SAM).", "questions": "1. What is the augmentation policy during baseline training? \n2. I do not understand the experimental setup in the ablation study (Section 4.4). Is the training without DA and WA similar to the regular training without augmentations? Is the training without DA but with WA similar to Lookaround, where the ordering of batches is the only source of randomness between branches?\n3. Section 3.2.2, convergence on deterministic quadratic functions. How branches of Lookaround are different from each other if there is no randomness in gradient steps?\n4. Could the authors elaborate on how the Lookaround method relates to the methods proposed in the contemporaneous works (https://arxiv.org/abs/2302.14685, https://arxiv.org/abs/2304.03094)?", "limitations": "The main limitation of the proposed method is the increased training budget, which is highlighted by the authors. However, there is no fair comparison to networks trained for a larger number of epochs.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The idea of using weight averaging during optimization with various augmentations is novel and leads to a better generalization of neural networks.\n2. This paper has a theoretical analysis of the quadratic noise setup which demonstrates faster convergence of Lookaround compared to SGD and Lookahead.", "weaknesses": "Despite the fact that the paper proposes a novel and interesting method, in my opinion, it is not quite ready for publishing in its current form. I believe that addressing the following concerns could help to make it a much stronger submission.\n1. The text of the paper is hard to follow, contains some vague parts and numerous typos.\n    - The theoretical reasoning is heavily inspired by the Lookahead paper, which makes it impossible to understand the theory without reading the original paper (e.g., I could not understand Proposition 1 and the definition of $\\alpha$ without reading the Lookahead paper). Moreover, there is no remark that equations 4 and 5 are derived in the Lookahead paper. \n    - Line 5-6: weight averaging and ensembles are mixed up. \n    - Line 313: it is not clear how gradient boosting is related to this setup.\n2. I have found two contemporaneous works published on arxiv (https://arxiv.org/abs/2302.14685, https://arxiv.org/abs/2304.03094) that propose approaches very similar to Lookaround. I believe they need to be at least discussed in the related work section.\n3. The experimental part of the paper raises some questions and is not convincing enough, in my opinion.\n    - The main results (Table 1) lack standard deviations, even though they are claimed in the OpenReview form. \n    - The baselines seem too weak, i.e., the Lookahead paper reports >75% ImageNet test accuracy on ResNet-50 for SGD baseline, while this paper shows 72.27%. \n    - The augmentation policy during baseline training is not clear. If the same set of augmentations as for Lookaround is used, then this may be the reason for the bad quality of the baselines. \n    - Despite the paper proposing a new optimization method, it lacks experiments on a wider choice of architectures. Table 1 illustrates only VGG-19 and four networks from the ResNet family. It would be beneficial to add experiments on more modern convolutional architectures and more extensive experiments on image transformers.\n    - The paper lacks a fair comparison to logit ensembles. If I understood correctly, different models of the ensemble are trained with different Lookaround augmentations. This leads to the suboptimal quality of each model and, as a result, to a suboptimal ensemble.\n    - The comparison to model soups in Appendix D is not fair as well, i. e., the original paper fine-tunes models with various hyperparameters and does a greedy search for the combination with the best validation accuracy. Near zero accuracy of the average of $\\theta_1$ and $\\theta_2$ may be due to improper fine-tuning hyperparameters. Moreover, this is an important baseline, and it would be better to move this comparison to the main part of the paper.\n    - A proper ablation study of augmentations is required. Is it possible to train Lookaround without augmentations for the around step “branches” (e.g., the only source of randomness is batch ordering)? What if each branch utilizes the same set of augmentations? Why are the mentioned augmentations used (and not some other ones, e. g. vertical flip seems to be a strange augmentation)? Probably, some of these setups are covered in Section 4.4, but it is not clear from the text.\n    - Top-5 accuracy is redundant and makes it difficult to read the tables.\n4. Minor issues/typos:\n    - Line 25: the sentence is about LMC, but the citation [7] leads to the paper about permutations (Entezari et al., 2022). \n    - Line 112-113: we provide\n    - Line 147-148: repetition\n    - Line 156: we analyze\n    - Line 162: it seems that it should be $\\mathbb{E} [c_i^2]$\n    - Line 242: CIFAR", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed comments. We hope the following response will address your concerns.\n\n**C1: Two contemporaneous works propose approaches very similar to Lookaround.**\n\nThanks for sharing the two great works! After carefully reading the two papers, we found these three works share moderate similarities, meanwhile with some notable differences:\n\n**Methodolegy.** DART starts weight averaging until the second half training process. PAPA do weight averaging more frequently like Lookaround throughout the training process. However, PAPA combines the weight of the sub-model with the average weight. The weighting factor varies in different training stages. Such a strategy is more sophisticated, yet introduces more hyperparameters, which increases the complexity of the method. Lookaround adopts a consistent weight averaging strategy throughout the training process, making it easier to use in practice.\n\n**Theory.** DART proves that the average while training is more robust from the perspective of noise feature. Lookaround proves that it can get lower expected loss under the setting of quadratic noise function. PAPA mainly focuses on empirical validation, without any theoretical analysis.\n\n**Experiments.** PAPA mainly makes comparisons with Model Soups. DART validate its effectiveness in domain generalization. Lookaround is tested in the senarios of finetuning and training from scratch. \n\nIn conclusion, although sharing a similar essence, three methods differ substantially in theoretical foundations, experimental approaches, and method instantiations. They complement one another and each offers noteworthy contributions to the research community.\n\n**C2: The experimental part of the paper raises some questions.** \n\n**The standard deviations**. The error bar (i.e., standard deviations) is depicted in Figure 6 in our paper. The error bar of Lookaround trained from scratch are provided in Table S1. We will make it clearer in the revised version.\n\nTable S1: Top-1 accuracy of Lookaround (with standard deviation).\n|Method|Dataset|VGG19|Resnet50|Resnet101|Resnet152|ResNext50|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Lookaround|CIFAR10|94.42 ($\\pm$0.09)|96.56 ($\\pm$0.02)|96.77 ($\\pm$0.08)|96.94 ($\\pm$0.12)|96.67 ($\\pm$0.11)|\n|Lookaround|CIFAR100|74.2 ($\\pm$0.25)|81.12 ($\\pm$0.63)|81.89 ($\\pm$0.53)|81.99 ($\\pm$0.24)|81.17 ($\\pm$0.51)|\n\n**The augmentation policy and the ImageNet accuracy**: For the sake of fair comparisons in training data and computation cost, each method (Lookaround and its competitors) was trained with the same data augmentations (random horizontal flip, random vertical flip, and RandAugment). With such a setup, we guarantee consistency in the data volume utilized by each method, thereby ensuring fair comparisons in terms of computation. The divergence in performance of Lookahead can be attributed to the differing data augmentation techniques employed.\n\n**More modern architectures**: We have tested the proposed Lookaround optimizer with diverse and popular model architectures including VGG19, ResNet, ResNext50 and ViT-B/16. We believe these representative achitecture are sufficient to validate the effectiveness of the proposed method across architectures.\n\n**Fair comparison to logit ensembles:** We have tried different training strategies for the base models in Logit Ensemble, including training different models with different augmentations, and training all the base models with the same mixed augmentations. In our experiments, no matter how the base models are trained, Lookaround exhibit consistently superior performance to logit ensembles. We will revise the paper to make this point clearer.\n\n**The comparison to model soups:** Here we would like to remind the reviewer that Lookaround is proposed for a quite different problem setting compared to Model Soups. Model Soups is designed for more efficient utilization of existing models, assuming that these models have already been pre-trained on specific datasets. Lookaround, on the other hand, serves as an optimizer for training deep models, devoid of any assumptions regarding the initialization of the model parameters. This distinction is why we included the comparison with Model Soups in the supplementary material. In terms of fairness, we have conducted experiments using both greedy search and uniform search. The results are presented in Table S2, where Lookaround outperforms the greedy Model Soups, further validating the effectiveness of our approach.\n  \nTable S2: Comparison between Lookaround and Model Soups. \"M=18\" represents 18 different hyperparameters in Model Soups. \"M=3\" represents 3 different data augmentation techniques.\n|Optimizer|Method (M=18)|CIFAR100|\n|:-:|:-:|:-:|\n|SGDM|Model Soups (Greedy)|82.07|\n|SGDM|Model Soups (Uniform)|1.00|\n|AdamW|Model Soups (Greedy)|79.18|\n|AdamW|Model Soups (Uniform)|78.28|\n|**Lookaround**|Lookaround (M=3)|**83.62**|\n\n**Ablation study**: In Section 4.4 of the paper we set up the exact experiment you're talking about. The only WA experiment represents that each branch of Lookaround utilizes the same set of augmentation. We use the random vertical flip method as it can bring more image diversity, which is conducive to the guide model to have the characteristics of rotation invariance. We will specify these differences in the revised version.\n\n**C3: How branches of Lookaround are different from each other if there is no randomness in gradient steps?**\n\nThis issue can be referred to in Appendix B.2. Here, we have made an approximate calculation. The idea behind Lookaround is to choose models from different branching points in the loss landscape for weight averaging. In this deterministic quadratic function, we chose model points corresponding to every point in the $k$-step trajectory for weight averaging.\n\n**C4: The text of the paper is hard to follow.**\n\nWe are sorry for any confusion caused by our writing. We will fixed the issues you mentioned in our revisions."}, {"review_id": "1YX6xmJa2h", "reviewer": "Reviewer_d83u", "summary": "This paper introduces a new optimization algorithm named Lookaround, which draws inspiration from the recent achievements of weight averaging (WA) techniques in deep learning. The proposed Lookaround optimizer looks around nearby points by performing multiple gradient computations for a given training input using different data augmentations and averages them to obtain better generalizing solutions.", "questions": "There are several issues in the experimental results of the current version of the paper as I mentioned above, and I would respectfully suggest the followings that could make the paper solid:\n\n* __The absence of SAM in Table 4.__  \nIs there a specific reason for excluding SAM from the results presented in Table 4?\n\n* __The disparity in results for ResNet50 on CIFAR100 between Tables 1 and 6.__  \nTable 1 presents Top-1/5 accuracies of 81.60/95.99, while these values are not reflected in Table 6.\n\n* __Top-5 accuracy does not provide a meaningful comparison in the main tables.__  \nIt is suggested to exclude the Top-5 accuracy metric from the main results presented in Tables 1 and 4 for CIFAR10/100.  Specifically, in the case of CIFAR-10, evaluation metrics where all techniques achieve 99.9% accuracy hold little significance. Instead, including the Top-5 accuracy in Table 2 would be more appropriate, as it provides a more meaningful performance evaluation for the ImageNet experiments.\n\n* __It would be nice to provide uncertainty metrics.__  \nWhile the Top-1 accuracy is the main evaluation metric for classification, it would be advantageous to incorporate additional metrics that can effectively demonstrate the superiority of the proposed algorithm. For instance, including the negative log-likelihood (NLL) as a measure of in-domain uncertainty would provide valuable insights into the algorithm's performance.\n\n* __It would be beneficial to offer an analysis of the training costs involved.__  \nSeveral algorithms, including the proposed one, impose additional computational burdens, which could impact their practicality (e.g., SAM requires performing double forward and backward passes; SWA requires an additional model copy in memory). Providing detailed information about the specific computational requirements of each algorithm would greatly benefit future researchers and engineers. The most straightforward analysis is measuring wall-clock time for training. It would be nice to replace the meaningless \"#param.\" column with the training runtime in Table 4.", "limitations": "The authors stated the additional training cost incurred by the proposed algorithm.\n\n---\n__References:__  \nZhang et al. (2019). _Lookahead optimizer: k steps forward, 1 step back_.  \nIzmailov et al. (2018). _Averaging weights leads to wider optima and better generalization_.  \nForet et al. (2021). _Sharpness-aware minimization for efficiently improving generalization_.", "rating": 5, "confidence": 3, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "The theoretical analysis in this paper largely relies on Zhang et al. (2019), including noisy quadratic analysis and deterministic quadratic convergence. While it does not introduce a novel form of analysis, it is significant as it establishes a theoretical foundation within the existing framework.\n\nFor experiments, the baseline comprises commonly employed optimization techniques, such as SWA (Izmailov et al., 2018) and SAM (Foret et al., 2021), which are widely accepted as standard approaches in the field. In addition to convolutional neural networks of various scales, the authors also considered experiments on vision transformers.", "weaknesses": "Experimental issues:\n\n* __There are no error bars.__  \nWhile the authors answered \"yes\" for \"Error Bars\", the paper only provide a single value across the tables and figures. It is unclear how many experiments were conducted to derive those values. It would be preferable if the authors also included averages accompanied by standard deviations to provide a more comprehensive representation of the experimental results.\n\n* __The outcomes obtained from the ImageNet experiments appear to be strange.__  \nDespite the authors' efforts to demonstrate the scalability of the proposed algorithm, the Top-1 accuracy of 72.27% reported in Table 2 appears to be comparatively low. After reviewing Appendix C.1.2, it seems that the experimental setup follows the PyTorch convention (https://github.com/pytorch/examples/tree/main/imagenet), except for some additional augmentations. It is widely recognized that the ResNet-50 model typically achieves an accuracy of approximately 76% on the ImageNet dataset using this standard setup, which significantly differs from the 72.27% accuracy reported by the authors. It would be beneficial to provide clarification on the reasons behind the considerable performance drop observed in the ImageNet results.\n\nPractical issues:\n\n* __Excessive training costs incurred by the proposed algorithm.__  \nThe Lookaround algorithm, as proposed, demands $d$ times the number of forward and backward passes for each optimization step. This is considerably higher compared to SAM (Foret et al., 2021), which only requires twice the number of passes. One could argue that the training epoch is effectively enlarged by a factor of $d$ and this is the actual reason for the performance improvements. It would be valuable to present results akin to Table 2 in Foret et al. (2021), that is, exploring the impact of increasing the number of training epochs through an ablation study.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments. In the following, your comments are first stated and then followed by our point-by-point responses.\n\n**Q1: There are no error bars.**\n\nThe error bar (i.e., standard deviations) is depicted in Figure 6 in our paper. The detailed results are provided in Table S1. We will make it clearer in the revised version.\n\nTable S1: Top-1 accuracy of Lookaround (with standard deviation).\n|   Method   | Dataset|VGG19|Resnet50|Resnet101|Resnet152|ResNext50|\n| :--------: | :------: | :---------------: | :---------------: | :---------------: | :---------------: | :---------------: |\n| Lookaround | CIFAR10  | 94.42 ($\\pm$0.09) | 96.56 ($\\pm$0.02) | 96.77 ($\\pm$0.08) | 96.94 ($\\pm$0.12) | 96.67 ($\\pm$0.11) |\n| Lookaround | CIFAR100 | 74.2 ($\\pm$0.25)  | 81.12 ($\\pm$0.63) | 81.89 ($\\pm$0.53) | 81.99 ($\\pm$0.24) | 81.17 ($\\pm$0.51) |\n\n**Q2: The outcomes obtained from the ImageNet experiments appear to be strange.**\n\nFor the sake of fair comparisons in training data and computation cost, each method (including Lookaround and the competitors) was trained with the same data augmentations (random horizontal flip, random vertical flip, and RandAugment). With such a setup, we guarantee consistency in the data volume utilized by each method, thereby ensuring fair comparisons in terms of computation. The divergence in performance observed between Lookahead, as reported in the original paper and the current study, can be attributed to the differing data augmentation techniques employed.\n\n**Q3: Excessive training costs incurred by the proposed algorithm.**\n\nIn order to compare the performance of Lookaround and standard SGDM training more fairly, we shorten the number of epochs of Lookaround training to 1/4 of SGDM. As shown in Table S2, Lookaround's performance is still competitive.\n\nTable S2: Comparison between SGDM and Lookaround with ResNet50 on CIFAR100. Here \"H\" denotes random horizontal flip data augmentation. \"V\" denotes random vertical flip data augmentation. \"R\" denotes RandAugment data augmentation.\n|    Method   | Data Augmentation Strategy | Epoch | CIFAR10 | CIFAR100 | Training time |\n| :--------: | :------------------------: | :---: | :-----: | :------: | :-----------: |\n|SGDM|\"R\"|200|95.57|78.59|1 Budget|\n|SGDM|\"H\" + \"V\" + \"R\"|200|95.96 |79.61|3 Budget|\n| Lookaround |\"H\" + \"V\" + \"R\"|50|95.82 |78.62|3/4 Budget|\n| Lookaround |\"H\" + \"V\" + \"R\"|200|**96.59** |**81.60**|3 Budget|\n\n**Q4: The absence of SAM in Table 4.**\n\nWe supplement the SAM experiments of ResNet50 and ViT-B/16 under pre-training on Table S3. In the pre-training experiment, Lookaround and SAM significantly improve the standard training process, among which SAM is more suitable for ViT architecture, while Lookaround is more suitable for CNN architecture. At the same time, the two are orthogonal, and combining the two can achieve higher performance improvement.\n\nTable S3: Top-1 accuracy of SAM and Lookaround on CIFAR dataset under pre-training. \n|Backbone |Method|  CIFAR10  | CIFAR100  |\n|:------: | :------------: | :-------: | :-------: |\n|ResNet50 |SGDM|96.08|82.04|\n|ResNet50 |SAM|97.02|82.75|\n|ResNet50 |Lookaround|96.79|83.62|\n|ResNet50 |Lookaround+SAM|**97.53**|**83.91**|\n|ViT-B/16 |Adam|92.91|74.50|\n|ViT-B/16 |SAM|98.02|89.13|\n|ViT-B/16 |Lookaround|95.23|78.38|\n|ViT-B/16 |Lookaround+SAM|**98.84**|**92.04**|\n\n\n**Q5: The disparity in results for ResNet50 on CIFAR100 between Tables 1 and 6.**\n\nThe result differences of ResNet50 on CIFAR100 in Table 1 and Table 6 come from the different settings of $k$. In Table 1, the choice of $k$ is 5, meaning an average is taken every 5 batches. In Table 6, the parameter $k$ we chose is \"one-epoch\", indicating that in Lookaround, different branches train for a complete epoch each before performing a weight averaging. Thank you very much for pointing this out, we supplement this experiment in Table S4.\n\nTable S4: Top-1 accuracy (%) of different data augmenta-\ntion (DA) number by using ResNet50 on CIFAR100 dataset.\n|# of DA|1|2|3|4|5|6|\n|:-------:|:--:|:--:|:--:|:--:|:--:|:--:|\n|Top-1 (%)|78.2|80.82|81.60|81.19|81.74|82.02|\n|Top-5 (%)|94.5|95.19|95.99|95.65|95.85|96.02|\n\n**Q6, Q7: Top-5 accuracy does not provide a meaningful comparison in the main tables. It would be nice to provide uncertainty metrics.**\n\nThank you very much for your suggestion. We will revise the paper accordingly following your suggestions. We present the NLL loss for the test set for part of the experiment, and the data in this table correspond to the CIFAR100 dataset in Table 1 in the paper. \n\nTable S5: NLL Loss values of different methods under ResNet family.\n|Method|ResNet50|ResNet101|ResNet152|ResNext50|\n|:--------:|:-------:|:-------:|:-------:|:-------:|\n|SGDM|0.899|0.894|0.895|0.820|\n|SWA|0.858|0.856|0.856|0.796|\n|Lookahead|0.836|0.825|**0.809**|0.787|\n|Lookaround|**0.81**|**0.823**|0.823|**0.73**|\n\n**Q8: It would be beneficial to offer an analysis of the training costs involved.**\n\nIn the experiment of comparing Lookaround with other optimization methods, we train each method with the same number of epochs, and the corresponding data of each epoch comes from the sum of $d$ times of augmentation data. Therefore, non-SAM methods will have the same number of forward and back propagation, while SAM requires twice as many forward and back propagation. Although each optimizer will update momentum or average weight, the time consumption of this step in the overall training is very small. \n\nExperimentally, we run each method for 50 epochs and calculate the average execution time of ResNet50 for one epoch on a 3090 graphics card on CIFAR100. The results are shown in Table S6. \n\nTable S6: Average execution time of each method traversing an epoch on ResNet50 on CIFAR100.\n|Method| SGDM | Lookahead | AdamW | SAM  | Lookaround |\n| :------------: | :--: | :-------: | :---: | :--: | :--------: |\n| Execution Time ($s$)|123|123|125|246|123|"}, {"review_id": "GqNNN3cZ7B", "reviewer": "Reviewer_V9QB", "summary": "This work provides a new optimizer, \"Lookaround optimizer,\" which is built upon a previous proposed \"Lookahead optimizer\" [40]. By incorporating data augmentation, this work shows an improved convergence rate under low condition numbers. It also empirically shows some improvement in classification accuracy under several datasets.", "questions": "What is the d (number of data augmentation) used in the experiment? I cannot find it in the context.", "limitations": "The author addresses that the data augmentation process consumes extra time, which is likely to be the main limitation.", "rating": 6, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "This work proposes a new optimizer and shows consistent (despite little) improvement in various settings. The supplementary material shows a solid derivation.", "weaknesses": "1. The major concern lies in the limited improvement over the \"Lookahead optimizer.\" The improvement in Figure 2 and Table 1-3 all seems to be subtle. Since the proposed methods see more augmented data in each epoch than others, it is questionable whether the improvement actually comes from these additionally augmented data. Also, the idea itself is almost identical to the Lookahead optimizer, so the novelty is rather limited.\n\n2. Minor mistakes: In table 1, row CIFAR 10, column ResNext50 Top5 results, the best results happen in Lookahead (99.96), not the proposed method (99.95).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your encouraging words and constructive comments. We sincerely appreciate your time in reading the paper, and our point-to-point responses to your comments are given below.\n\n**Q1：What is the d (number of data augmentation) used in the experiment?**\n\nIn the experiments corresponding to Table 1, Table 2, and Table 4, we adopted a setting with $d=3$. The three data augmentation methods are RandomHorizontalFlip, RandomVerticalFlip, and RandAugment. This information can be found in lines 221-223 of the paper and also in the appendix C.\n\n**Q2: Whether the improvement in accuracy comes from these additionally augmented data.**\n\nWe place great emphasis on experimental settings, ensuring that the improvements observed in the experiments are primarily attributed to our method and not just to data augmentation. Specifically, within a single epoch, both the proposed Lookaround and the competitors undergo training on an identical $d$ times the data augmentations. With such a setup, we guarantee consistency in the data volume utilized by each method, thereby ensuring fair comparisons in terms of computation.\n\nThe table below presents the experimental results of various networks on the CIFAR100 dataset. In this context, H(data), V(data), and R(data) each represent specific data augmentation applied to the original data. H(data) + V(data) + R(data) signifies training for one epoch after combining these three types of data augmentations. Throughout the 200 epochs of experimentation, Lookaround, a variant of SGDM, demonstrated significant performance improvements.\n\nTable S1: Top-1 accuracy of different networks under CIFAR100. Here \"H\" denotes random horizontal flip data augmentation. \"V\" denotes random vertical flip data augmentation. \"R\" denotes RandAugment data augmentation.\n|       Per epoch data        | Method | Vgg19 | ResNeXt50 | ResNet50 | ResNet101 | ResNet152 | Execution Time | amount of data |\n| :-------------------------: | :---------------: | :---: | :-------: | :------: | :-------: | :-------: | :------------: | :------------: |\n| H(data) + V(data) + R(data) |       SGDM        | 73.84 |   79.10   |  79.61   |   79.91   |   80.16   |       3x       |       3x       |\n| H(data) + V(data) + R(data) |    Lookaround     | 74.29 |   81.14   |  81.60   |   81.97   |   82.22   |       3x       |       3x       |\n\n**C3: Minor mistakes.**\n\nThank you very much for pointing out this issue. We will fix it in the revised version."}, {"review_id": "KmtkPfEgra", "reviewer": "Reviewer_3YPD", "summary": "The paper presents Lookaround, a novel optimizer for weight average ensembling (WA). Unlike existing approaches that perform weight averaging post-training, Lookaround adopts a two-step process throughout the training period. In each iteration, the \"around\" step trains multiple networks simultaneously on transformed data using different augmentations, while the \"average\" step combines these networks to obtain an averaged network as the starting point for the next iteration.The approach demonstrates clear superiority over state-of-the-art methods in extensive experiments on CIFAR and ImageNet datasets using both traditional CNNs and Vision Transformers. The paper provides theoretical justification and commits to open science by making the code publicly available.", "questions": "1. With the Lookaround optimizer requiring k around steps and 1 average step, I am unclear about how the epoch and iteration are defined in this setting. Are they counted based on the average step or the around step? Could you provide clarification on how the epoch and iteration are structured in Lookaround?\n\n2. Considering that each batch is augmented k times in Lookaround, I am curious to understand how this approach differs from simply applying `repeated augmentations`[A] to each batch. What distinguishes the effect of k augmentations and 1 step update in Lookaround versus performing each step update individually and then averaging all checkpoints? In other words, what advantages does Lookaround's approach provide in terms of diversity and ensemble performance compared to the alternative method of averaging checkpoints from individual updates?\n\n[A] Hoffer E, Ben-Nun T, Hubara I, et al. Augment your batch: Improving generalization through instance repetition[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020: 8129-8138.", "limitations": "As discussed in the limitation section \"additional cost of network searching using different data augmentation, resulting in a longer training time proportional to the the number of trained networks\", the running time is a important concern. ", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. `Innovative Approach`: The introduction of Lookaround as a straightforward and effective optimizer for weight average ensembling brings a novel perspective to the field. The two-step process during training enhances network diversity and preserves weight locality, addressing the limitations of post-hoc weight averaging approaches.\n\n2. `Theoretical Justification`: The paper offers strong theoretical support for the superiority of Lookaround through convergence analysis. Proposition 1 provides insights into the convergence variance of Lookaround in comparison to typical SGD and Lookahead optimizers. This rigorous theoretical analysis enhances the credibility and significance of the proposed method, demonstrating a solid foundation for its effectiveness and performance.\n\n3. `Extensive Experimental Validation`: The authors conduct extensive experiments on popular benchmarks, including CIFAR and ImageNet, with both traditional CNNs and Vision Transformers (ViTs). The clear superiority of Lookaround over state-of-the-art methods on these datasets demonstrates its effectiveness and applicability.\n\n4. `Commitment to Open Science`: The authors state their intention to make the code publicly available, fostering reproducibility and enabling further research in the field.", "weaknesses": "1. `Computational Complexity and Comparison`: The paper lacks a comprehensive discussion of the computational complexity introduced by the Lookaround optimizer. As Lookaround incorporates an additional around step during training, it is crucial to assess its computational requirements and compare them to existing optimization methods. A detailed analysis of the computational trade-offs, including runtime and memory usage, would provide a more comprehensive understanding of Lookaround's practical applicability and scalability.\n\n2. `Limited Experiments on Datasets Beyond Image Classification`: The paper primarily focuses on evaluating Lookaround on image classification tasks using datasets like CIFAR and ImageNet. However, to establish the versatility and effectiveness of Lookaround, it is important to explore its performance on datasets beyond image classification. Conducting experiments on diverse tasks such as language modeling or segmentation/detection tasks would demonstrate the generalizability of Lookaround across various domains and provide a more comprehensive evaluation of its performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your encouraging words and constructive comments. We sincerely appreciate your time in reading the paper, and our point-to-point responses to your comments are given below.\n\n**Q1: How are the epoch and iteration defined in Lookaround's setting?**\n\nUnder the standard training process, an epoch means the model fully traverses the origin dataset once, typically consisting of $N$ batches. However, in the Lookaround setting, an epoch represents that the model fully traverses the dataset $d$ times, equivalent to $d*N$ batches. This is because each batch of data undergoes $d$ different data augmentations and is trained by the model. Lookaround, $k$ steps around, 1 step average. One iteration means that each branch of Lookaround does an around step. We will modify it in the pseudo-code section of the paper to make it easier to understand.\n\n**Q2: Lookaround versus repeated augmentations.**\n\nThanks for mentioning this wonderful work. While our method does share some similarities with it, there are also significant differences. Firstly, the motivations behind both approaches are distinct. The central idea of the paper is to use multiple data augmentations within a single batch to average gradients, hoping to achieve a more stable gradient representation. In contrast, the goal of Lookaround is to increase the diversity of the model in the loss landscape. It performs gradient descent separately under different data augmentations $k$ times, followed by an average once. From this perspective, \"Augment your batch\" can be seen as a special case of the Lookaround method where $k$ is set to 1. \n\n**Q3: Computational Complexity and Comparison.**\n\nThe Lookaround optimizer can be viewed as maintaining multiple backups of model weights during the optimization process. Each backup is only trained under its corresponding data augmentation strategy (forward and backward propagation) and updates the parameters using weight averaging after multiple iterations.\n\nIn general, if the time for an iteration is defined as $\\Omega$, and the time to perform a weight average is $\\omega$, given that the dataset contains $B$ batches, the standard time complexity of SGDM to complete an epoch is $O(B\\Omega)$. The time complexity of Lookaround is $O(dB\\Omega+\\frac{B\\omega}{d})$. Given that $\\omega$ is much smaller than $\\Omega$, this can be approximated as $O(dB \\Omega)$.\n\nIn our experiments (corresponding to Table 1 to 4 in the paper), to establish fair comparisons in computational among different approaches, we adopt the same augmentations for the competitors. Specifically, within a single epoch, both the proposed Lookaround and the competitors undergo training on an identical d times the data augmentations. With such a setup, we guarantee consistency in the data volume utilized by each method, thereby ensuring fair comparisons in terms of computation. \n\nThe following is an example of the running time of different methods in our experiments. We run each method for 50 epochs and calculate the average execution time of ResNet50 for one epoch on a 3090 graphics card on CIFAR100. The results are shown in Table S1. \nTable S1: Average execution time of each method traversing an epoch on ResNet50 on CIFAR100.\n|     Method     | SGDM | Lookahead | AdamW | SAM  | Lookaround |\n| :------------: | :--: | :-------: | :---: | :--: | :--------: |\n| Execution Time ($s$) | 123  |    123    |  125  | 246  |    123     | \n\n  When considering memory usage, we can divide it into two parts: one part is generated by the model parameters, and the other part is produced by the intermediate results during the model training process, where the latter occupies the main part of the memory. In terms of memory usage for model parameters, the memory occupied by parameters in the Lookaround method is d times that of SGDM. As for the intermediate results produced by the computational graph during model training, since it only updates on the corresponding weight backup and does not increase with the number of data augmentation strategies, its memory usage is the same as conventional methods (such as SGDM).\n\n**Q4: Limited Experiments on Datasets Beyond Image Classification.**\n\nWe really appreciate your suggestions. It's a great idea to apply Lookaround to broader areas including language modeling and segmentation/detection tasks. In this work, we focuses on visual classification. The broader application are left to our future work. Thank you again for your kind suggestions positive comment on our work.\n"}]

Author rebuttal:

Meta-review: This paper proposes a new  Lookaround optimizer for network training. It is inspired by weight averaging (WA) techniques in deep learning, and thus it looks around nearby points by performing multiple gradient computations and then averages them to obtain better generalizing solutions. Experimental results show the effectiveness of Lookaround. All the reviewers unanimously accepted the paper, and so did the final decision. The authors should also revise their paper according to the reviewers' suggestions in the final version.
Decision: Accept (poster)