SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} V _ {L o o k a r o u n d} ^ {*} = \frac {\alpha^ {2} (\mathbf {I} - (\mathbf {I} - \gamma \mathbf {A}) ^ {2 k}) + 2 \alpha (1 - \alpha) (\mathbf {I} - (\mathbf {I} - \gamma \mathbf {A}) ^ {k})}{\alpha^ {2} (d \mathbf {I} - (d - 1) (\mathbf {I} - \gamma \mathbf {A}) ^ {2 k})} V _ {L o o k a h e a d} ^ {*} \\ = \frac {\mathbf {I} - B ^ {2} + 2 \frac {1 - \alpha}{\alpha} (\mathbf {I} - B)}{d \mathbf {I} - (d - 1) B ^ {2}} V _ {L o o k a h e a d} ^ {*} \\ = \frac {\frac {2 - \alpha}{\alpha} \mathbf {I} - B ^ {2} - \frac {2 - 2 \alpha}{\alpha} B}{d \mathbf {I} - (d - 1) B ^ {2}} V _ {L o o k a h e a d} ^ {*}. \\ \end{array}
$$
Let the coefficient matrix be denoted as  $\mathbf{C}$ , when  $d \geqslant 3$ , for each diagonal element  $C_{ii}$  of  $\mathbf{C}$ , we can scale the denominator of this expression as follows:
$$
C _ {i i} \leqslant \frac {\frac {2 - \alpha}{\alpha} - B _ {i i} ^ {2} - \frac {2 - 2 \alpha}{\alpha} B _ {i i}}{3 - 2 B _ {i i} ^ {2}},
$$
Then, we can derive the range of  $\alpha$  by restricting the right-hand side expression as follows:
$$
\frac {\frac {2 - \alpha}{\alpha} - B _ {i i} ^ {2} - \frac {2 - 2 \alpha}{\alpha} B _ {i i}}{3 - 2 B _ {i i} ^ {2}} \leqslant 1,
$$
As  $0 \leqslant B_{ii} \leqslant 1$ , we can multiply both sides of the inequality by the denominator, then rearrange and combine like terms to obtain the following form:
$$
B _ {i i} ^ {2} - \frac {2 - 2 \alpha}{\alpha} B _ {i i} + \frac {2 - 4 \alpha}{\alpha} \leqslant 0.
$$
Skipping the detailed steps, we can obtain  $\alpha \geqslant 0.5$  by solving the quadratic equation. Therefore, in the case where  $\alpha \in [0.5,1)$  and  $d\geqslant 3$  ( $\alpha < 1$  is subject to Lookahead's settings), the coefficient matrix  $C\leqslant \mathbf{I}$ , so the convergence speed of Lookaround is slower than Lookahead.
B.2 Deterministic quadratic convergence
Our method samples data from multiple data augmentation strategies, which can be analogously viewed as averaging the sampling of historical trajectories during the convergence analysis of quadratic functions. Thus, for weight averaging, we select model points that align with each point in the  $k$ -step trajectory. From this perspective, we examine and compare the convergence rates of Lookaround and Lookahead.
We first show the state transition equation for the classical momentum method in quadratic functions:
$$
\mathbf {v} _ {t + 1} = \beta \mathbf {v} _ {t} - \nabla_ {\theta} f (\theta_ {t}) = \beta \mathbf {v} _ {t} - \mathbf {A} \theta_ {t}, \tag {13}
$$
$$
\theta_ {t + 1} = \theta_ {t} + \gamma \mathbf {v} _ {t + 1} = \gamma \beta \mathbf {v} _ {t} + (\mathbf {I} - \gamma \mathbf {A}) \theta_ {t}. \tag {14}
$$
Here,  $\mathbf{v}$  stands for the momentum term. We can generalize this to matrix form:
$$
\left[ \begin{array}{l} \mathbf {v} _ {t + 1} \\ \theta_ {t + 1} \end{array} \right] = \left[ \begin{array}{c c} \beta & - \mathbf {A} \\ \gamma \beta & \mathbf {I} - \gamma \mathbf {A} \end{array} \right] \left[ \begin{array}{l} \mathbf {v} _ {t} \\ \theta_ {t} \end{array} \right].
$$
Thus, given the initial  $\theta_0$ , we can obtain the convergence rate with respect to  $\theta$  by the maximum eigenvalue of the matrix. Referring to Zhang et al. [49] and Lucas et al. [30], we obtain the state transition matrix regarding our algorithm as follows:
$$
\left[ \begin{array}{c} \boldsymbol {\theta} _ {t, 0} \\ \boldsymbol {\theta} _ {t - 1, k} \\ \vdots \\ \boldsymbol {\theta} _ {t - 1, 1} \end{array} \right] = L B ^ {(k - 1)} T \left[ \begin{array}{c} \boldsymbol {\theta} _ {t - 1, 0} \\ \boldsymbol {\theta} _ {t - 2, k} \\ \vdots \\ \boldsymbol {\theta} _ {t - 2, 1} \end{array} \right],
$$
where L, B and T denote the average weight matrix, the single-step transfer matrix and the position transformation matrix respectively:
$$
L = \left[ \begin{array}{c c c c c} \frac {1}{k + 1} I & \frac {1}{k + 1} I & \dots & \frac {1}{k + 1} I & \frac {1}{k + 1} I \\ I & 0 & \dots & \dots & 0 \\ 0 & I & \ddots & \ddots & \vdots \\ \vdots & \ddots & \ddots & 0 & \vdots \\ 0 & \dots & 0 & I & 0 \end{array} \right],
$$
$$
B = \left[ \begin{array}{c c c c c} (1 + \beta) I - \eta A & - \beta I & 0 & \dots & 0 \\ I & 0 & \dots & \dots & 0 \\ 0 & I & \ddots & \ddots & \vdots \\ \vdots & \ddots & \ddots & 0 & \vdots \\ 0 & \dots & 0 & I & 0 \end{array} \right],
$$
$$
T = \left[ \begin{array}{c c c c c c} I - \eta A & \beta I & - \beta I & 0 & \dots & 0 \\ I & 0 & \dots & \dots & 0 & \vdots \\ 0 & I & \ddots & \dots & \vdots & \vdots \\ \vdots & \ddots & \ddots & 0 & \vdots & \vdots \\ \vdots & \dots & 0 & I & 0 & 0 \\ 0 & \dots & 0 & 0 & I & 0 \end{array} \right].
$$
After specifying the appropriate parameters and performing matrix multiplication to obtain the state transition matrix, the convergence rate  $\rho$  can be obtained by calculating the largest eigenvalue of the matrix. Note that since this linear dynamical system corresponds to  $k$  updates, we finally have to compute the  $k^{th}$  root of the eigenvalues to recover the correct convergence bounds.
C.1 Random Initialization
C.1.1 CIFAR10 and CIFAR100
Data augmentation details. For the CIFAR10 dataset, we use [RandomCrop + *] for data augmentation, and for the CIFAR100 dataset, we use [RandomCrop + * + RandomRotation] for data augmentation. * can be replaced by three different data augmentation methods of random horizontal flip, random vertical flip, or RandAugment [4].
Training details. For the CIFAR10 and CIFAR100 datasets, we have applied some common settings. The initial learning rate is set to 0.1, and the batch size is set to 128. Additionally, a warm-up phase of 1 epoch is implemented. Subsequently, different learning rate schedulers are used based on the specific dataset. For the CIFAR100 dataset, we utilize the MultiStepLR scheduler. The learning rate is decayed at the 60, 120, and 160 epochs, with a decay factor of 0.2. For the CIFAR10 dataset, we employ the CosineAnnealingLR learning rate scheduler. In comparison with other optimizers or optimization methods, we use the default settings of the other methods in the corresponding papers.
C.1.2 ImageNet.
For the ImageNet dataset, our data augmentation strategy involves [RandomResizedCrop + * + RandomRotation]. Here, * can be substituted by one of the three data augmentation methods: random horizontal flip, random vertical flip, or RandAugment [4]. We adopt the following training settings: an initial learning rate of 0.1 and a batch size of 256. The model undergoes training for a total of 100 epochs. Additionally, we make use of the MultiStepLR scheduler, which decays the learning rate by a factor of 0.1 at the 30th, 60th, and 90th epochs.
C.2 Finetuning
In this stage, all images are resized to  $224 \times 224$  pixels to match the input size of the pretrained model. All models use the ImageNet-1k pretrained weights sourced from the PyTorch library. Throughout the finetuning process, we employed three data augmentation methods: random horizontal flip, RandAugment, and AutoAugment.
For the training of ResNet50, we performed a grid search as shown in Table 8 to determine the optimal learning rate and weight decay. Every training process lasted for 50 epochs with a batch size
Table 8: Hyperparameter search for finetuning on ResNet50.
Dataset	Epochs	Base LR	Weight Decay
CIFAR10	50	{0.001, 0.003, 0.01, 0.03}	{0.0001, 0.00005, 0.00001}
CIFAR100	50	{0.001, 0.003, 0.01, 0.03}	{0.0001, 0.00005, 0.00001}
Table 9: Comparison between different training strategies with ResNet18 on Stanford Cars dataset. Here "H" denotes random horizontal flip data augmentation. "R" denotes RandAugment data augmentation. "A" denotes AutoAugment data augmentation. "Logit Ensemble + SWA" denotes a logit ensemble approach that employs SWA models, which have been acquired through various data augmentation techniques. "T-Budget" and "I-Budget" respectively represent Training budget and Inference budget.
Method	Data Augmentation Strategy	Top-1 Acc	Top-5 Acc	T-Budget	I-Budget
SGDM	"H"	86.57	97.52	1x	1x
SWA	"H"	87.60	98.21	1x	1x
SGDM	"R"	87.30	98.01	1x	1x
SWA	"R"	88.00	98.21	1x	1x
SGDM	"A"	87.63	98.05	1x	1x
SWA	"A"	88.11	98.15	1x	1x
Logit Ensemble	"H" + "R"	89.24	98.44	2x	2x
Lookaround	"H" + "R"	89.35	98.47	2x	1x
Logit Ensemble	"H" + "R" + "A"	90.19	98.67	3x	3x
Lookaround	"H" + "R" + "A"	90.76	98.78	3x	1x
Logit Ensemble+SWA	"H" + "R" + "A"	90.39	98.80	3x	3x
Lookaround+SWA	"H" + "R" + "A"	91.04	98.86	3x	1x
set to 128. For the training of ViT-B/16, we used the Adam optimizer, starting with a learning rate of 0.00001. We set  $\beta_{1}$  to 0.9,  $\beta_{2}$  to 0.999, and applied a weight decay of 0.01. To optimize memory usage, the batch size was set to 64.
C.3 Compared with Sharpness-Aware Minimization
We adopt the default hyperparameters from the origin paper of SAM [10] and combine them with the optimal hyperparameters found during random initialization or finetuning to conduct comparative experiments between SAM and Lookaround.
C.4 Compared with Ensemble Method
We compare Lookaround with Logit Ensemble and Snapshot Ensemble [18]. In the setting of Logit Ensemble, we train multiple models separately using different data augmentation methods, and then average the outputs of these models for prediction. However, this approach requires more inference time. In the setting of Snapshot Ensemble, we use the CosineAnnealingWarmRestarts learning rate scheduler to collect four snapshots during the training process. Then, we average the outputs of these different snapshots for prediction. This approach also requires more inference time.
Besides, we conducted a full experimental validation of Logit Ensemble and SWA under a new dataset Stanford Cars, as shown in Table 9. Under the Stanford Cars dataset, Lookaround still performs better than Logit Ensemble. Additionally, adding SWA into the training process of Lookaround can further improve performance. Under the limit of single-model inference, Lookaround+SWA achieves a Top-1 accuracy of  $91.04\%$ , exceeding the optimal  $88.11\%$  (SWA with AutoAugment).
C.5 Ablation Study
In our ablation experiments, we evaluate the individual contributions of two components: Data Augmentation (DA) and Weight Averaging (WA) to the effectiveness of the Lookaround. Below are the specific settings for different ablation experiments.
For the experiments without both DA and WA, three models were trained independently, each utilizing a distinct data augmentation strategy, for a duration of 200 epochs. After training, we identified
and reported the highest accuracy achieved by these models on the test set. In the experiments that employed only DA, we trained a single model using all data, combining the three data augmentation strategies, without the use of weight averaging. For the experiments with only WA, three models were trained separately for 200 epochs each, employing different data augmentation strategies. Following this, we selected the model that showcased the best performance on the test set.
C.6 Additional Analysis
In the robustness experiments with the number of Data Augmentation methods, the six data augmentation methods are given as: RandomVerticalFlip, RandomHorizontalFlip, RandAugment, AutoAugment, RandomPerspective, RandomEqualize. All the Augmentation methods are from the PyTorch library. When more data augmentation methods are used, the training time will be correspondingly increased in our method. Therefore, in this paper's main experiments, we only select three data augmentation methods for comparison to reduce the time consumption.
D Contemporaneous Work
After completing our work, we discovered similar work at CVPR 2023 within the same year. Inspired by similar motivations, the DART [20] method proposed by Samyak Jain et al. utilizes a similar training structure that includes different data augmentation training of multiple sub-models, with weight averaging occurring at intervals.
Methodology. DART starts weight averaging until the second half training process and adopts a sparser weight averaging strategy. Meanwhile, the DART scheme uses the ERM [15] strategy to initialize the model. In contrast, Lookaround maintains a consistent and dense weight averaging strategy throughout its training, making it more user-friendly in real-world applications.
Theory. DART proves that the average while training is more robust from the perspective of noise feature. On the other hand, Lookaround proves that it can get lower expected loss under the setting of quadratic noise function.
Experiments. DART validates its effectiveness in domain generalization. Lookaround is tested in the scenarios of finetuning and training from scratch.
In conclusion, although sharing a similar idea, two methods differ substantially in theoretical foundations and experimental approaches. Both methodologies are complementary and furnish significant insights for the broader research community.
And we compared Lookaround with DART with the same data augmentation, the results are shown in Table 10. Our results were better than DART, the reason behind this may come from the fact that we used more frequent "average steps" during training and started "average steps" earlier in the cycle. More frequent "average steps"
Table 10: Comparison with the DART method with ResNet18 on CIFAR10 and CIFAR100. The results of  $\mathrm{DART}^{\dagger}$  are taken from reference [20].
Method	CIFAR10	CIFAR100
Dart†	97.14	82.89
Lookaround	97.22	83.16
are more conducive to the training of Lookaround, as referenced in the main body of the paper.
E Relationship with Model Soups
Model Soups [46] is a framework for finetuning a common pretrained model using different hyperparameters and then averaging the weights of different finetuned models to improve model performance and generalization. In the finetuning settings, we conducted an in-depth comparison between Lookaround and Model Soups, with the related results presented in Table 11. The "Update BN" suggestion originated from the reviewer's feedback. Given that we employed a CNN architecture, the averaging of means and variances in the batch normalization (BN) layers of different models within Model Soups might lead to distortions, failing to accurately represent the statistical characteristics of the true hidden layer inputs. To address this, we passed the averaged model through
Table 11: Comparisons between Lookaround, Model Soups on ResNet50. "M=18" represents 18 different hyperparameters. "M=3" represents that Lookaround is trained using 3 different data augmentation techniques. 18 different configurations come from the combinations of three data augmentations (RandomHorizontalFlip, RandAugment, AutoAugment), three initial learning rates:  $\{0.01, 0.003, 0.001\}$  and label smooth or not. Uniform  $Sous_{s}^{+}$ represents that we use the top-10 accuracy models for uniform soups (the accuracy of these models ranges from  $83.91\%$  to  $84.64\%$ ).
Method	Update BN	Top-1 Acc	Top-5 Acc
Uniform Soups	-	82.93	97.44
Uniform Soups	✓	84.53	98.10
Uniform \(Soups^+\)	-	84.60	97.81
Uniform \(Soups^+\)	✓	85.57	98.04
Greedy Soups	-	85.52	98.02
Greedy Soups	✓	86.09	98.11
Lookaround (M=3)	-	85.20	97.15
the entire training set using forward propagation to individually update the BN layer's running mean and running var.
The experimental results indicate that the Lookaround method, employing only three types of data augmentation, still remains comparable in performance to the Model Soups method, which utilizes 18 types of data augmentation. This further attests to Lookaround's generalization capability even under limited data augmentation conditions. Additionally, we observed that the Lookaround method is applicable to tasks with random initialization, while the Model Soups, due to its prerequisite of pretrained weights, cannot be employed in such scenarios.
F Computational Complexity and Comparison
The Lookaround optimizer can be viewed as maintaining multiple backups of model weights during the optimization process. Each backup is only trained under its corresponding data augmentation strategy (forward and backward propagation) and updates the parameters using weight averaging after multiple iterations.
In general, if the time for an iteration is defined as  $\Omega$ , and the time to perform a weight average is  $\omega$ , given that the dataset contains  $B$  batches, the standard time complexity of SGDM to complete an epoch is  $O(B\Omega)$ . The time complexity of Lookaround is  $O(dB\Omega + \frac{B\omega}{d})$ . Given that  $\omega$  is much smaller than  $\Omega$ , this can be approximated as  $O(d\Omega B)$ .
In our experiments (corresponding to Table 1 to 5 in the paper), to establish fair comparisons in computational among different approaches, we adopt the same augmentations for the competitors. Specifically, within a single epoch, both the proposed Lookaround and the competitors undergo training on an identical  $d$  times the data augmentations. With such a setup, we guarantee consistency in the data volume utilized by each method, thereby ensuring fair comparisons in terms of computation.