SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Fine-tuning hyperparameters grid search for ResNet and Swin-T models. We divide the training data into training and validation subsets in 1 to 10 ratio for CIFAR10, CIFAR100 and in 1 to 5 ratio for SUN397 as in the work of Kornblith et al. (2019) and 1 to 5 ration for Clipart and Chest-X. We use grid search for the values of training epochs, initial learning rate and weight decay giving the best quality of individual fine-tuned networks on validation subsets. For ResNet-50 (both types of pre-training), we consider weight decays from  $[2 \cdot 10^{-5}, 1 \cdot 10^{-4}, 5 \cdot 10^{-4}]$  and initial learning rates from  $[5 \cdot 10^{-4}, 10^{-3}, 2.5 \cdot 10^{-3}, 5 \cdot 10^{-3}, 10^{-2}, 2.5 \cdot 10^{-2}, 5 \cdot 10^{-2}, 10^{-1}, 2 \cdot 10^{-1}]$ . For Swin-T, the ranges are  $[10^{-2}, 5 \cdot 10^{-2}, 2.5 \cdot 10^{-1}]$  and  $[2.5 \cdot 10^{-5}, 5 \cdot 10^{-5}, 10^{-4}, 2.5 \cdot 10^{-4}, 5 \cdot 10^{-4}, 10^{-3}]$ , respectively. As for the number of training epochs, we determine the value corresponding to 20000 SGD steps (this procedure is taken from Kornblith et al. (2019); the value differs across datasets, making up 102 for CIFAR-10, CIFAR-100, 256 for SUN-397, 8 for Chest-X and 19 for Clipart) and consider its multipliers from the list  $[×0.03125, ×0.0625, ×0.125, ×0.25, ×0.5, ×1]$  rounded down. The resulting optimal sets of hyperparameters are shown in Table 2.
CLIP. We use a CLIP ViT-B/32 model pre-trained by the authors on a large dataset of image-text pairs. For fine-tuning of individual models for Local DEs, we follow Wortsman et al. (2022) using AdamW (Loshchilov and Hutter, 2019), linear learning rate warm-up for 500 SGD steps and the best combination of the training hyperparameters from their random search (number of epochs 11, batch size 512, learning rate  $1.23 \cdot 10^{-5}$ , label smoothing 0.134, and augmentation Rand-Augment with  $M = 19$ ,  $N = 0$ ). For SSE and StarSSE, we also use AdamW, learning rate warm-up, and the same batch size, label smoothing, and augmentations in each cycle.
Hyperparameters of more local/optimal/more semi-local experiments. The hyperparameter multipliers for more local, SSE optimal, StarSSE optimal and more semi local experiments are presented in Table 3.
Amount of computations. We use NVIDIA TESLA V100 and A100 GPUs for computations in our experiments. Total expenses on pre-training constitute 8K GPU hours for self-supervised ResNet-50 (BYOL) and supervised Swin-T. Fine-tuning models for Local and Global DEs and cyclical learning
$^{2}$ https://github.com/SamsungLabs/pytorch-ensembles
Table 2: Optimal values of training epochs, initial learning rate and weight decay for fine-tuning an individual model.
Pre-train.	Target dataset	epochs	lr	wd
SV ResNet-50	CIFAR-10	51	2.5 · 10-2	2 · 10-5
CIFAR-100	25	5 · 10-3	5 · 10-4
SUN-397	256	5 · 10-4	5 · 10-4
ChestX	8	2.5 · 10-2	5 · 10-4
Clipart	19	2.5 · 10-2	5 · 10-4
SSL ResNet-50	CIFAR-10	25	5 · 10-2	10-4
CIFAR-100	25	5 · 10-2	10-4
SUN-397	16	10-1	10-4
ChestX	8	2 · 10-1	5 · 10-4
Clipart	19	10-1	2 · 10-5
SV Swin-T	CIFAR-100	102	10-4	2.5 · 10-1
SUN-397	256	2.5 · 10-5	5 · 10-2
Table 3: Considered multipliers of the learning rate and the number of epochs for more local, SSE optimal, StarSSE optimal, and more semi-local experiments for different datasets and types of pretraining.
Pre-train.	Target dataset	more local	SSE optimal	StarSSE optimal	more semi-local
lr	ep.	lr	ep.	lr	ep.	lr	ep.
SV ResNet-50	CIFAR-10	×0.25	×1	×2	×0.25	×2	×0.5	×2	×2
CIFAR-100	×1	×1	×2	×1	×4	×1	×8	×4
SUN-397	×0.5	×1	×32	×0.125	×32	×0.125	×32	×1
ChestX	×0.25	×1	×4	×2	×2	×2	×8	×4
Clipart	×0.25	×0.5	×1	×0.5	×2	×0.5	×4	×2
SSL ResNet-50	CIFAR-10	×0.5	×1	×1	×1	×2	×1	×4	×4
CIFAR-100	×0.25	×1	×2	×0.5	×4	×0.5	×4	×4
SUN-397	×0.5	×1	×1	×1	×2	×1	×4	×8
ChestX	×0.5	×1	×1	×1	×1	×2	×2	×4
Clipart	×0.5	×0.5	×4	×0.5	×4	×0.5	×8	×2
SV Swin-T	CIFAR-100	×0.5	×0.25	×2	×0.25	×4	×0.5	×4	×1
SUN-397	×1	×1	×16	×0.125	×16	×0.25	×16	×1
CLIP ViT-B/32	ImageNet	×1	×0.25	×2	×1	×2	×1	×6	×1
rate ensembles took another 17K GPU hours, resulting in a total amount of approximately 25K GPU hours.
C Non-cyclical local ensemble methods
In this section, we experiment with non-cyclical local ensemble methods in the transfer learning setup and show that they are generally less effective than the cyclical ones. We choose three non-cyclical methods: KFAC-Laplace (Ritter et al., 2018), SWAG (Maddox et al., 2019) and SPRO (Benton et al., 2021). KFAC-Laplace is the most simple one and does not require any additional training. Given one usually trained network, it computes the Laplace approximation with a block Kronecker
Table 4: Accuracy and diversity of ensembles of size 5 for non-cyclical local methods, Local DE, and optimal SSE and StarSSE. Self-supervised pre-training.
Dataset	Local DE	SSE (optim.)	StarSSE (optim.)	KFAC Laplace	SPRO	SWAG
CIFAR-acc.100	acc. div.	87.33±0.2166.25±3.56	87.11±0.0671.57±5.94	87.63±0.1271.51±2.08	86.01±0.0235.38±2.07	86.46±0.0833.94±16.96	86.99±0.0836.74±5.86
SUN-397	acc. div.	65.77±0.1947.41±2.48	65.79±0.1057.69±6.51	66.37±0.2560.74±1.94	64.39±0.0717.87±0.94	64.71±0.0924.37±9.87	65.83±0.1149.45±5.70
factored (KFAC) estimate of the Hessian matrix and then samples network weights from the Gaussian distribution with the trained network as a mean and the obtained KFAC estimate as a covariance matrix. SWAG also constructs the Gaussian distribution but uses the SWA solution as a mean and approximates the covariance matrix based on the SWA checkpoints. SWAG is shown to be the most effective alternative to cyclical methods in the regular training setup (Ashukha et al., 2020). SPRO shows competitive results to SWAG by constructing a simplex in the weight space with the given trained network as one of the vertexes and then sampling ensemble members from it.
In the experiments, we consider ResNet-50 with self-supervised pre-training on CIFAR-100 and SUN-397 datasets. We use the official SWAG repository<sup>3</sup> for the implementations of KFAC-Laplace and SWAG, and our own implementation of SPRO. We use grid search to determine the optimal hyperparameters for the non-cyclical local methods:
- KFAC-Laplace. We search for the optimal sampling scale from the list [0.2, 0.4, 0.6, 0.8, 1.0, 1.2]. The resulting values are 1.0 for CIFAR-100 and 0.6 for SUN-397.
- SWAG. We continue training the fine-tuned checkpoints with a constant learning rate (0.08 for CIFAR-100 and 0.16 for SUN-397) for 19 more epochs, gathering new models at the end of each epoch (overall, there are 20 models counting the fine-tuned one). Then, we search for the value of sampling scale from the list [0.1, 0.2, 0.4, 0.6, 0.8, 1.0, 1.2] similar to KFAC-Laplace. The optimal values are 0.1 for CIFAR-100 and 0.2 for SUN-397.
- SPRO. We start with a fine-tuned checkpoint and train 2 more points to form 2-simplexes. We vary the learning rate ([0.0125, 0.025, 0.05, 0.1]) and the number of epochs (CIFAR-100: [5, 10, 20], SUN-397: [4, 8, 16]), as well as the regularization coefficient  $([10^{-6}, 10^{-5}])$  to train these points. The optimal tuples are  $(0.05, 10, 10^{-5})$  for CIFAR-100 and  $(0.05, 16, 10^{-6})$  for SUN-397.
In Table 4, we compare accuracy and diversity of 5-network ensembles obtained with the considered non-cyclical local methods, Local DE baseline, and optimal SSE and StarSSE (see Appendix J for the definition of the diversity metric). KFAC-Laplace and SPRO show inferior accuracy and diversity compared to both Local DE and SSE; hence, these methods' behavior is extremely local. SWAG demonstrates stronger results, even comparable to Local DE and SSE on SUN-397. However, StarSSE is more effective than SWAG on both datasets. In conclusion, even though cyclical methods can not benefit from the exploration of several low-loss basins in transfer learning setup, they are still preferable to other local methods, similarly to the regular training setup. Based on this analysis, we decided to focus on cyclical methods in the paper.
D Variants of cyclical ensemble methods
In this section, we discuss other variants of cyclical learning rate methods and explain our choice of SSE with a fixed first model fine-tuning schedule for the main study. In all the experiments, we consider ResNet-50 on CIFAR-100 with self-supervised pre-training.
Figure 6: Results of ensembles (left plots) and individual models (right plots) for our variant of SSE with a fixed schedule for the first network fine-tuning and Standard SSE, which uses the same hyperparameters in each cycle. CIFAR-100, self-supervised pre-training. Hyperparameters for all three differently behaving experiments are the same for both methods.
D.1 SSE vs Standard SSE: why we fine-tune the first model with a fixed schedule
Standard SSE (Huang et al., 2017) trains all the models using the same learning rate schedule in each cycle. However, we choose to use a separate fixed schedule for the first model fine-tuning. This separation allows us to experiment with different hyperparameters for loss landscape exploration without harming the first fine-tuning cycle, which is responsible for moving from the pre-trained checkpoint to the high-quality area in the weight space for the target task.
In the right pair of plots in Figure 6, we compare the behavior of individual model quality from our variant of SSE and the standard SSE with the same hyperparameter values (we use the values for three differently behaving SSEs specified in Appendix B). The first model from the optimal standard SSE experiment has adequate quality, which can also be improved a bit by choosing hyperparameter values optimal for Local DE instead of the ones optimal for our SSE. Since hyperparameter values of the more local and the more semi-local experiments are very different from the hyperparameters usually used for individual model fine-tuning in Local DEs, the quality of the first model from standard SSE is very low in these cases. Moreover, going through more cycles does not help due to slow quality improvement for low hyperparameter values and progressive quality degradation for high hyperparameter values. As a result, the ensemble quality of standard SSE with different hyperparameter values varies a lot, not only due to changes in exploration but also due to different fine-tuning quality in the first place (see the left pair of plots in Figure 6).
To solve the described problem, the first low-quality models are usually dropped from the standard SSE. However, this can not help in the case of high hyperparameter values, which is important to us in this paper. Moreover, for different hyperparameter values, we would need to drop a different number of first models, which would make the comparison of different SSEs and DEs much less clear.
D.2 cSGLD
cSGLD (Zhang et al., 2020) technically is very similar to SSE and uses the same cyclical cosine learning rate schedule. However, instead of taking only one final model from each cycle, cSGLD samples several models on the last epochs using Langevin dynamics. In our experiments, we compare SSE and cSGLD, which use separate hyperparameters for the first cycle due to the conclusion from the previous section. All other aspects of the cSGLD procedure are the same as in the original paper, which includes the number of sampled models in each cycle equal to 3.
In the right pair of plots in Figure 7, we show that models from the same cycle of cSGLD are located in the same basins. Hence, exploration of the weight space mostly depends on obtaining the models from different cycles, which can indeed end up in different basins, similarly to the SSE ones (see the left pair of plots in Figure 7). Based on these observations, we compare SSE and cSGLD in
<sup>3</sup>https://github.com/wjmaddox/swa_gaussian/tree/master
Different cSGLD cycles
The same cSGLD cycle
Figure 7: Linear connectivity analysis of cSGLD on CIFAR-100 with self-supervised pre-training. We show train and test accuracy along line segments between two random networks in Local DEs, between the last network of the first cycle (#3) and the last network of the last cycle (#15) of cSGLD (left plots), and between the first (#1) and the last network (#3) from the first cycle of cSGLD (right plots).
Individual models
Figure 8: Results of ensembles (left plots) and individual models (right plots) for SSE and cSGLD. CIFAR-100, self-supervised pre-training. Hyperparameters for all three differently behaving experiments are the same for both methods. Every cycle of cSGLD adds 3 new models, so cSGLD ensembles have 3 times more models than SSE, resulting in a slightly better quality.
Figure 9: Results of SSEs (left plots) and FGEs (right plots) of different sizes on CIFAR-100 with self-supervised pre-training for varying values of cycle hyperparameters (maximum learning rate and a number of epochs). Local and Global DEs are shown for comparison with gray dotted lines.
Interpolations of model weights
Individual models
Figure 10: Linear connectivity analysis (left plots) and individual model quality (right plots) of three differently behaving FGEs on CIFAR-100 with self-supervised pre-training. On the left plots, we show train and test accuracy along line segments between two random networks in Local DEs and between the first and the last (5-th) networks in FGEs.
Figure 8, fixing the number of cycles instead of the number of models in an ensemble. We conclude that cSGLD behaves very similarly to SSE and choose SSE for the main study because it gives almost the same quality using fewer networks.
D.3 FGE
FGE (Garipov et al., 2018) uses a triangular schedule instead of the cosine one for each cycle and trains the first model with a separate schedule in the same manner as we do in our version of SSE. In our experiments, we train the first model of FGE with the same hyperparameters as the first model of SSE.
In Figure 9, we show the results of FGE with different cycle hyperparameters and compare them to the SSE ones. FGE experiments show similar types of behaviors, as we discussed for SSE in the main text. However, due to the triangular learning rate schedule, FGE behaves more locally than SSE with the same hyperparameter values. We also analyze the loss along line segments between the models from FGE and their individual quality in Figure 10. The results show that FGE can work in local and semi-local regimes, and it achieves the optimal ensemble quality in the local one.
E Ensembling results of SSE and StarSSE
In this section, we provide additional results of SSE and StarSSE on CIFAR-10 and SUN-397 datasets. Figure 11 is complementary to Figure 1 from the main text and compares SSE and StarSSE with different hyperparameter values to Local and Global DEs. The results for self-supervised pre-training are very similar between different datasets. With supervised pre-training, the difference between SSE/StarSSE and Local DE varies for different datasets, probably due to higher variance in optimal hyperparameters. For SUN-397, SSE and StarSSE show very strong results close to the Global DE, while for CIFAR-10, their results are inferior to the Local DE.
F Analysis of SSE and StarSSE behavior
In this section, we provide additional plots on the analysis of SSE and StarSSE for the case of supervised pre-training. Figure 12 is analogous to Figure 2 from the main text and demonstrates the behavior of train and test accuracy along line segments between the models from SSE and StarSSE. Figure 13 is analogous to Figure 3 from the main text and demonstrates the quality of individual models from SSE and StarSSE. The general observations are the same as in Sections 5.3 and 5.4. Ultimately, SSE and StarSSE may work in either local or semi-local regimes, but they achieve optimal results in the local one.
Figure 11: Results of SSEs and StarSSEs of different sizes on CIFAR-10 and SUN-397 for different types of pre-training and varying values of cycle hyperparameters (maximum learning rate and a number of epochs). Local and Global DEs are shown for comparison with gray dotted lines.
Figure 12: Linear connectivity analysis of Local DEs, SSEs (left plots) and StarSSEs (right plots) on CIFAR-100 with supervised pre-training. We show train and test accuracy along line segments between two random networks in Local DEs, between the first and the last (5-th) network in three differently behaving SSEs, and between the first and any other consequent network in three differently behaving StarSSEs. Hyperparameters for more local and more semi-local experiments are the same for SSE and StarSSE, while hyperparameters for the optimal experiments may differ.
Figure 13: Train and test accuracy of individual models from three differently behaving SSEs (left plots) and StarSSEs (right plots) on CIFAR-100 with supervised pre-training (with a single fine-tuned model for comparison). Hyperparameters for more local and more semi-local experiments are the same for SSE and StarSSE, while hyperparameters for the optimal experiments may differ.
Figure 14: 2D visualization of test and train accuracy around SSE and StarSSE models on the CIFAR-100 dataset with self-supervised pre-training. The markers with black outline correspond to the points generating the projection plane. The axes scale matches the actual distances between the points of the plane in the weight space. The stars without black outline in the SSE plots show the projection of SSE models from previous cycles (note that these are only the projections: the models themselves lay outside of the projection plane and their actual accuracy is different to the one plotted).
Figure 15: Results of ensembles (left plots) and model soups (right plots) of different sizes on CIFAR-10 and SUN-397 for SSE and StarSSE with supervised and self-supervised pre-training. Hyperparameters for more local and more semi-local experiments are the same for SSE and StarSSE on the same dataset, while hyperparameters for the optimal experiments may differ.
G Accuracy behavior on 2D planes in the weight space
In this section, we plot 2-dimensional visualizations of high-accuracy regions around SSE and StarSSE models similar to Garipov et al. (2018). To construct a 2D visualization, one needs to select three points in the weight space and then plot the metric values over the plane generated by these points. Overall, for SSE, we consider four points: the first SSE model (i.e., a model from a Local DE) and three networks from three differently behaving SSE experiments; each network is taken from the last (5-th) SSE cycle. Importantly, the three SSE experiments are all launched starting from the same first model, which is one of the four points considered. We then take all combinations of size three out of these four points and plot separate 2D plots for them. We repeat the same procedure for StarSSE models.
Each combination of three points forms a triangle, which is projected into the image plane. The first model is projected into the point  $(0,0)$ . We make the  $x$ -axis parallel to the side connecting the first and the second models. The  $y$ -axis is parallel to the altitude of the triangle passing through the third model. The side and the altitude are projected, preserving their original length, so the whole plane is projected without distortion. Finally, we consider a coordinate grid over the projected plane,
update the batch normalization statistics at the grid nodes (we use  $10\%$  of the training set for faster processing), and evaluate test and train accuracy at these points.
The resulting 2D visualization plots are shown in Figure 14. In terms of train accuracy, the first and the last local and optimal SSE models are located in the same basin, while the last semi-local SSE model "goes round the corner" — it is almost linearly connected with the last optimal model but we see more significant barriers while connecting it to the first and the last local SSE models. The same observations hold for StarSSE. In terms of test accuracy, the first and the last local and optimal SSE models show similar results, however, the optimal SSE experiment goes much further from the first model and ends up "on the opposite wall" of the high-accuracy region, while the last local SSE model stays "on the same wall" as the first model. This observation holds for StarSSE too. It also matches our model soups experiments, where optimal SSE/StarSSE usually demonstrate substantially stronger results than the local ones.
Semi-local SSE/StartSSE end up very close to the border of the high-accuracy region, both in terms of test and train metrics, in the considered 2-dimensional weight subspaces. When we look at the segment connecting the first and the last semi-local models, we see an abrupt degradation of quality for the models laying beyond the semi-local end. The first and the last local and optimal models are much more stable in this sense. This effect makes us suspect that leaving the pre-train basin makes semi-local SSE/StarSSE converge to sharper minima. Moreover, the track of semi-local SSE experiment shows that the 2-nd SSE cycle moves further from the first model than all subsequent cycles (at least when projected into the image plane). This indicates that even a single cycle with high hyperparameters forces the model to leave the pre-train basin and shows why semi-local StarSSE is not efficient (despite behaving more locally than the semi-local SSE).
H Results of soups of SSE and StarSSE models
In this section, we provide additional results of SSE and StarSSE soups on different datasets and with different types of pre-training. Figure 16 is analogous to Figure 4 from the main text, and its first row demonstrates the results on the CIFAR-100 dataset with supervised pre-training. Additionally, Figure 15 shows results for CIFAR-10 and SUN-397 for both supervised and self-supervised pretraining. Similarly to Appendix E, the results for self-supervised pre-training are very similar for all datasets, while for supervised pre-training they vary more. Also, SSE and StarSSE experiments on CIFAR-10 with supervised pre-training become unstable for high hyperparameter values, therefore, the more semi-local experiments here behave similarly to the optimal ones.
I OOD
In this section, we show the robustness analysis of SSE and StarSSE with supervised pre-training complementary to Section 7. Figure 16 is analogous to Figure 4 from the main text and compares results on in-distribution CIFAR-100 and out-of-distribution CIFAR-100C data. Results for the supervised case are similar to the self-supervised one: SSE and StarSSE are less stable to the data corruption than Local and Global DEs, however, soups of StarSSE models still show strong results compared to Local DE.
J Prediction diversity
In this section, we quantitatively compare the prediction diversity of the networks in SSE, StarSSE, and DE baselines to confirm that the SSE and StarSSE improvement over Local DE is indeed the consequence of higher network diversity. To evaluate the diversity, we use an average normalized prediction difference between model pairs on test data, following Fort et al. (2019) (the higher, the more diverse):
$$
d i v e r s i t y = 1 0 0 \cdot \mathbb {E} _ {m _ {1} \neq m _ {2}} \frac {\mathbb {E} _ {i m a g e s} [ p r e d _ {1} \neq p r e d _ {2} ]}{\max (e r r _ {1} , e r r _ {2})},
$$
where  $m_{i}$  stands for a model from an ensemble with predictions  $pred_{i}$  and test error level  $err_{i}$ . We normalize the prediction difference by the maximum of two errors to pay less attention to the diversity of the models with lower accuracy. Table 5 shows the accuracy of individual models and prediction
Figure 16: Results of ensembles (left plots) and model soups (right plots) of different sizes on ID (CIFAR-100 test set, top row) and OOD (CIFAR-100C, bottom row) for SSE and StarSSE with supervised pre-training. For OOD, we measure the average accuracy over all possible corruptions and severity values. Standard deviations are calculated over different pre-training checkpoints and/or fine-tuning random seeds.