To Stay or Not to Stay in the Pre-train Basin: Insights on Ensembling in Transfer Learning
Authors: Ildus Sadrtdinov
Dmitrii Pozdeev
Dmitry P. Vetrov
Ekaterina Lobacheva
Venue: NeurIPS 2023 poster (2023)
Keywords: ensembles
transfer learning
loss landscape basins
model soups
TL;DR: We study the effectiveness of the exploration of the pre-train basin and its close vicinity for ensembling in transfer learning.

Abstract: Transfer learning and ensembling are two popular techniques for improving the performance and robustness of neural networks. 
Due to the high cost of pre-training, ensembles of models fine-tuned from a single pre-trained checkpoint are often used in practice. 
Such models end up in the same basin of the loss landscape, which we call the pre-train basin, and thus have limited diversity. 
In this work, we show that ensembles trained from a single pre-trained checkpoint may be improved by better exploring the pre-train basin, however, leaving the basin results in losing the benefits of transfer learning and in degradation of the ensemble quality. 
Based on the analysis of existing exploration methods, we propose a more effective modification of the Snapshot Ensembles (SSE) for transfer learning setup, StarSSE, which results in stronger ensembles and uniform model soups.

To Stay or Not to Stay in the Pre-train Basin: Insights on Ensembling in Transfer Learning
Ildus Sadrtdinov
Dmitrii Pozdeev Dmitry Vetrov Ekaterina Lobacheva" HSE University "Technical University of Munich (TUM)
$Constructor University, Bremen *Independent researcher
isadrtdinov@hse ru, dmitrii wow.iiivozeeettu....mmm de
dretrov@conntruccor. university, lobacheva. www.jjjgaailllccooooomm
Abstract
Transfer learning and ensembling are two popular techniques for improving the performance and robustness of neural networks. Due to the high cost of pre-training, ensembles of models fine-tune mee-tuned from a single pre-trained checkpoint are often used in practice. Such models end up in the same basin of the loss landscape, which we call the pre-train basin, and thus have limited diversity. In this work, we show that ensembles trained from single pre-trained checkpoint may be improved by better exploring the pre-train basin, however, leaving the basin results losing the benefits of transfer learning and in degradation of the ensemble quality. Based on the analysis of existing exploration methods, we propose a more effective modification of the Snapshot Ensembles (SSE) for transfer learning setup, StarSSE, which results in stronger ensembles and uniform model soups.
Introduction
Ensembling neural networks by averaging their predictions is one of the standard ways to im- prove model quality. This technique is known improve not only the task-specific quality metric, g., accuracy, but also the model calibration [Lakshminarayanan et al. (017)) and robustness to noise (Gustafsson al. (2020), domain shift (Ovadia al., 2019)) and adversarial attacks (Strauss) et al. 2017). In practice, the importance of the solution robustness often goes hand in hand with the scarcity of data and computational resources for training. A popular way to obtain a high-que quality solution for a task with limited data using less computation is transfer learning, which initializes the training process on the target task with the model checkpoint pre-trained on large source task.
In this paper, we investigate the effectiveness of neural network ensembling methods in transfer learning setup since combining the techniques is non-trivial problem. The diversity of networks crucial for obtaining substantial quality gain with ensembling. Therefore, in classic deep ensembles (DE by Lakshminarayanan et al. (2017), networks are trained independently from different random initializations. However, in the transfer learning setup, networks are not trained from scratch but fine-tuned from pre-trained checkpoints. Fine-tuning networks from different independently pre- trained checkpoints results in a diverse set of models and a high-quality final ensemble but requires a huge amount of computation due to the large cost the pre-training process. Fine-tuning networks from the same pre- pre-trained checkpoint is much more resource-effectivee however, the resulting set of networks are usually located in the same basin of the loss landscape (Neyshabur et al., 2020). are less diverse, and their ensemble has lower quality (Mustafa et al.) 2020). So, the main question of our paper is the following: can we improve the quality of the ensemble of networks obtained by fine-tuning from the same pre-trained checkpoint by better exploring the loss landscape of the target task?
FFrstt two authors contributed equally.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

We focus on the image classification target tasks and consider two types of pre-training: super- vised (Kornblith al. 2019) and self--uper (Grill et al. 2020). First of all, in Section [ we confirm that in this setup, there is indeed substantial quality gap between ensembles of models fine-tuned from the same and different pre-trained checkpoints. Thus, methods aiming at closing this are gap are worth researching. We also show that models fine-tuned from the same pre-trained checkpoint lay in the same basin of the loss landscape, meaning there is no significant high-loss barrier on the linear interpolation between them. We refer to this basin as the pre-train basin throughout the paper.
To understand whether the ensemble quality can be improved by better exploring the pre-train basin or the close vicinity outside of it, we take inspiration from the literature on efficient ensembling methods in regular non non-transfer learning setup. We focus our study on cyclical learning rate methods, such as Snapshot Ensembling (SSE by Huang et al. (2017)), because they can either explore one basin of the loss landscape or go outside of the basin depending on the chosen hyperparameters. In Section 5.2 we show that with accurate hyperparameter tuning, these methods can explore the pre-train basin and produce ensembles of similar or even higher quality than the independent fine-tuning of several neural networks. However, they are not able to leave the pre-train basin without losing the advantages of transfer learning, followed by significant degradation in the quality of individual models and the whole ensemble. Thus, we conclude that the gap between ensembles of networks fine-tuned from the same and from different independently pre-trained checkpoints can not be closed by exploring the loss landscape of the target task using existing methods.
Based on the analysis of the behavior of the standard sequential variant of SSE in Section 533 we propose its parallel variant StarSSE, which is more suitable for exploring the pre-train basin in the transfer learning setup (see Section 5.4). StarSSE fine-tunes the first model from the pre-trained checkpoint in usual way and then trains all the following models, independently initializing them with the first fine-tuned one. In contrast to the SSE, StarSSE does not move further from the pre-trained checkpoint with each training cycle; therefore, it does not suffer from the progressive degradation of individual model quality. Even though StarSSE is technically very similar to the independent fine-tuning of all the models from the pre-trained checkpoint, it has an important conceptual difference: it separates 1) model fine-tuning from the pre-train checkpoint to the area with high-quality solutions for the target task and 2) exploration of this area for obtaining diverse models. As a result, StarSSE can collect a more diverse set of networks without sacrificing the quality of individual models.
Finally, we analyze different aspects of SSE and StarSSE that can be important for their practical application. Firstly, in Section 6 we show that the set of models, collected by SSE or StarSSE in the pre-train basin, can not only constitute an effective ensemble but also result in a strong uniform model soup (Wortsman et al. 2022: Rame et al., (022), i. e., model obtained by weight-averaging all models in the set. Then, in Section [ we study the robustness of the resulting ensembles and model soups to the test data distribution shifts. Lastly, in Section 8 we confirm that SSE and StarSSE
are effective in more practical large-scale experiments with CLIP model (Radford et al. (021 on the
ImageNet classification target task (Russakovsky et al. 2015). In all additional experiments, Star also outperforms the standard SSE
Our code is available
transfer
2 Ensembling methods in regular training
Deep ensemble (DE) is the most common method of neural network ensembling in a regular non- transfer learning setup. A DE consists of several networks of the same architecture trained separately from different random initializations using the same training methods and hyperparameters. Training from different initializations makes neural networks diverse and results in high-quality ensemble. However, separate training of all the networks requires a lot of resources. From the loss landscape perspective, DE can be called a global method: due to diverse initializations, networks in such an ensemble are independent and come from different basins of low loss (Fort et al. 2019).
To reduce training costs, many methods propose to train the networks jointly or consequentially, Most of these methods can be assigned to one of two groups based on the degree of coverage of different loss landscape basins: local and semi-local. Local methods (SWAG by Maddox et al. (2019). SPRO by [Benton et al. (202T), FGE by Garipov et al. (2018), etc.) train the first network and then efficiently sample other ensemble members from the basin of low loss around it. Semi-local methods (SSE

---

by Huang et al. (0117), cSGLD by Zhang et al. (2020), etc.), instead of focusing on one basin, try to explore a larger portion of the loss landscape by sequentially training networks using a cyclical learning rate schedule. At the beginning of each cycle, the learning rate is increased to help the network leave the current low-loss basin, and at the end of the cycle, it is decreased to obtain a high-quality checkpoint in the next basin to add to the ensemble. Because of the knowledge transfer between the consecutive models, such methods can use training cycles shorter than usual model training, which decreases the overall ensemble training cost. In terms of network diversity and the final ensemble quality, semi-local methods work better than local ones, given the same number of networks in ensemble. However, methods from both of these groups are inferior to the global ones (Ashukha et al. 2020, Fort et al., 2019).
The described categorization of ensembling methods into three groups. local, semi-local, and global, not striet. For example, FGE, which is usually assigned to the local group, and SSE, which is usually assigned to the semi-local group, are both based on cyclical learning rate schedules and can be used as either local or semi-local methods, depending on the hyperparameters. Also, local and semi-local methods can be combined with global ones by training several networks from different random initializations, then sampling some networks locally around each of them, and finally ensembling all the obtained networks (Fort et al. 2019). Moreover, recent works on loss landscape analysis hypothesize that two networks trained independently from different random initializations are effectively located in the same basin up to the neuron permutation (Entezari et al., [002]] Ainsworth iet al.] 2023) and activations renormalization (Jordan et al. (023). which blurs the boundaries between local and global methods. However, existing local and semi-local methods are not able to find as diverse networks as the global ones can, so in practice, these groups are still easily distinguishable.
If several networks are located in the same basin of low loss, they can be weight-averager instead of ensembling. The Stochastic Weight Averaging technique (SWA by [zmallov et (2018)))) averages several checkpoints from one standard training run models from local cyclical learning rate schedule method to improve the model quality and robustness.
3 Scope of the paper: transfer learning perspective
In the transfer learning setup, DE for the target task can be trained in two ways: using different separately pre-trained checkpoints to initialize each network or using the same pre-trained checkpoint for all the networks. In terms of the categorization from the previous section, only the former variant can be called global because the networks fine-tuned from the same pre-trained checkpoints are usually located in the same basin of low loss (Neyshabur al., 2020). Thus, we call the former variant Global DE while the latter is Local DE. Similarly to the relation between local and global methods in regular training, in the transfer learning setup, Global DEs contain more diverse networks and show better quality than Local DEs (Mustafa et al. 2020). At the same time, the difference in training cost between Local and Global DEs is even more pronounced because the pre-training cost is usually huge compared to the fine-tuning cost.
One of the goals of our paper is to understand whether the existing local and semi-local ensembling methods can be beneficial in the transfer learning setup and help close the quality gap between Local and Global DEs. We focus the situation when only one pre-trained checkpoint is available, as in Local DE, and study the effectiveness of cyclical learning rate methods. This group of methods contains such methods FGE, SSE, and cSGLD. We choose cyclical methods because they can work in both local and semi-local regimes and show the strongest results in closing the gap with global methods in the regular training setup (Ashukha al., 2020). In Appendix C we show that in the transfer learning setup, they are also more effective than other local methods. Cyclical methods have very similar structure and differ mostly in training scheme for the first network and learning rate schedules in the cycles. FGE trains the first network with a more standard long schedule, while SSE and eSGLD use the same schedule in all the cycles. Also, FGE uses short cycles with a triangular learning rate schedule and low maximum learning rate, while SSE and cSGLD use long cycles with a cosine learning rate schedule and relatively high maximum learning rate. In the paper, we mostly experiment with version of SSE, which fine-tunes the first network with fixed standard fine-tuning schedule and then trains all the following networks with the same cosine schedule, for which we vary the length and the maximum learning rate. Depending on the cycle hyperparameters, SSE can behave as either local method (short cycles with low maximum learning rate) or a semi-local method (longer cycles with higher maximum learning rate). We choose to fine-tune the first network in
3

---

fixed way for easier comparison with Local and Global DEs. The results for other variants of cyclical methods are similar and discussed in Appendix D
To the best of our knowledge, cyclical methods were not previously studied in the transfer learning setup. Moreover, based on our analysis of the SSE behavior, we propose its parallel version, StarSHE, which shows better results in this setup. Several recent works (Mustafa et al.) 2020; Wortsman et al. 2022, Rame al. 2022) study another technique for improving the diversity of Local DEs by training networks using different hyperparameters (learning rate, weight decay, data augmentation, etc.). In Appendix KJ we show that this technique can be also effective for Global DEs and StarSSEs, however, it is not able to close the gaps between the methods and change the comparison results.
Since models in a Local DE are usually located in the same basin of low loss, they can be weight- averaged to obtain a single model called a model soup (Wortsman et al. 2022 Rame et al. 2022). If the set of models to average is chosen properly using validation data, a model soup provides quality boost without the high computational costs of ensembles in the inference stage. Model soups usually show slightly inferior results on the in-distribution data compared to Local DE, but they are more robust to distributional shifts, We also study the weight-averaging potential of SSE and StarSSE in the transfer learning setup, including the robustness of the resulting models to distributional shifts.
4 Experimental setup
Data, architectures, and pre-training, In most of the experiments, we use a standard ResNet-50
architecture (He et al. 2016) and consider two types of pre-training on the ImageNet dataset (Rus
sakovsky et al. [ supervised and self-supervised with the BYO method (Grill et al. 2020)
We use independently pre-trained checkpoints in both cases. As for target tasks, choose three image classification tasks: CIFAR-10 (Krizhevsky et al. a). CIFAR-10 100 (Krizhevsky et b). and SUN-397 (Xiao al. (0100). For SUN-397, we use only the first train/test split of the ten specified in
the original paper. We also experiment with a CLIP model (Radford et al.) 2021) on the ImageNet
target task in Section 88 other non-natural image classification target tasks in Appendix N and Swin-T
transformers (Liu et al. (02T) in Appendix M
Fine-tuning of individual models. We replace the last fully-connected layer with a randomly initialized layer having an appropriate number of classes and then fine-tune the whole network using mini-batch SGD with batch size 256, momentum 0.9, and cosine learning rate schedule (Loshchilov and Hutter 2017). For each target task and each type of pre- pre-training, we choose the optimal number of epochs, weight decay, and initial learning rate for fine-tuning by grid search, allocating validation subset from the training set (see Appendix E for more detail).
The first network in SSE or StarSSE is always trained in the same manner as an individual fine-tuned network described in the previous paragraph with the optimal number of fine-tuning epochs and maximum learning rate. We denote these optimal hyperparameter values for each task as x1. For training all the following networks, we use the same learning schedule, for which we vary the number of epochs and maximum learning rate in a cycle. We consider different hyperparameter intervals for different tasks to cover all types of method behavior and denote the values relatively to the optimal ones for individual model fine-tuning (for example, x0.5 or x2).
SSE and StarSSE.
Aggregating results. All ensemble accuracy values presented in the plots are averaged over several runs, so mean and standard deviation are shown. For Local and Global DEs, we average over at least 5 runs (we fine-tune 5 networks from each pre-trained checkpoint and average over different pre-trainings and fine-tuning random seeds). For SSE and StarSSF, we average over two runs, each from a different pre-trained checkpoint.
Connectivity analysis. To analyze the behavior of ensemble methods, we look at the linear connec- tivity of ensemble members. Usually, two models are called linearly connected and laying in the same low-loss basin if there is no high-loss barrier on the linear interpolation between them, meaning the loss occurred when linearly connecting the models lower than the linear interpolation of their losses with the same interpolation coefficients (Entezari et al., (022). In our analysis, we use accuracy instead of loss. The two metrics behave similarly, however, the absolute accuracy values are easier to interpret. Hence, we examine whether ensemble members lay in the same high-accuracy area (high-accuracy basin) or if there is zone of lower accuracy on the linear interpolation between them dlw-accuracy barrier).

---

Table 1: Accuracy of ensembles of size 5 of ResNet-50 models with supervised (SV) and self-
supervised (SSL) pre-training. Green numbers indicate the gap between Local and Global DE.
Pre-train. Dataset
Local DE
Global DE
SSE (opt.)
StarSSE (opt.)
CIFAR- 100
85.65=0.12
86.82+0.14
(+1.17)
86.04
86.30
SV
CIFAR-I0
90.52=0.01
97.69 (+0.17)
97.30+0.05
97.39=0000
SUN-397
62.23
64.65++0.10
(+2.42)
63.63+t0.14
64.20=#0.21
CIFAR-100
$7.33am0 10.21
88.10.10.11
(+0.77)
87.1110.00
87 1..33.0.12
SSL
CIFAR-10 10
97.97 1.005
98.14::0.05
(+-.117)
17.94
98.06:10.02
SUN-397
65. 10.19
67.04
(+1.277
65.79
66.37 10.25
Supervised pre-training
Self-supervised pre-training
Buselinos
fix sp. x1, very In
Gx vary ep-
Boselines
Gix xl, vary ln:
fr very op.
Global DE
Global DE
Local
x1
Locd DE
test accuracy
86
Leat accuracy
SSE
87
2
3
2 ensemble 4lon
Atsomblle sl:0
#:oeernble blo s126
onseemblle
07
88
Lcol accuraacy 3t
StarSSE
accurack
86
60
2
2
5
ocsomble 8020
ensemble size
ensemble size
onsemblle size
Figure 1: Results of SSEs (top row) and StarSEEs (bottom row) of different sizes on CIFAR-I00 for different types of pre-training and varying values of cycle hyperparameters (maximum learning rate and number of epochs). Local and Global DEs are shown for comparison with gray dotted lines.
5 Empirical analysis of ensembling in transfer learning
5.1 Local and global deep ensembles
As first step of our empirical analysis of ensembling methods in transfer learning, compare the quality of Local and Global DEs in our setup. In Table we show the results for ensembles of maximal considered size 5. Also, the results for ensembles of variable sizes visualized in Figure [ with gray dashed lines (see Appendix [ for the results on other datasets). For both types of pre-training and all considered target tasks, there is significant quality gap between Local and Global DES. Thus, the problem of closing this gap is relevant to our setup.
We also confirm that networks from the Local DE lay in the same high-accuracy basin. Gray baseline lines in Figure show accuracy along line segments between random pairs of networks from the same Local DE (see Appendix [ for results on other datasets). There are no significant low-accuracy barriers in training and test metrics.
5.2 Local and semi-local Snapshot Ensembles
Since a Local DE only explores the pre-train basin, we experiment with the SSE method with varying values of cycle hyperparameters to answer two questions: 1) can we improve a Local DE quality by better exploring the pre-train basin, and 2) can we improve it even further by exploring the close vicinity outside the pre-train basin? In this section, we demonstrate the results only on the CIFAR-100 task; the results for other target tasks similar and presented in Appendix
5

---

SSE
StarSSE
acmurnyy, 96
100
test accuracy %
train accuracy.
86
train accuracy,
86
98
98
96
96
82
82
94
94
weights interpolation
weights interpolation
weights interpolation
weig interpolation
Local DE
optimal exp, 1
Local DE local
optimal exp, any
local
more semi-local
more
exp,
exp, -
exp, any
exp, any
Figure 2: Linear connectivity analysis of Local DEs, SSEs (left plots), and StarSSEs (right plots) CIFAR-100 with self supervisdd pre-training. We show train and test accuracy along line segments between random networks in Local DEs, between the first and the last (5-th) network in three differently behaving SSEs, and between the first and any other consequent network in three differently behaving StarSSEs. Hyperparameters for more local and for SSE and StarSSE, while hyperparameters for the optimal experiments may differ.
semi-local experiments the same
SSE
StarSSE
*e
100
train accuracy, *
100
Irain accuracy.
tnst aocurage
86
tast accurace
86
84
84
99
82
99
82
model number
model number
model number
model number
single mode]
optimal
single model
optimal exp
more local
more semi-local exp
more local exp
more semi-local exp
Figure 3: Train and test accuracy of individual models from three differently behaving SSEs (left
plots) and StarSSEs (right plots) CIFAR-100 with self-supervised pre-training (with single fine-tuned model for comparison). Hyperparameters for more local and more semi-local experiments are the same for SSE and StarSSE, while hyperparameters for the optimal experiments may differ.
We vary the maximum learning rate and number of epochs in SSE cycles the top row of plots in Figure[[] Varying these hyperparameters similarly changes method behavior because training with higher learning rate or for larger number of epochs allows the network to go further from the pre- train checkpoint. For low values of hyperparameters, especially the learning rate, SSE collects very closely located checkpoints; therefore, their ensembling does not substantially improve the prediction quality. When we increase the values, SSE collects more diverse checkpoints, and ensemble quality rises. However, for high hyperparameter values, the ensemble quality significantly degrades with the number of networks due degradation of the quality of individual models as they move further and further away from the pre-train checkpoint (see next sub-sections).
Overall, SSE with optimally chosen hyperparameters some cases even better, quality than Local DE but is not able to close the gap with Global DE (see Table (). Interestingly, SSE shows strong results for small ensembles, sometimes even matches the Global DE for ensembles of size 2, however, for large ensembles, SSE results become weaker (see Figure D SSE shows stronger improvements in the case of supervised pre-training, most likely because of the higher difference in the optimal hyperparameter values for SSE and usual Local DEs.
construct ensemble of similar, and in
5.3 Analysis of Snapshot Ensemble behavior
To better understand the properties of different types of SSE behavior and the reasons behind them, we analyze three experiments for each setup more detail. We consider the following three experiments: the more local experiment (with low learning rate and very slow-growing ensemble quality), the more semi-local experiment (with high learning rate, large number of epochs, and ensemble quality degradation for large ensemble sizes), and the optimal experiment (with medium learning rate and

---

number of epochs, and the best quality for an ensemble of size 5). See Appendix B for the exact values of hyperparameters for these three experiments. In this section, we demonstrate the results only for experiments with self-supervised pre-training and the CIFAR-100 dataset. The results with
supervised pre-training are similar and presented in Appendix F
In the left pair of plots in Figure 2 we show how train and test accuracy behave along the line segment between the first and the last (5-th) network in the SSE. Metric values along the line segment between two random networks from the same Local DE are shown as baselines. From the loss landscape perspective, more local and more semi-local experiments indeed demonstrate the expected local and semi-local behavior, respectively. Interestingly. the behavior of optimal experiments is also local, without low-acuuracy barriers. These results confirm that SSE can behave as both local and semi-local methods and that its local variant provides the best results. In Appendix G we provide additional locality analysis based on the accuracy behavior on 2D planes in the weight space.
In the left pair of plots in Figure 3 we show how train and test accuracy of individual models in SSE changes with each cycle. In all the experiments, model test accuracy degrades with each cycle, even though the train accuracy increases or stays at the maximum value. From this, we can conclude that the further individual models walk away from the pre-train checkpoint, the more they overfit the training data of the target task and, as a result, lose the advantages of transfer learning.
SSE with optimal hyperparameters can improve over Local DE by better exploring the pre-train train basin and collecting more diverse networks (see Appendix[]oo the detailed diversity analysis). The diversity of these networks even makes up for the gradual decrease in individual model quality. However, leaving the pre-train basin with SSE results in the most pronounced degradation of individual model test accuracy, which leads to decreasing ensemble quality. We also hypothesize that due to the decrease in the quality of individual models, SSE with any values of parameters is not able to effectively construct large ensembles in the transfer learning setup (any SSE plot will start decrease if we grow the ensemble size further).
Whether staying in the pre-train basin is essential condition to take advantage of knowledge transfer and obtain high-quality ensembles or the considered methods explore the vicinity of the pre-train basin in a non-effective way is an interesting question for future research. We suspect that the main reason why SSE can not leave the pre-train basin without losing the transfer learning advantage is the fact that SSE only looks at the target task loss landscape without taking into consideration the source task one. [Shwartz-Ziv et al. (2022) show that accounting for the source loss landscape shape around the pre-trained checkpoint can improve fine-tuning accuracy of individual models and local ensembles. However, to do so, they fit a normal prior on the source task, centered in the pre-trained checkpoint, which most likely can be effective only locally in the pre-train basin (see Appendix L for additional experiments on SSE with similar regularizations). Hence, more advanced techniques should be developed to account for the source loss landscape shape outside of the pre-train basin.
5.4 StarSSE a better version of Snapshot Ensemble for transfer learning
Our study of SSE behavior shows that, in the transfer learning setup, standard cyclical learning rate methods suffer from individual network quality degradation due to moving too far from the pre-train checkpoint and overfitting to the target training data, and this effect becomes more pronounced with each cycle. SSEs with long cycles and high learning rates are affected the most by this, but at the same time, they show high quality for small ensemble sizes (see results for ensembles of size 2 at the top row of Figure [D. So, even though the second network found by SSE is weaker as an individual model, it may be a better candidate for the ensemble with the first one than another independently fine-tuned network, as in Local DE.
Based on these observations, we propose parallel version of SSE, StarSSE, which is more suitable for the transfer learning setup. StarSSE trains the first model with the standard fine-tuning schedule, as in Local DE or SSE. Then, all the following models are trained with the same schedules as in SSE cycles but in parallel rather than sequential manner. Each of these models is initialized with the weights of the first fine-tuned model. As a result, we have a "star" with the first model in the center and all other models at the ends of the "rays". Due to the parallel structure, StarSSE does not move further from the pre-trained checkpoint with each training cycle as SSE, but at the same time, explores the area with good candidates for ensembling around the first fine-tuned network.
7

---

We show how StarSSE with different cycle hyperparameters behaves in the bottom row of plots in Figure I (for consistency with SSE. we call hyperparameters for training all the networks except the first one as cycle hyperparameters). The results on other datasets are presented in Appendix H StarSSE demonstrates more stable results for medium and high values of cycle hyperparameters than SSE, and as a result, it produces better larger size ensembles. Even though for the highest considered hyperparameter values, StarSSE quality still decreases with increasing ensemble size, its degradation is much less severe. As can be seen from Table I the optimal StarSSE outperforms the optimal SSE,
To better understand the difference between StarSSE and SSE behavior, we conduct the same analysis of linear connectivity and individual model quality for three distinct StarSSE experiments, as in the previous section for SSE. For easier comparison, we choose the same cycle hyperparameters for more local and more semi-loce experiments as for their SSE counterparts, For the optimal experiment, we choose its own hyperparameter values to compare the best results of the methods. The exact values of hyperparameters can be found in Appendix [B] In the right pair of plots in Figure we show how train and test accuracy of StarSSE behave along the line segment between the first and any of the following models (it does not matter which model to choose because all of them are trained independently in the same manner). The more semi-local StarSSE experiment demonstrates much more local behavior from the loss landscape perspective than the SSE one, which confirms that StarSSE does not go too far from the pre-train train basin, even with relatively high hyperparameter values. The optimal StarSSE experiment demonstrates local behavior; therefore, StarSSE, similarly to SSE, should be used only as a local method in the transfer learning setup. As shown in the right pair of plots in Figure all the models in StarSSE, starting from the second one, have a very similar quality, which makes StarSSE more stable for larger ensemble sizes. For the optimal experiment, the quality degradation between the first and the following models is very low and can be compensated by the model diversity in StarSSE to achieve higher results than Local DE (see Appendix P for detailed diversity analysis).
In conclusion, StarSSE is more effective alternative to cyclical learning rate methods for the transfer learning setup, which can work with higher values of cycle hyperparameters to obtain diverse models without gradual degradation of individual model quality. StarSSE is technically very similar to Local DE, however, there is an important conceptual difference between them. StarSSE separates the two processes: 1) model fine-tuning from the pre-train checkpoint to high-quality area for the target task, and 2) exploration of this area for obtaining diverse models. On the contrary, Local DE tries to do both of them jointly each time it fine-tunes network. The design of StarSSE allows it to make exploration with higher hyperparameter values, which are suboptimal for the initial fine-tuning process, and obtain more diverse set of models. The fact that optimal hyperparameter values for StarSSE are higher than the optimal x 1 values for Local DE (see Figure supports this reasoning.
6 Soups of SSE and StarSSE models
In the previous section, we show that SSE and StarSSE methods can be effective in transfer learning but only in constructing local ensembles. Even though the locality usually leads to limited model diversity, also has its benefits: it allows to weight-average set of models to obtain a single, more accurate, and robust model. To constitute a strong weight-averaged model, a set of models needs to lie in the same low-loss basin and have convex loss landscape between the models. Diversifying the models in set can improve the averaging results; therefore, model soups (Wortsman) et al. 2022 Rame et al. (2022), which average independently fine-tuned models, show better results than S SWA Izzmillo et al.] 2018), which averages several checkpoints from the same training run. Moreover, for even higher diversity, model soups are usually constructed from the models fine-tuned with variable hyperparameters. However, the relation between the final quality and the diversity of the models is not as straightforward for weight-averaging as for ensembles because diversifying models too much may result in a non-convex loss between them. Because of this, averaging all fine-tuned models into a uniform model soup does not result in a high-quality model, and, to constitute an effective greedy model soup, a subset of models needs to be chosen using the validation data.
Linear connectivity analysis in F shows that averaging two models from the Local DE does not lead to a strong model, while models from optimal SSE and StarSSE experiments lay in a more convex part of the pre-train basin and average much better. This observation motivates us to investigate the weight-averaging of models from SSE or StarSSE to construct uniform model soups. We compare ensembles and uniform model soups of models obtained with Local DE, SSE, and StarSSF in the top row in Figure (see Appendix H for results with supervised pre- pre-training). Firstly, vvighi-averaging
8

---

Ensembles
Model soups
Buselioss
local sap
Buselines Local DE
local exp
exp
Global DE
optimal wap
optimal eig
optimal wep
optimal exp
Local DE
more soms local tsp
more semi local exp
soup
more somi local exp
more semi local. exp
SSE
StarSSE
SSE
StarSSE
test uccuracy. %
88
88
88
8.7
87
tast accuraey
877
87
TD
86
86
86
86
85
85
85
85
2
4
2 eusemble size
2
ensemble sizee
soup size
soup size
SSE
StarSSE
SSE
StarSSE
test accuracy.
test aocuracy.
OOD
60
60
600
60
5.8
58
58
58
2
4
2 3 4 ensemble size
2
4
2
3
4
ensemble size
soup size
soup size
Figure 4: Results of ensembles (left plots) and model soups (right plots) of different sizes on ID (CIFAR-100 test set, top row) and OOD (CIFAR-100S 100C, bottom row) for SSE and StarSSE with self-supervised pre-training. For OOD, we measure the average accuracy over all possible corruptions and severity values. Standard deviations are calculated over different pre-training checkpoints and/or fine-tuning random seeds.
Local DE models does not improve the quality over individual models. Model soups from more local and optimal SSE and StarSSE experiments show better results than an individual fine-tuned model. However, for the more local experiments, the improvement is minor due the low model diversity. Weight-averaging networks from the more local SSE in our setup is very close to standard SWA. Models from more semi-local SSE and StarSSE experiments can not constitute adequate model soups due to the low-accuracy barriers between the models. Comparison between SSE and StarSSE optimal experiments shows the advantage of the latter: the soup of StarSSE models demonstrates better results than from SSE ones and matches the quality of the ensemble of the same models.
To conclude. StarSSE can collect set of models in the pre-train basin, which are diverse enough to constitute a strong local ensemble and, at the same time, are located in convex part of the basin and result in strong uniform model soup after weight-averaging. usually fine-tuned with varying hyperparameters for higher diversity, and StarSSE can also benefit from this hyperparameter diversification (see Appendix K). This makes the further analysis of their combination a promising direction for future research.
standard model soups, models are
7 Robustness analysis
One of the main benefits of ensembles and model soups is their robustness to different types of test data modifications. In this section, we evaluate the robustness properties of the ensemble and soup variants of SSE and StarSSE on the corrupted version of the CIFAR- 100 dataset. CIFAR- dataset (Hendrycks and Dietterich] 2019) includes 19 synthetic corruptions (c. g.. Gaussian noise or JPEG compression), each having different severity values. We compare the performance of all the methods on the in-distribution CIFAR- 100 (ID) and the out-of-distribution CIFAR- 100C (OOD) in Figure 4 (see Appendix for results with supervised pre- vreirrainnng) For SSE and StarSSE, we consider the same three differently behaving experiments that are used in the previous sections.
First of all, we observe that more local ensembles lose their potential on OOD completely and have near-constant accuracy for all the considered ensemble sizes. In contrast, the behavior of more semi-local ensembles very similar on ID and OOD in comparison to the Local DE. This difference indicates that model diversity is even more important under domain shifts. The quality of optimal

---

Ensembles
Model soups
Basalines
local sap
local sap
Buselines
local sap
eap
Local DE
optimel eep
more somi local exp
more semi local
optimal egg
+
soup
Local DE
optimal exp
more somt local exp
optimal exp
more soms local exp
SSE
StarSSE
SSE
StarSSF
test acouras %
81.0
81.0
test accurrae 90
81.0
81.0
80.5
80.5
80.5
80.5
80.0
80.0
80.0
80.0
79.5
79.5
79.5
79.0
79.0
79.0
79.0
2
3
4
5
2
3
ensemble
enssomble size
soup size
soup size
Figure 5: Results of ensembles (left plots) and model soups (right plots) of CLIP models on ImageNet. Three differently behaving SSE and StarSSE experiments are shown with Local and Global DEs for comparison. Hyperparameters for more local and more semi-local experiments are the same for SSE and StarSSE, while hyperparameters for the optimal experiments may differ.
ensembles also decreases compared to Local DE, agreeing with their local behavior. Overall, ensemble variants of SSE and StarSSE are less robust than DEs to the test data corruption. We discuss the reasons for such behavior in more detail in Appendix !
For model soups, we observe similar changes in the behavior of more local and more semi-local experiments. The former can not provide a better quality model on OOD, while the results of the latter even improve in comparison the soup of Local DE models. Thus, model diversity matters for the soups too. Interestingly, optimal soups significantly outperform optimal ensembles, while on the clean data, these two paradigms show almost the same results. This observation matches with the findings of Wortsman et al. (2022), who claim the superiority of standard model soups over ensembles under distribution shifts. Overall, the best results on OOD data are achieved with an optimal soup of StarSSE models (of those trained from a single pre-trained checkpoint). It not only outperforms its ensemble variant but also provides considerably better results than Local DE. Thus, a soup of StarSSE models is good option for fine-tuning robust model given a single pre-trained checkpoint.
8 Large-scale experiments
Our main analysis of SSE and StarSSE methods is focused on moderate-size networks and smaller target tasks be able to accurately compare the results with both Local and Global DE baselines. However, to confirm that the main conclusions of the paper stand in more practical large-scale setups, we experiment with the CLIP ViT-B/32 model pre-trained on the large dataset of the image-text -ext pairs and the ImageNet target task. In these experiments, we drop the Global DE baseline due to its extremely high computational cost. Figure we compare the results of three differently behaving SSEs and StarSSEs (see Appendix B for specific hyperparameter values) and the Local DE baseline. SSE and StarSSE with optimal hyperparameter values significantly outperform Local DE both as ensembles and as uniform model soups. The comparison of StarSSE and SSE confirms previously discussed benefits of the parallel training strategy: StarSSE ensembles and model soups, trained with high hyperparameter values, show less quality degradation than their corresponding SSE counterparts. We conclude that StarSSE is effective for constructing better local ensembles in practical large-scale setups too.
9 Conclusion
In this work, we study the effectiveness of cyclical learning rate methods for training ensembles from a single pre-trained checkpoint in the transfer learning setup. We demonstrate that while better exploration of the pre-train basin with these methods can be beneficial, they can not leave the basin without losing the advantages of transfer learning and significant quality degradation. Based on our analysis, we also propose a parallel version of the SSE method for transfer learning, StarSSE. Models from StarSSE not only constitute a better ensemble but also lay a more convex part of the pre-train basin and can be effectively combined into a high-quality model soup.
10

---

Acknowledgments and Disclosure of Funding
We would like to thank Maxim Kodryan, Arsenii Kuznetsov, and the anonymous reviewers for their valuable comments. Also, are very grateful to Polina Kirichenko and Timur Garipov for presenting the paper at the NeurIPS poster session, which we could not attend, unfortunately. The results on ResNet with self-supervised pre-training were obtained by Ildus Sadrtdinov with the support of the grant for research centers in the field of Al provided by the Analytical Center for the Government of the Russian Federation (ACRF) in accordance with the agreement and the agreement with HSE University
the provision of subsidies
(identifier of the agreement 17702002-00113 The empirical results were supported in part through the computational resources of HPC facilities at HSE University (Kostenetskiy et al. 2021.
20000D730221SQQ000
References
Ainsworth, S., Hayase, J., and Srinivasa, S. (2023). Git reabsin: Merging models modulo permutation symmetries. In International Conference on Learning Representations (ICLR).
Ashukha, A., Lyzhov, A., Molchanov, D., and Vetrov, D. (2020). Pitfalls of in-domain uncertainty esti- mation and ensembling in deep learning. In International Conference on Learning Representations (ICLR).
Benton, G. W., Maddox, W. J., Lotfi, S., and Wilson, A. G. (2021). Loss surface simplexes for mode connecting volumes and fast ensembling. In International Conference on Machine Learning (ICML).
da Costa, V. G. T., Fini, E., Nabi, M., Sebe, N., and Ricci, E. (2022). solo-learn: A library of superviue methods for visual representation learning. Journal of Machine Learning Research (JMLR), 23(5)).
Entezari, R., Sedghi, H., Saukh, O., and Neyshabur, B. (2022). The role of permutation invari- ance in linear mode connectivity of neural networks. In International Conference on Learning Representations (ICLR).
Ericsson, L., Gouk, H., and Hospedales, T. M. (2021). How Well Do Supervieed Models Transfer? IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Fort, S., Hu, H., and Lakshminarayanan, B. (2019). Deep ensembles: A loss landscape perspective. arXiv preprint arXiv: 1911::191..02757.
Garipov, T., Izmailov, P., Podoprikhin, D., Vetrov, D. P., and Wilson, A. G. (2018). Loss surfaces, mode connectivity, and fast ensembling of dnns. In Advances in Neural Information Processing Systems (NeurIPS).
Gontijo-Lopes, R., Dauphin, Y., and Cubuk, E. D. (2022). No one representation to rule them all: Overlapping features of training methods. In International Conference on Learning Representations (ICLR).
Grill, J.-B., Strub, F., Altché, F,, Tallec, C., Richemond, P., Buchatskaya, E., Doersch, C., Avila Pires, B., Guo, Z., Gheshlaghi Azar, M., Piot, B., kavukcuoglu, k., Munos, R., and Valko, M. (2020). Bootstrap your own latent new approach to self-supervised learning. In Advances in Neural Information Processing Systems (NeurIPS).
Gustafsson, F. K., Danelljan, M., and Schön, T. B. (2020). Evaluating scalable bayesian deep learning methods for robust computer vision. IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops.
He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In IEEE Conference Computer Vision and Pattern Recognition (CVPR).
Hendrycks, D. and Dietterich, T. (2019). Benchmarking neural network robustness common corruptions and perturbations. International Conference Learning Representations (ICLR).

---

Huang, G., Li, Y., Pleiss, G., Liu, Z., Hopcroft, J. E., and Weinberger, K. Q. (2017). Snapshot ensembles: Train 1, get M for free. In International Conference on Learning Representations (ICIR).
Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D. P., and Wilson, A. G. (2018), Averaging weights leads to wider optima and better generalization. In Conference on Uncertainty in Artificial Intelligence (UAI).
Jordan, K., Sedghi. H., Saukh, O., Entezari, R., and Neyshabur, B. (2023). Linear mode connectivity of deep neural networks via permutation invariance and renormalization. In International Conference on Learning Representations (ICLR).
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan,
J., Ramalho, T., Grabska-Barwinska Barwinska, A., Hassabis, D., Clopath, C., Kumaran, D., and Hadsell, R. (2017). Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Science, 1 114(13):3511-3222
Kornblith, S., Shlens, J., and Le. Q. V. (2019). Do better imagenet models transfer better? In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
Kostenetskiy, P. S., Chulkevich, R. A., and Kozyrev, I. (2021). HPC resources of the higher school of economics. Journal of Physics: Conference Series, 1740:012050
Krizhevsky, A., Nair, V., and Hinton, G. CIFAR-10 (canadian institute for advanced research).
Krizhevsky, A., Nair, V., and Hinton, G. CIFAR-100 (canadian institute for advanced research).
Lakshminarayanan. B., Pritzel, A., and Blundell, C. (2017). Simple and scalable predictive uncer- tainty estimation using deep ensembles. In Advances in Neural Information Processing Systems (NeurlPS).
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. (2021). Swin trans- former: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV).
Loshchilov. I. and Hutter, F. (2017). SGDR:
Stochastic gradient descent with warm restarts. In International Conference on Learning Representations (ICIR).
Loshchilov, I. and Hutter, (2019). Decoupled weight decay regularization. International Conference on Learning Representations (ICLR).
Maddox, W. Izmailov, P., Garipov, T., Vetrov, D. P., and Wilson, A. G. (2019). A simple baseline for bayesian uncertainty in deep learning. In Advances in Neural Information Processing Systems (NeurIPS).
Mustafa, B., Riquelme, C., Puigcerver, J., Pinto, S., Keysers, D., and Houlsby, N. (2020). Deep ensembles for low-data transfer learning. arXiv preprint arXiv:2010.06866
Neyshabur, B., Sedghi, H., and Zhang, C. (2020). What is being transferred in transfer learning? In Advances in Neural Information Processing Systems (NeurIPS).
Ovadia, Y. Fertig, E., Ren, J., Nado, Z., Sculley, D., Nowozin, S., Dillon, J., Lakshminarayanan, B., and Snoek, J. (2019). Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. In Advances in Neural Information Processing Systems (NeurIPS).
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E.,, DeVito, Z., Raison, M., Tejani, A., Chil- amkurthy, S.. Steiner, B., Fang, L., Bai, J., and Chintala. S. (2019). Pytorch: An imperative style, high- perffomrrfeeeeee deep learning library. In Advances in Neural Information Processing Systems (NeurIPS), pages 8024-8035.
Peng, X., Bai, Q. Xia, X., Huang, Z.,, Saenko, K., and Wang. B. (2019). Moment matching for multi-source domain adaptation. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV).
12

---

Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, (2021). Learning transferable visual models from natural language supervision. In Meila, M. and Zhang, T., editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8748 8763. PMLR.
Rame, A., Kirchmeyer, M., Rahier, T., Rakotomamoniy, A., patrick gallinari, and Cord, M. (2022). Diverse weight averaging for out-of-distributin generalization. In Advances in Neural Information Processing Systems (NeurIPS).
Ritter, H., Botev, A., and Barber, D. (2018). A scalable laplace approximation for neural networks.
In International Conference on Learning Representations (ICLR).
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C.,, and Fei-Fei, (2015). Imagenet large scale visual recognition challenge. In International Journal of Computer Vision (IJCV).
Shwartz-Ziv, R., Goldblum, M., Souri, H., Kapoor, S., Zhu, C.. LeCun, Y., and Wilson, A. G. (2022). Pre-train your loss: Easy bayesian transfer learning with informative priors. Advances in Neural Information Processing Systems (NeurIPS).
Strauss, T., Hanselmann, M., Junginger, A., and Ulmer, H. (2017). Ensemble methods defense to adversarial perturbations against deep neural networks. arXiv preprint
Wang. X., Peng. Y., Lu, I.., Lu, Z.,, Bagheri, M., and Summers, R. M. (2017). ChestX-ray8 Hospital- scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Wortsman, M., Ilharco, G., Gadre, Y., Roelofs, R., Gontijo-Lopes, R., Morcos, A. S., Namkoong, H., Farhadi, A., Carmon, Y., Kornblith, S., and Schmidt, L. (2022). Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference Machine Learning (ICML).
Xiao, J., Hays, J., Ehinger, K. A., Oliva, A., and Torralba, A. (2010). Sun database: Large-scale scene recognition from abbey to zoo. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR).
Zhang, R., Li, C., Zhang, J., Chen, C., and Wilson, A. G. (2020). Cyclical stochastic gradient MCMC for bayesian deep learning, In International Conference on Learning Representations (ICLR).
13

---

A Limitations and societal impact
The main limitation of our work is choosing the hyperparameter values for Local and Global DEs based on the validation accuracy of individual models. Even though it is standard practice for DEs, such hyperparameters may lead to suboptimal ensemble quality. However, hyperparameters for the training of the first SSE and StarSSE models, so their quality may also be suboptimal. Also, an extension of the study to other domains may be interesting addition to work.
used the same
To the best of our knowledge, our work does not have any negative societal impact, except the substantial computational costs of the conducted study. However, our work improves understanding of the ensembles and model soups construction using only one pre-trained checkpoint, which can help to reduce computational costs in future projects.
B Experimental setup details
All the described experiments were carried out using the PyTorch framework (Paszke et al. 2019).
Pre-training of ResNets. For the supervised case, we use available ResNet-50 checkpoints pre- trained by Ashukha et al. (2020). We take five of them trained from different random initializations with the same hyperparameters. For the self-supervised case, we also pre-train five checkpoints from scratch for 1000 epochs using the hyperparameters from the original B YOL paper. We use the solo-learn library for YOL pre-training (da Costa et al. 2022).
Swin-T. We pre-train five Swin-T T checkpoints from scratch in a supervised manner on ImageNet for 300 epochs. We use the official code and hyperparameters from the original Swin paper. For fine-tuning of individual models for Local and Global DEs, we use AdamW (Loshchilov and Hutter 2019) and linear learning rate warm-up for 10% of the fine-tuning epochs. For SSE and StarSSE, we also use AdamW for each cycle but the learning rate warm-up is applied only in the first cycle.
Fine-tuning hyperparameters grid search for ResNet and Swin-T models. We divide the training data into training and validation subsets in to 10 ratio for CIFARIO, 10, CIFARI00 100 and in 1 to 5 ratio for SUN397 in the work Kornblih et al. 2019) and 1 to 5 ration for Clipart and Chest-X. We use grid search for the values of training epochs, initial learning rate and weight decay giving the best quality of individual fine-tuned networks on validation subsets. For ResNet-50 (both types of pre- tree-traiing), we consider weight decays from [2- 10 10-5, 1. 10
-5- 5- 10 and initial learning rates 20 For Swin-T, 5. 10-100- 10 10-1, 10-1, 10-3)
from [5 the ranges are [10-2, 2.- 10-2, 10-11 and [2.5
10 3 2.5 10 5.
50-10- 10-3 3 10 2.5
10 2
5. 10
10
respectively. As for the number of training epochs, we determine the value corresponding to 20000 SGD steps (this procedure is taken from Kornblith et al. (2019); the value differs across datasets, making up 102 for CIFAR- 10, CIFAR- 100, 256 for SUN-397, for Chest-X and 19 for Clipart) and consider its multipliers from the list [x 0.031225 x0.0625, 10.125, x0.25, x0.5, x rounded down. The resulting optimal sets of hyperparameters are shown in Table 2]
CLIP. We use CLIP ViT-B/32 model pre-trained by the authors on a large dataset of image- text pairs. For fine-tuning individual models for Local DEs, we follow Wortsman et al.) (2022) using AdamW (Loshchilov and Hutter 2019), linear learning rate warm-up for 500 SGD steps and the best combination of the training hyperparameters from their random search (number of epochs 11, batch size 512, learning rate 1 1.23 10-6, label smoothing 0. 134, and augmentation Aagment with M = 19, = 0). For SSE and StarSSE, we also use AdamW, W, learning rate warm-up, and the same batch size, label smoothing. and augmentations in each cycle.
Hyperparameters of more local/optimal/more semi-local experiments. The hyperparameter multipliers for more local, SSE optimal, StarSSE optimal and more semi local experiments are presented in Table
Amount of computations. We use NVIDIA TESLA V100 and A100 GPUs for computations in our experiments. Total expenses on pre-training constitute 8K GPU hours for self-supervised ResNet-50 (BYOL) and supervised Swin-T. Fine-tuning models for Local and Global DEs and cyclical learning
14

---

Table 2: Optimal values of training epochs, initial learning rate and weight decay for fine-tuning an individual model.
Target dataset
Pre-train.
epochs
Ir
wd
CIFAR-I0
51
2.5 10
2 10
CIFAR-100
25
5-10-3
5010
SV ResNet-50
SUN-397
256
5-10-
5-10-
ChestX
8
2.5
5010-1
Clipart
19
2.5 10
5 10
CIFAR-I0
25
5 5-10-2
10-4
CIFAR-100
25
5 10-2
10 4
SSL
SUN-397
16
10-1
10-4
ResNet-50
ChestX
8
2 1-101
50
Clipart
19
10-1
2-
SV Swin-T
CIFAR-100
102
10-4
2.5
10-1
SUN-397
256
2.5 10-5
50102
Table 3: Considered multipliers of the learning rate and the number of epochs for more local, SSE optimal, StarSSE optimal, and more semi-local experiments for different datasets and types of pre- training.
more local
SSE
StarSSE optimal
more semillocal
Target dataset
Pre-train.
optimal
Ir
ep.
Ir
ep.
lr
ep.
ep.
CIFAR-I0
x0.25
x2
x0.25
x0.5
x2
CIFAR-100
x
x1
x2
x
x4
x
x8
x4
SV
SUN-397
x0.5
x1
x32
x0.125
x32
x0.125
x x32
x1
ResNet-50
ChestX
x0.25
x1
x4
x2
x2
x2
x8
x4
Clipart
x0.25
x0.5
x1
x0.5
x2
x0.5
x4
CIFAR-I0
x0.5
x1
x1
x2
x1
x4
CIFAR-100
x0.25
x1
x2
x0.5
x4
x0.5
x4
x4
SSL
SUN-397
x0.5
x1
x1
x1
x2
x1
x4
x8
ResNet-50
ChestX
x0.5
x1
x1
X1
x1
X2
x2
x4
Clipart
x0.5
x0.5
x4
x0.5
X0.5
x8
SV
CIFAR-100
x0.5
x0.25
x2
x0.25
X4
x0.5
x4
x1
Swin-T
SUN-397
x1
x1
X x16
x0.125
x x16
x0.25
x x16
x1
CLIP
ImageNet
x1
x0.25
x2
x1
x2
x1
x6
x1
VIT-B/32
rate ensembles took another 17K GPU hours, resulting in a total amount of approximately 25K GPU hours.
C Non-cyclical local ensemble methods
In this section, we experiment with non-cyclical local ensemble methods in the transfer learning setup and show that they are generally less effective than the eyclical ones. choose three non-cyclical methods: KFAC-Laplace (Ritter et al. 2018), SWAG (Maddox et al., 2019) and SPRO (Benton al. 202T). KFAC-Laplace the most simple one and does not require any additional training. Given usually trained network, it computes the Laplace approximation with block Kronecker
15

---

Table 4: Accuracy and diversity of ensembles of size for non-cyclical local methods, Local DE, and optimal SSE and StarSSE. Self-supervised pre-training.
Local DE
SSE
StarSSE
KFAC
Dataset
SPRO
SWAG
(optim.)
(optim.)
Laplace
CIFAR-
acc.
87.33 =0.21
87.11
87.63=00 0.112
86.01
86.46:10.0s
86.99
100
div. #3.56
71.57=5.91
71.51=2.00 22.08
35.38+2.07
33.94=16.90
36.74
SUN-
acc.
65.77:0.19
65.79
66.37
64.39
64.71
60.83
397
div.
12.48
57.69
60.74 1.94
17.87.10.94
24.37 19.87
15.75
factored (KFAC) estimate of the Hessian matrix and then samples network weights from the Gaussian distribution with the trained network as mean and the obtained KFAC estimate covariance matrix. SWAG also constructs the Gaussian distribution but uses the SWA solution as a mean and approximates the covariance matrix based on the SWA checkpoints. SWAG is shown to be the most effective alternative to cyclical methods in the regular training setup (Ashukha et al. (020). SPRO shows competitive results to SWAG by constructing simplex in the weight space with the given trained network as one of the vertexes and then sampling ensemble members from
In the experiments, we consider ResNet-50 with self-supervised pre-training on CIFAR-100 and SUN-397 datasets. We the official SWAG repository for the implementations of KFAC-Laplace and SWAG, and our own implementation of SPRO. We use grid search to determine the optimal hyperparameters for the non-cyclical local methods:
d
KFAC-Laplace. We search for the optimal sampling scale from the list [0.2, 0.4, 0.6, 0.8, 1.0, 1.2]. The resulting values are 1.0 for CIFAR-100 and 0.6 for SUN-397.
SWAG. We continue training the fine-tuned checkpoints with a constant learning rate (0.08 for CIFAR-I00 and 0.16 for SUN-397) for 19 more epochs, gathering new models at the end of each epoch (overall, there are 20 models counting the fine-tuned one). Then, we search for the value of sampling scale from the list [0.1, 0.2, 0.4, 0.6, 0.8, 1.0, 1.2] similar to KFAC-Laplace. The optimal values are 0.1 for CIFAR- 100 and 0.2 for SUN-397.
SPRO. We start with fine-tuned checkpoint and train 2 more points form 2-simplexes. We vary the learning rate (0.0125, 0.025, 0.05, 0.1) and the number of epochs (CIFAR-10M [5, 10, 20), SUN-397: [4, 8, 16)), well as the regularization coefficient ([10 6 10 1-05)) these points. The optimal tuples are (0.05, 10, 10-5) for CIFAR- and (0.05, 16, 10-6) for SUN-397.
5)) i) to train
In Table 4 we compare accuracy and diversity of 5-network ensembles obtained with the considered non-cyclical local methods, Local DE baseline, and optimal SSE and StarSSE (see Appendix for the definition of the diversity metric). KFAC-Laplace and SPRO show inferior accuracy and diversity compared both Local DE and SSE; hence, these methods' behavior is extremely local. SWAG demonstrates stronger results, even comparable to Local DE and SSE on SUN-397. However, StarSSE is more effective than SWAG on both datasets. In conclusion, even though cyclical methods can not benefit from the exploration of several low-loss basins in transfer learning setup, they are still preferable to other local methods, similarly to the regular training setup. Based on this analysis, we decided to focus on cyclical methods in the paper.
D Variants of cyclical ensemble methods
In this section, we discuss other variants of cyclical learning rate methods and explain our choice of SSE with a fixed first model fine-tuning schedule for the main study. In all the experiments, we consider 5e CIFAR- 100 with self-supervised pre-training.
16

---

Ensembles
Individual models
esp
more local exp
more local esp
more exp
+-
Global DE
optimal eep
op timal exp
+ model
single
optimal exp
optimall exp
Local DE
semi local exp
semi-local
semi
semni exp
SSE
Standard SSE
SSE
Standard SSE
88
88
Last acturacy. %
acturacy.
86
86
84
84
82
82
1
2
4 5
2 3
5
3
4
1
3 4
ensemble size
ensemble size
model number
model number
Figure 6: Results of ensembles (left plots) and individual models (right plots) for variant of SSE with a fixed schedule for the first network fine-tuning and Standard SSE, which uses the same hyperparameters in each cycle. CIFAR- 100, self-suue pre- pre-training. Hyperparameters for all three differently behaving experiments are the same for both methods.
D.1 SSE Standard SSE: why we fine-tune the first model with fixed schedule
Standard SSE (Huang et al. 2017) trains all the models using the same learning rate schedule in each cycle. However, we choose to use separate fixed schedule for the first model fine-tuning. This separation allows us to experiment with different hyperparameters for loss landscape exploration without harming the first fine-tuning cycle, which is responsible for moving from the pre-trained checkpoint to the high-quality area in the weight space for the target task.
In the right pair of plots in Figure 6 compare the behavior of individual model quality from variant of SSE and the standard SSE with the same hyperparameter values (we use the values for three differently behaving SSEs specified in Appendix (B). The first model from the optimal standard SSE experiment has adequate quality, which can also be improved a bit by choosing hyperparameter values optimal for Local DE instead of the ones optimal for our SSE. Since hyperparameter values of the more local and the more semi-local experiments are very different from the hyperparameters usually used for individual model fine-tuning Local DEs, the quality of the first model from standard SSE is very low in these cases. Moreover, going through more cycles does not help due to slow quality improvement for low hyperparameter values and progressive quality degradation for high hyperparameter values. As a result, the ensemble quality of standard SSE with different hyperparameter values varies a lot, not only due to changes exploration but also due to different fine-tuning quality in the first place (see the left pair of plots in Figure 6)
To solve the described problem, the first low-quality models are usually dropped from the standard SSE However, this can not help in the case of high hyperparameter values, which is important to us in this paper. Moreover, for different hyperparameter values, we would need to drop different number of first models, which would make the comparison of different SSEs and DEs much less clear.
D.2 eSGLD
eSGLD (Zhang et al.) 2020) technically is very similar to SSE and uses the same cyclical cosine learning rate schedule. However, instead of taking only one final model from each cycle, cSGLD samples several models the last epochs using Langevin dynamics. In our experiments, we compare SSE and cSGLD, which use separate hyperparameters for the first cycle due to the conclusion from the previous section. All other aspects of the cSGLD procedure are the same as in the original paper, which includes the number of sampled models in each cycle equal to 3.
In the right pair of plots in Figure we show that models from the same cycle of cSGLD are located in the same basins. Hence, exploration of the weight space mostly depends on obtaining the models from different cycles, which can indeed end up in different basins, similarly to the SSE ones (see the left pair of plots in Figure Based on these observations, we compare SSE and eSGLD in
17

---

Different cSGLD cycles
The same cSGLD eycle
Lrain accuracy. 96
100
accuracy, %
100
test accuracy; %
train accuracy.
98
84
84
96
96
82
82
0.0 0.25 0.5 0.75 1.0
0.0 0.25 0.5 0.75 1.0 weights interpolation
0.0
0.25 0.5 0.75 1.0
0.0 0.25 0.5 0.75 1.0 weights interpolation
weights interpolation
weights interpolation
Local
optimal exp.
Local
optimal exp, 3 semi-local exp,
local
semi-local
local
exp.
exp. 3 15
exp,
Figure 7: Linear connectivity analysis of cSGLD on CIFAR-100 with self-supervised pre-training. We show train and test accuracy along line segments between two random networks in Local DEs, between the last network of the first cycle (#3) and the last network of the last cycle (#15) of cSGLD (left plots), and between the first (#1 D) and the last network (#3) from the first cycle cSGLD (right plots).
Ensembles
Individual models
BaseTines
more local exp
more local exp op timall
Baselines
meew local eep
more local eag optimal more semi local exp
Global
Local DE
more semi local exp
more semi locall exp
model
more semi Jocal exp
SSE
cSGLD
SSE
cSGLD
tost accuracy. 96
88
88
accuracy.
87
87
84
84
82
82
86
86
ensemble le size
number of cycles
model number
model number
Figure 8: Results of ensembles (left plots) and individual models (right plots) for SSE and eSGLD. CIFAR- 100, self-supervised pre- tre-training. Hyperparameters for all three differently behaving periments are the same for both methods. Every cycle cSGLD adds new models, so cSGLD ensembles have 3 times more models than SSE, resulting in slightly better quality.
SSE
FGE
Baselines Global
Fx op. xt, vary le: x2
Fx ir X2, vary op.:
Baselines Global
ep. x1, vary In: x0.5
Ix 12, vary op.:
x2
Local DE
x4
Local
xt
test scrurarg H
R8
sccurace
87
onsembble size
ansomible $1.0
ensemble size
casemble size
Figure 9: Results of SSEs (left plots) and FGEs (right plots) of different sizes on CIFAR-100 with self-super pre-training for varying values of cycle hyperparameters (maximum learning rate and a number of epochs). Local and Global DEs are shown for comparison with gray dotted lines.
18

---

Interpolations of model weights
Individual models
train accuracy: %
100
86
train acouracy
100
test: acouracy %
84
84
82
82
96
test
80
99
80
weights interpolation
weights interpolation
model number
model number
Local DE
optimal exp,
local
more semi-local
single model
optimal exp
exp,
exp, -
more local exp
more semi-local exp
Figure 10: Linear connectivity analysis (left plots) and individual model quality (right plots) of three differently behaving FGEs on CIFAR-100 100 with self--uuprrvieed pre-training. On the left plots, we show train and test accuracy along line segments between two random networks in Local DEs and between the first and the last (5-th) networks in FGEs.
Figure fixing the number of cycles instead of the number of models in an ensemble. We conclude that cSGLD behaves very similarly to SSE and choose SSE for the main study because it gives almost the same quality using fewer networks.
D.3 FGE
FGE (Garipov et al. 2018) uses triangular schedule instead of the cosine one for each cycle and trains the first model with separate schedule the same manner as we do in our version of SSE. In our experiments, we train the first model of FGE with the same hyperparameters as the first model of SSE.
In Figure 9 we show the results of FGE with different cycle hyperparameters and compare them to the SSE ones. FGE experiments show similar types of behaviors, as we discussed for SSE in the main text. However, due to the triangular learning rate schedule, FGE behaves more locally than SSE with the same hyperparameter values. We also analyze the loss along line segments between the models from FGE and their individual quality in Figure TO The results show that FGE can work in local and semi-local regimes, and achieves the optimal ensemble quality in the local one.
E Ensembling results of SSE and StarSSE
In this section, we provide additional results of SSE and StarSSE on CIFAR-IO and SUN-397 datasets. Figure I is complementary to Figure I from the main text and compares SSE and StarSSE with different hyperparameter values to Local and Global DEs. The results for self-supervised pre-training very similar between different datasets. With supervised pre-training, the difference between SSE/StarSSE and Local DE varies for different datasets, probably due to higher variance in optimal hyperparameters. For SUN-397, SSE and StarSSE show very strong results close to the Global DE, while for CIFAR-10, their results are inferior to the Local DE.
Analysis of SSE and StarSSE behavior
In this section, we provide additional plots on the analysis of SSE and StarSSE for the case of supervised pre-training. Figure i2 analogous to [ the main text and demonstrates the behavior of train and test accuracy along line segments between the models from SSE and StarSSE. Figure 13 is analogous to Figure from main text and demonstrates models from SSE and StarSSE. The general observations are the same as in Sections 5.3 and [.4 Ultimately, SSE and StarSSE may work either local semi-local regimes, but they achieve optimal results in the local one.
quality of individual
19

---

Supervised pre- pre-training
Self' supervised pre-training
Busliius
Ga wp
vacy
22,
very op.
Buselinus
up. xl, X0.5
vary
x2,
up-
Clobal DE
25.25
x1
x0.
Global DE
20.5
Local DE
x0.5
x0.5
Local DE
97.8
96.2
1ea aetarnnng 45 97.6
L02t lotacuuacc
598.0
SSE
97.48
97.8
97.8
CIFAR-10
97.0
97.4
3
97.8
97.6
StarSSE
97.4
97.8
97.2
97.6
96.8
97.4
ozsemble size
ensemble size
ensemble size
ensemble sizo
Buslinus DE
To upp vary
Tx Ir a32,
Buseiins DF
xl, lo:
le
+
tost acauray
SSE
60
80
SUN-397
64
oxsemble $120
ecaemble $12e
ensemble size
ensemble si2o
accurccyy
64
StarSSE
63
82
size
Figure 11: Results of SSEs and StarSSEs of different sizes on CIFAR-10 and SUN-397 for different types of pre-training and varying values of cycle hyperparameters (maximum learning rate and number of epochs). Local and Global DEs shown for comparison with gray dotted lines.
SSE
StarSSE
Lrain accuracy. %
100
test accuracy, %
acmuracy.
troin accuracy.
84
84
82
82
93
93
80
80
0.0 0,25 0.5 0.75 1.0
0.0 0.25 0.5 0.75 1.0 wrights interpolation
0.0 0.25 0.5 0.75 1.0
0.0 0.25 0.5 0.75 1.0 weights interpolation
weights interpolation
weights interpolation
more local
more semi-local exp, 1 -
more local expp. any
more semi-local exp, any
exp. 1 -
optimal exp,
optimal exp. any
Figure 12: Linear connectivity analysis of Local DEs, SSEs (left plots) and StarSSEs (right plots) on CIFAR-100 with supervised pre-training. We show train and test accuracy along line segments between two random networks in Local DES, between the first and the last (5-th) network in three differently behaving SSEs, and between the first and any other consequent network in three differently behaving StarSSEs. Hyperparameters for more local and more semi-local experiments are the same for SSE and StarSSE, while hyperparameters for the optimal experiments may differ.
20

---

SSE
StarSSE
train accuracy. %
100
train accuracp 6P 100
accuracy
84
test accuracy
84
99
99
82
82
3 4
3
4
2
3
2
3
model number
model number
model number
model number
single model
optimal exp more semi-local excp
single model
optimal exp
more local exp
more local exp
more semi-local exp
Train and test accuracy of individual models from three differently behaving SSEs (left plots) and StarSSEs (right plots) on CIFAR- 1-0 with supervised pre-training (with single fine-tuned model for comparison). Hyperparameters for more local and more semi-loca experiments are the same for SSE and StarSSE, while hyperparameters for the optimal experiments may differ.
Figure 13:
CIFAR- 100, test set
CIFAR- 100, train set
30
30
99
81
98
80
97
best acnuracy
train acnurec
75
150
0
150
300
0
75
150
0
150
300
96
SSB
80
80
80
95
72
94
40
68
40
93
0
0
*
0
0
*
92
first model
optimal
<60
first model
optimal
<90
10
>86
10
*
99
5
5
*
80
.97
StarSBE
50
100
150
300
776
rost aocummy
50
100
0
150
3000
troint acouracy
60
60
.95
30
30
30
30
68
-93
0
0
#
0
*
0
*
64
optimal
model
optimal
semi Jocal
local
semi local
local
Figure 14: 2D visualization of test and train accuracy around SSE and StarSSE models the CIFAR- 100 dataset with self-supervised pre-training. The markers with black outline correspond to the points generating the projection plane. The axes scale matches the actual distances between the points of the plane in the weight space. The stars without black outline in the SSE plots show the projection of SSE models from previous cycles (note that these only the projections: the models themselves lay outside of the projection plane and their actual accuracy is different to the one plotted).
21

---

Ensembles
Model soups
Baselines
local
Baselines
optimal wap
optimel
optimal eop
ogtimal nep
++
more somi local cup
mooe somi oup
more senu local exp
10.00 oop
SSE
StarSSE
SSB
StarSSE
Self super.
accurscc
98
test acnuracy,
97
97
97
CIFAR-10
ensemble size
ensemble size
soug sizze
soup size
SSB
StarSSE
SSE
StarSSE
97.6
97.6
97.6
97.6
Supervised
test acouracy
97.4
97.4
Lest BADOUERO
97.4
97.4
97.2
97.2
97.2
97.2
97.0
97.0
97.0
97.0
96.8
96.8
96.8
96.8
ensembbe size
easemble 5ize
soup size
soup size
SSE
StarSSE
SSE
StarSSE
accuracy
66
66
test accuracy
66
66
Self super.
64
64
84
SUN-397
ensemble size
ensemble size
soug sizee
soup Size
SSE
StarSSE
SSE
StarSSE
Lest acpuracy. %
64
tost accurary, %
64
6-1
Supervised
62
62
62
62
soup Size
solu
Figure 15: Results of ensembles (left plots) and model soups (right plots) of different sizes on CI and SUN-397 for SSE and StarSSE with supervised and self-supervised pre-training. Hyperparameters for more local and more semi-local experiments are the same for SSE and StarSSE on the same dataset, while hyperparameters for the optimal experiments may differ.
G Accuracy behavior on 2D planes in the weight space
In this section, we plot 2-dimensional visualizations of high-accuracy regions around SSE and StarSSE models similar to Garipov et al. (2018). To construct 2D visualization, one needs to select three points in the weight space and then plot the metric values over the plane generated by these points. Overall, for SSE, we consider four points: the first SSE model (i. e., a model from Local DE) and three networks from three differently behaving SSE experiments; each network is taken from the last (5-th) SSE cycle. Importantly, the three SSE experiments are all launched starting from the same first model, which is one of the four points considered. We then take all combinations of size three out of these four points and plot separate 2D plots for them. We repeat the same procedure for StarSSE models.
Each combination of three points forms a triangle, which is projected into the image plane. The first model projected into the point (0, 0). We make the r-axis parallel to the side connecting the first and the second models. The y-axis is parallel to the altitude of the triangle passing through the third model. The side and the altitude are projected, preserving their original length, so the whole plane is projected without distortion. Finally, we consider coordinate grid over the projected plane,
22

---

update the batch normalization statistics at the grid nodes (we use 10% of the training set for faster processing), and evaluate test and train accuracy these points.
The resulting 2D visualization plots are shown in Figure 14 In terms of train accuracy, the first and the last local and optimal SSE models are located in the same basin, while the last semi-local SSE model "goes round the corner" - it is almost linearly connected with the last optimal model but we see more significant barriers while connecting it to the first and the last local SSE models. The same observations hold for StarSSE. In terms of test accuracy, the first and the last local and optimal SSE models show similar results, however, the optimal SSE experiment goes much further from the first model and ends up "on the opposite wall" of the high-accurage region, while the last local SSE model stays "on the same wall" as the first model. This observation holds for StarSSE too. It also matches our model soups experiments, where optimal SSE/StarSSE usually demonstrate substantially stronger results than the local ones.
Semi-local SSE/StartSSE end up very close to the border of the high-accuracy region, both in terms of test and train metries, the considered 2-dimensional weight subspaces. When we look the segment connecting the first and the last semi-local models, we see an abrupt degradation of quality for the models laying beyond the semi-local end. The first and the last local and optimal models are much more stable in this sense. This effect makes us suspect that leaving the pre-train basin makes semi-local SSE/StarSSE converge to sharper minima. Moreover, the track of semi-local SSE experiment shows that the 2-nd SSE cycle moves further from the first model than all subsequent cycles (at least when projected into the image plane). This indicates that even single cycle with high hyperparameters forces the model to leave the pre-train basin and shows why semi-local StarSSE is not efficient (despite behaving more locally than the semi-local SSE).
H Results of soups of SSE and StarSSE models
In this section, we provide additional results of SSE and StarSSE soups on different datasets and with different types of pre-training. Figure 16 is analogous Figure 4 from the main text, and its first row demonstrates the results on the CIFAR- R-100 dataset with supervised pre-training. Additionally, Figure 15] shows results for CIFAR- 10 and SUN-397 for both supervised and self-supervised pre- training. Similarly to Appendix E the results for self-supervised pre-training are very similar for all datasets, while for supervised pre-training they vary more. Also, SSE and StarSSE experiments on CIFAR-10 10 with supervised pre-training become unstable for high hyperparameter values, therefore, the more semi-local experiments here behave similarly to the optimal ones.
I OOD
In this section, we show the robustness analysis of SSE and StarSSE with supervised pre-training complementary to Section 7 Figure [ analogous to Figure 4 from the main text and compares results on in-distribution CIFAR- 100 and -iit--idittiittiionn CIFAR-1006 data. Results for the supervised case are similar to the self-supervised one: SSE and StarSSE are less stable to the data corruption than Local and Global DEs, however, soups of StarSSE models still show strong results compared to Local DE.
J Prediction diversity
In this section, we quantitatively compare the prediction diversity of the networks in SSE, StanSSE, and DE baselines to confirm that the SSE and StarSSE improvement over Local DE is indeed the consequence higher network diversity. To evaluate the diversity, we use an average normalized prediction difference between model pairs test data, following Fort al. (2019) (the higher, the more diverse):
Emsages [pred, F pred,)
diversity = 100 Emz
max err1, err2)
where m, stands for model from an ensemble with predictions pred, and test error level err; We normalize the prediction difference by the maximum of two errors to pay less attention to the diversity of the models with lower accuracy. Table shows the accuracy of individual models and prediction
23

---

Ensembles
Model soups
Becelines
local
more local
Boselines
lncal
more local
optimal exp
Local
ogtimal
optimal exp
Local DE
mere sems local EXD
more semi local exp
more semi Jocal exp
more somi local exp
SSE
StarSSE
SSE
StarSSE
87
87
877
87
test accurbey. %
test acsuracy. %
86
86
86
86
ID
85
erssemble size
ensemmbee size
soup size
soup size
SSE
StarSSE
SSE
StarSSE
test sccuracy
test socuracy
OOD
2
1
2
4
2
1
2
3
1
ensemble
soup
soup size
Figure 16: Results of ensembles (left plots) and model soups (right plots) of different sizes on ID (CIFAR-100 test set, top row) and OOD (CIFAR- 100C, bottom row) for SSE and StarSSE with supervised pre-training. For OOD, we measure the average accuracy over all possible corruptions and severity values. Standard deviations are calculated over different pre-training checkpoints and/or fine-tuning random seeds.
Table 5: Accuracy of individual models and ensemble diversity for in-distribution (ID, CIFAR-100 test set) and out-of--istribution (OOD, CIFAR-100C) 100C) data for supervised (SV) and self-supervised (SSL) pre-training. The OOD results are averaged over different corruptions and values of severity. Standard deviations for both IDD and OOD are shown over different pre-training checkpoints and training seeds.
accuracy (single)
diversity
Pre-train.
Method
ID
OOD
ID
OOD
Local DE
60.21
58.72+2.00
84.47
57.46=0.81
Global DE
82.58
79.05=22.32
more local
84.24+0.23
56.30
64.01 #8.91
60.80++8.32
SSE
optimal
84.18
56.38 1.27
70.50 28.78
07.79-t0.00
SV
more semi-local
82.03 11.45
54.38 12.52
81.35 1:1.32
78.40 1:1.63
local
84.35-10.20
57.06=4068
55.30
53.62.12.29
StarSSE
optimal
84.55
56.77 1.003
73.28 14.42
69.07 1:4.23
more semi-local
82.95
55.31 +: 1.80
82.87
78.63.20.00
Local DE
6..25+3.56
62.75-t3.00
85.96++0.28
58.53+0.82
Global DE
80.21
75.93
more local
85.87
57.83
47.46
44.61 1:7.64
SSE
optimal
85.27+0048
57.56+ 1.26
71.57 5.994
66.98-t6.30
SSL
more semi-local
82.52
54.20
80.93 19.57
78.68 10.37
more local
85.95 0...2
58. 10.7
38.94 1.557
16.12.11.80
StarSSE
optimal
85.79.10.25
57.91 L0.97
71.51 12208
07.37.13.12
more semi-local
83.65
15.48.12.1
82.42 18.59
79.22

---

Table 6: Accuracy of individual models and ensembles of size 5 for Local DE, Global DE and optimal StarSSE, trained with and without hyperparameter and augmentation diversification of ensemble members. Self--uper pre-training.
Diversity
Local DE
Local DE
Global DE (ensemble)
StarSSE (single)
StarSSE (ensemble)
Dataset
hyp. aug.
(single)
(ensemble)
$5.9620.22
87 20.2
88.10=0.11
85.79
87.63#0.12
CIFAR-
+
85.95 10.27
87.45
88.20
85.91
87.82 00.04
100
+
85.85=201 00.28
87.60
88.25 20.05
85.69 20.27
87.39 20.06
+
+
85.24 L0.38
10.14
1..17
85.81 1..24
87.74.0.00
64.32
65.77:0.19
67.04
63.98
66.37:10.25
SUN-
+
64.25 10.41
65.84.0.22
66.99...00
6..17
66.69 00.15
397
+
10.52
66.17:20 0..228
67.33 20.05
63.93.0040
66.30
+ +
10.46
66.51 10035
67.49 20.10
64.30
66.59 10006
diversity in-distribution CIFAR-100 (ID) and out-of-distribution CIFAR-100C (OOD) data for both types of pre-training.
Despite Local and Global DEs consisting of the models of the same quality level, the latter includes significantly more diverse networks, explaining the quality gap between the two. Generally, when we increase the hyperparameters of SSE/StarSSF, the networks become more diverse, but their ID) accuracy decreases. The models of the optimal SSE/StarSSE experiments are much more diverse than the ones of the more local experiments, which makes up for lower ID accuracy when constructing ensembles. Morcover, the high diversity of models optimal SSE/StarSSE exceeds the one of the Local DE models, allowing optimal StarSSE (and sometimes optimal SSE) to outperform the Local DE on clean data. The models of the more semi-local experiments achieve even higher diversity, though it is not enough to overcome their significant quality degradation.
Our robustness analysis in Section 7 (Figure 4 and Appendix [ (Figure 16) shows that optimal SSE and StarSSE are less preferable under the distribution shifts than Local DE. The results in Table 5 confirm that the main reason for this is not the lower diversity but the lower quality of individual models. Generally, SSE and StarSSE models tend to overfit to target data - this is the main reason why semi-local methods do not show positive results in the transfer learning setup. Choosing the optimal hyperparameters for SSE/StarSSE results in low/almost no degradation of individual model quality on ID data (see Figures 1 and Table 5). However, overfitting becomes noticeable faster on OOD data, hence, individual SSE/StarSSE model quality degrades more significantly in this case.
K Diversifying ensembles
One of the most common techniques to improve ensemble quality is to diversify networks by training them in different ways (varying optimization algorithm, learning rate, weight decay, data augmentation, etc.). This technique is shown to be effective for both Local (Mustafa et al. 2020 Wortsman et al., 2022] Rame et al. 2022) and Global DE (Gontijo- et al., 2022). In this section, show that such diversification can also improve StarSSE and thus does not change the results of methods comparison.
We consider ResNet-50 with self- pre-training on CIFAR-100 and SUN-397 datasets and try two types of model diversification:
hyperparameter diversification instead of training ensemble members with a fixed learning rate, weight decay, and number of epochs, we sample these hyperparameters from the log-uniform distributions on segments around their optimal values:
CCIAR-1000 learning rate [0.035, 0.07). number of epochs |20, 50] for Local DE;
learning rate - (0.08, 0.12]. number of epochs - [5 10 1 2 10 10-1 for both.
-(1, [18,35) for StarSSE; weight decay- -
25

---

Table 7: Accuracy of individual models and ensembles of size 5 for three differently behaving SSEs without regularization, with EWC regularization to the pre-trained model (EWC-PT) and with L.2 regularization to the first fine-tuned model (L2-FT) on CIFAR-100 for self-supervised pre-training. StarSSE results are given for comparison.
more local
optimal
more semi-local
(single)
(ensemble)
(single)
(ensemble)
(single)
(ensemble)
SSE
85.87
86.57
85.27
87.11
82.52
85.60.10.10
+ EWC-PT
85.83
86.55:t0.05
85.57=0..
87.22 20.05
82.73:11.75
85.69:0.04
+ L2-FT
85.94
10.62.0.04
85.79 -0.22
87.20 10.05
85.65.10.2
87.26.10.0
StarSSE
85.95
86.41 +0.30
85.79=0..
87.63
83.65
86.20
SUN-397: learning rate - (0.1,..55] number of epochs - [15, 30] for Local DE; learning rate - 10011, 0.25), number of epochs - [15,23] 23] for StarSSE; weight decay - [5-10 15-10-6, 10 for both.
augmentation diversification - complementaarr to fixed data augmentation (random crop and random flip), use four color augmentations (adjusting brightness, contrast, hue, and saturation) similar to Mustafa et al. (2020), Appendix A.3 (though we needed to change the brightness parameters because of different parametrization used by torchvision (Paszke et al.] 2019)). We sample minimal and maximal limits for the strength of these augmentations before each model training:
brightness: Smin Uniform(O.22 0.75), 8max
Uniform(2 Smin 4 8min)
hue: 8max Uniform(0.01, 0.2), 6min -mmax
saturation: Smin Uniform (0.25, 0.75), Smax Uniform(2 Smin: 4 5min)
contrast: Smin ~ Uniform 0.75), Smax Uniform(2 Smin 4 Smin)
During training, each augmentation is applied with 50% probability, and its strength is sampled within the corresponding limits.
For diversified StarSSE, we train the first model with fixed optimal hyperparameters and augmenta- tions and then train all the following models utilizing the diversification techniques.
In Table we show how model diversification influences the average accuracy of individual models and ensembles of size for Local DE, Global DE, and StarSSE. All considered ensemble types benefit from the diversification: while Global and Local DEs improve more from hyperparameter diversification, StarSSE is stronger in combination with augmentation diversification. The latter shows that proper augmentations are very important for StarSSE since longer fine-tuning of StarSSE networks makes them more prone to overfitting. In conclusion, network training diversification can improve the results of all ensemble types and thus can not close the quality gaps between them.
L Regularizing SSE to stay in the pre-trained basin
In this section, we experiment with different regularization techniques to prevent SSE from overfitting and losing the advantages of transfer learning. As first technique, we consider Elastic Weight Consolidation (EWC, Kirkpatrick et al. (2017). popular continual learning method, which employs weighted 1.2 regularization to the pre-trained checkpoint. The weight of each parameter is a diagonal clement of the Fisher information matrix (FIM) evaluated on the pre- pre-training task. Hence, EWC not only regularizes the weights to not to go too far from the pre-train checkpoint, but also takes into account the pre-train loss landscape to control the direction of weights' changes. This makes EWC a very similar technique the informative prior of Shwartz-Ziv et al. (2022). which also utilises the pre-trai tre-train loss landscape to construct the Gaussian prior on network weights. We consider three values of the regularization coefficient λewo E [0.1, 1, 10), choosing the one performing best the test set > Agwo 10.
As second technique, we consider regularization of SSE to the first fine-tuned model to force local behavior. In this case, we use isotropic 1.2 regularization. The reasons for such a simplification are two-fold: 1) moved away from the pre-trained checkpoint, making it impossible to evaluate FIM
26

---

Ensembles
Model soups
Buselios
local sap
Buselines
local exp
Global DE
optimel wxp
optimal eig
Local DE
optimal wep
optimal exp
Local DE
more soms local tsp
more semi local exp
soup
more semi local exp
more semi local. exp
SSE
StarSSE
SSE
StarSSE
test accuracy %
test tast accuracy. %
90
90
90
90
CIFAR -100
8.9
89
89
89
8.8
8B
88
88
1
2
3
1
2
3
5
1
2
3
5
1
2
3
5
ensumble sizee
ensemble sized
soup
soup
SSE
StarSSE
SSE
StarSSE
70
70
70
70
test accuracy, %
test accuracy, %
SUN-397
66
66
66
66
64
64
64
64
2
3
5
2
3
3
onsomble size
ensemble size
size
size
Figure 17: Results of ensembles (left plots) and model soups (right plots) of Swin-T models with supervised pre-training. Three differently behaving SSE and StarSSE experiments CIFAR-100 and SUN-397 are shown with Local and Global DEs for comparison. Hyperparameters for more local and more semi-local experiments are the same for SSE and StarSSE the same dataset, while hyperparameters for the optimal experiments may differ.
on the pre-train task without refitting the classifier head, and there is need to evaluate FIM on the target task, as we already have access to the target loss landscape during the optimization. We consider three values of the regularization coefficient λς2 E [10-5 10-4, 10-3), choosing the one performing best on the test set Xi = 10-a, In both cases (EWC to the pre-trained checkpoint and L2 to the first fine-tuned model), the linear head is not regularized.
In Table [ we consider three differently behaving SSEs and add regularization to each of them, preserving the original hyperparameter values (i. e., the maximum learning rate and the number of epochs for each cycle). Overall, the EWC regularization has little impact on the accuracy of individual models and the ensemble. In contrast, L.2 regularization increases both metrics with the most pronounced improvements for the more semi-local experiment, which becomes similar to the optimal one. Generally, both regularization techniques are able improve the SSE results by promoting a more local behavior and preventing SSE from leaving the pre-train basin, however they can not help explore the vicinity of the pre-train basin more effectively. Moreover, regularized variants of SSE underperform the optimal StarSSE, making our parallel method the most effective for the local exploration of the pre-train basin.
M Additional experiments with Swin-T
In this section, we experiment with Swin-T models pre-trained in supervised manner. We consider only CIFAR-100 and SUN-397 target tasks because the gap between Local DE and Global DE on CIFAR-10 10 is too small. For more experimental details see Appendix B Figure 17 shows the results of ensembles and soups of models from three differently behaving SSE and StarSSE experiments. Generally, we observe the same types of behavior as for ResNet-50. Optimal ensemble variants of SSE and StarSSE show similar results Local DE, while soups of models from optimal SSE and StarSSE are much more accurate than the ones from Local DE models.
27

---

Ensembles
Model soups
Baselines
local
Baselines
optimal
optimel
opttmal eop
ogtimal eep
++
more somi cup
somi oup
more senul local exp
10.00 oop
StarSSE
StarSSE
79
79
79
Self super.
test acouracs,
-8
best acnuracy
78
78
76
Clipart
2
5
2
3
easemble size
e02 emble size
soup size
soup size
SSE
StarSSE
SSE
StarSSE
78
b55t sccuraie 38
78
Supervised
test: accuracyy
-7
77
77
75
ensemble size
005 emble size
soup 5ize
soup $120
SSE
StarSSE
SSE
StarSSE
leet eccursoy, %
tast accuracy, %
Self-super.
55
54
54
50
53
53
52
52
52
Chestx
ensemble size
BBS
soup size
soup
SSB
StarSSE
SSE
StarSSE
aceurary
test accuragg
Supervised
52.5
50.0
47.5
45.0
ensembble sizee
0:030 mibe Si.e
sougl size
80up size
Figure 18: Results of ensembles (left plots) and model soups (right plots) of different sizes on non-natural image classification datasets Clipart and ChestX for SSE and StarSSE with supervised and self-supervised pre- tre-trainnng. Hyperparameters for more local and more semi-local experiments the same for SSE and StarSSE on the same dataset, while hyperparameters for the optimal experiments may differ.
N Additional experiments on non-natural image datasets
In this section, experiment with two non-natural datasets: ChestX dataset (Wang et al.) 2017)
containing medical images and a clipart image collection DomainNet Clipart (Peng et al. 2019).
We consider the ResNet-50 network with supervised and self-supervised pre-training on ImageNet. Following Fricsson et al. (202T), we drop all multi-label images from ChestX and solve a usual classification problem. For more experimental details, see Appendix B
Figure shows the results of ensembles and soups of models from three differently behaving SSE and StarSSE experiments. We also provide the quality and diversity of ensemble members in Table for a deeper analysis of StarSSE results. Generally, we observe more extreme results, most likely due to higher discrepancy between source and target tasks.
Clipart, StarSSE and SSE either show very close or even inferior results compared to the Local DE. Analyzing the quality and diversity of ensemble members, we conclude that StarSSE does not find a balance between the two for this task. In the supervised case, StarSSE finds stronger individual models with lower diversity than the ones of a Local DE. Hence, the ensemble quality of StarSSE
28

---

Table 8: Accuracy of individual models and ensemble diversity for experimetns on non-natural image classification datasets with supervised (SV) and self-supervised (SSL) pre-training.
Clipart
Chestx
Pre-train.
Method
acc. (single) diversity
acc. (single)
diversity
Local DE
66.74+1.08
42.42=1.30
74.58
51.86:+0.2
SV
Global DE
75,56=1.70
43.81=1.45
StarSSE (opt.)
75.21
64.28 1.66
53.93+0.47
46.06=4.41
Local DE
56.61 1.88
38.62 1.66
+5.28
54.32 :0.49
SSL
Global DE
68.46 1.46
40.05 1.76
StarSSE (opt.)
14.18
67.24 11.2
54.42
46.03 22.70
is lower, but the time, the soup of StarSSE models improves the Local DE one. In the self-supervised case, models from StarSSE are more diverse but are of lower quality than the Local DE ones. As result, StarSSE and Local DE show very similar results in this case.
the contrary, the results on ChestX are very positive: SSE and StarSSE outperform both Local and Global DEs. Interestingly, Local and Global DEs work very similarly this task. In both supervised and self-supervised cases, StarSSE finds a much more diverse set of models a more convex part of the loss landscape, hence, resulting in better ensembles and models soups. Additionally, in the supervised case, StarSSE fine-tunes individual models more effectively than Local and Global DEs, improving the results even further.
29


Reviews:
[{"review_id": "HCvwH3uUvH", "reviewer": "Reviewer_XbMH", "summary": "The authors consider techniques to faciliatate more efficient ensembling when in the transfer learning context.They propose a novel cyclical learning rate method, StarSSE, and compare it to standard methods like Snapshot Ensembles (SSE). They demonstrate that leaving the pretraining basin leads to worse performance when forming ensembles from a pretrained checkpoint, and that cyclical learning rate methods can help to close the gap between purely local and global ensembling methods. Finally, they discuss their methods when forming model soups, and in the context of robusness. ", "questions": "- The original conception of Snapshot Ensembles relies upon the model converging to and escaping from local minima. Although the algorithm has been repurposed here to better explore the vicinity of a single local minimum, I am curious how the training loss fluctuates across multiple cycles of SSE as a result. \n- Throughout the main text, the diversity of ensembles created through StarSSE is discussed as a benefit. However, since StarSSE models rely only upon the randomness of SGD based training to generate diversity I would not expect them to be necessarily more diverse than SSE, or even Local DE (as would be supported by OOD results). Can you provide some metric of the predictive diversity in Local DE, SSE, and StarSSE ensembles? ", "limitations": "As the authors acknowledge, the choice of hyperparameter tuning is a potential limitation of the applicability of their study. ", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The authors provide a thoughtful experimental analysis of different fine-tuning setups (different models, datasets, pre-training, learning schedules, hyperparameters, fine tuning paradigms) that address an important question: the efficacy of ensembling methods when fine tuning. Their insights into the correspondence between good ensemble members and staying within the pre-training basin are interesting. \n- The authors provide a descriptive overview of a wide variety of ensembling methods, and provide categorizations into groups based on their coverage of the loss landscape, as are relevant to their work. ", "weaknesses": "- My biggest concern is that at the moment, it is difficult to quantitatively evaluate the claim that StarSSE ensembles perform better than SSE ensembles. Do StarSSE ensembles provide a significant improvement over SSE ensembles in the different ensembling conditions that you study? It is difficult for me to determine this from the plots provided in Figure 2, 4 and Appx. D. Could you provide a table of results, perhaps only for the optimal hyperparameter settings at a fixed model number? \n- The results on model soups and robustness appear slightly disjoint from the rest of the paper. A more thorough description of related work around robustness and models soups would be useful. It is also unclear why SSE and StarSSE underperform Local Deep Ensembles on OOD data- as robustness is a major reason for practitioners to use deep ensembles, at least understanding why this loss in performance occurs is important. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful review of our work! Please allow us to address your concerns and answer the questions.\n\n**W1. Unclear quantitative comparison of SSE and StarSSE ensembles.** \n\nThank you for pointing out that flaw in our presentation. In all the experiments StarSSE ensembles showed significant improvement over SSE. Below we provide a table of results for ensembles of 5 networks trained with optimal hyperparameter settings (see Table 3 in the Supplementary for the specific values). The results for Local and Global DEs of the same size are given in Table 1 in the paper. We will include results in numerical form in addition to figures in the final revision of the paper.\n||SV SSE|SV StarSSE|SSL SSE|SSL StarSSE|\n|---|---|---|---|---|\n|CIFAR-10|97.30 ± 0.05|97.39 ± 0.01|97.94 ± 0.04|98.06 ± 0.02|\n|CIFAR-100|86.04 ± 0.18|86.30 ± 0.18|87.11 ± 0.06|87.63 ± 0.12|\n|SUN-397|63.63 ± 0.14|64.2 ± 0.21|65.79 ± 0.10|66.37 ± 0.25|\n\n**W2 (part 1). Relevance of model soups and robustness sections.**\n\nWe briefly described our reasons for including these parts at the beginning of the corresponding sections. Since model soups are a more practical alternative to local ensembles and StarSSE constructs a high-quality local ensemble, we were interested to see how StarSSE models behave when averaged. And we obtained unexpected results: even though StarSSE models are more diverse, they lay in a much more convex region of the loss landscape. The goal of the robustness analysis was to compare StarSSE and Local DE in more challenging and practical conditions. The results showed an interesting limitation of StarSSE: it is less robust to the test data corruption as an ensemble method, however, averaging StarSSE models is still an effective technique even in this setup. We will try to explain our motivation for these sections more thoroughly in the final revision of the paper and discuss more related works following your suggestion. \n\n**Q1. Training dynamics of local SSE.**\n\nSSE in our experiments shows very similar behavior to the standard setup: train and test loss increase at the beginning of each cycle and then converge at the end of a cycle. Depending on the hyperparameters, the magnitude of the loss increase changes (the higher the learning rate, the higher the loss increase at the beginning of a cycle) as well as the locality of the method (the higher the learning rate and number of epochs, the further the next checkpoint ends up). Varying SSE hyperparameters similarly influence the locality of the method in standard and transfer learning setups. The only difference is that in standard setup training can still converge to a good solution even after leaving the previous local minima, while in a transfer learning setup leaving the pre-train basin results in overfitting and losing the transfer benefits. Hence, when we train SSE with high hyperparameter values in a transfer learning setup, we obtain low train loss at the end of all cycles, but the test loss degrades with each cycle. \n\nWe would also like to mention that FGE can be viewed as a local variant of SSE with a different learning rate schedule. Hence, cyclical methods were also used to explore the vicinity of a single local minimum in the standard training setup in the literature. The authors of FGE see the cyclical behavior of the test accuracy but with low magnitude. \n\n**Q2 and W2 (part 2). Diversity of StarSSE models and OOD results.**\n\nThank you for this interesting question. We conducted some additional analysis of our results, and we will definitely include this analysis in the final revision of the paper.\nFirstly, we computed the predictive diversity metric on both ID and OOD data. We chose an average normalized prediction difference between model pairs based on [1] (the higher, the more diverse): \n$$\n100 \\times \\mathbb{E}_{m_1 \\ne m_2} \\frac{\\mathbb{E} [pred_1 \\ne pred_2]}{\\max (err_1, err_2)},\n$$\n\nwhere $m_i$ stands for a model from an ensemble with predictions $pred_i$ and test error level $err_i$, the inner expectation is taken over test images.\n\nThe results for CIFAR-100 with self-supervised pre-training are shown in the table below. Optimal variants of SSE and StarSSE are indeed more diverse than Local DE on both ID and OOD data. Even though the only source of diversity in SSE and StarSSE is the randomness of SGD-based training, these methods benefit more from this source than Local DE. Due to the separation of fine-tuning and exploration, SSE and StarSSE can effectively use higher hyperparameter values in their cycles, which improves diversity.\n\n|||Local DE|Global DE|SSE (opt)|StarSSE (opt)|\n|---|---|---|---|---|---|\n|Diversity|ID|66.08 ± 3.42|80.03 ± 1.55|71.19 ± 5.71|71.46 ± 2.01|\n||OOD|62.68 ± 3.41|75.89 ± 2.37|66.91 ± 6.33|67.39 ± 3.09|\n|Test acc (ind model)|ID|85.96 ± 0.28|85.96 ± 0.28|85.27 ± 0.49|85.79 ± 0.28|\n||OOD|58.53 ± 0.82|58.53 ± 0.82|57.56 ± 1.26|57.91 ± 0.97|\n\nSo, why are the results of StarSSE and SSE less preferable to the Local DE on OOD data? The main reason is not the lower diversity but the lower quality of individual models. Generally, we see in our experiments that models tend to overfit the target data — this is the main reason why semi-local methods do not show positive results in the transfer learning setup. Choosing the optimal hyperparameters for SSE/StarSSE resulted in low/almost no degradation of individual model quality on ID data (see Figure 3 and the table above). However, overfitting becomes noticeable faster on OOD data, hence, individual SSE/StarSSE model quality degrades more significantly in this case. Nevertheless, the model soups of SSE/StarSSE remain effective even on OOD data (see Figure 4). \n\n[1] Fort et. al. Deep ensembles: A loss landscape perspective. 2019. \n\n\\\nWe hope we adequately addressed all your concerns and that you would consider reflecting that in your score. If you have any additional questions, we would be happy to answer them during the discussion period =)\n"}, {"review_id": "tzxgfBc90Z", "reviewer": "Reviewer_a9o6", "summary": "This paper focuses on ensembling in transfer learning scenarios, i.e., constructing ensembles starting from only one pre-trained checkpoint. Empirical results employing Snapshot Ensembles (SSE; Huang et al., 2017) demonstrate that leaving a pre-trained basin can degrade the quality of the ensemble. Based on this, the authors present a variant of SSE, StarSSE, which forces fine-tuned ensemble members to stay in the pre-trained basin.", "questions": "1. Does the diversity observed among the ensemble members in StarSSE originate solely from distinct SGD noise sources, such as random mini-batch order and data augmentation techniques?\n2. There are alternative ways to regularize fine-tuned solutions to stay in the pre-trained basin, e.g., Xuhong et al. (2018) and Shwartz-Ziv et al. (2022). It would be nice to compare the initialization with the starting point (StarSSE) and the regularization towards the starting point (e.g., L2 regularization, like Xuhong et al., 2018).\n3. Having visual representations of two-dimensional loss surfaces would be advantageous in providing supplementary evidence for the empirical analysis. According to the paper, it is expected that configurations described as \"more semi-local\" would deviate from the basin, while \"optimal\" and \"more local\" setups would stay within the basin.", "limitations": "Appendix A addresses the limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The empirical analysis conducted using Local DE, Global DE, and SSE is convincing and supports the main argument that staying in the pre-trained basin is beneficial for constructing ensembles in transfer learning scenarios. While earlier studies (Neyshabur et al., 2020; Wortsman et al., 2022) have already demonstrated the tendency of fine-tuned solutions to stay within the same pre-trained basin, this study goes a step further to argue that diverging from the pre-trained basin to enhance ensemble diversity does not result in improvements in ensemble performance. In summary, the paper provides insights into the current trend of favoring the fine-tuning of large-scale models over training them from scratch.", "weaknesses": "The proposed StarSSE is equivalent to the Local DE using a fine-tuned checkpoint instead of a pre-trained checkpoint as initialization. Although the experimental analysis was conducted using the SSE, I am not sure whether we can call the proposed algorithm as a \"snapshot.\" In contrast to the original snapshot approach, which employs a single continuous SGD trajectory with cyclical learning rates to explore loss surfaces, the proposed StarSSE method collects samples in the vicinity of the first fine-tuned solution through multiple short SGD runs.\n\nMoreover, concerning this aspect, it is strongly recommended to verify the comparative performance of the proposed StarSSE method against other \"local\" techniques which collect samples near the fine-tuned solution. Specifically, the following methods can be considered within the transfer learning context:\n* Constructing a Laplace approximation at the same fine-tuned solution and ensembling the obtained samples.\n* Constructing a SWA-Gaussian approximation centered around the fine-tuned solution (which would be the SWA solution in this case) and ensembling the collected samples.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful review of our work! Please allow us to address your concerns and answer the questions.\n\n**W1. StarSSE name and connection to Local DE.**\n\nYour assessment of the conceptual closeness of StarSSE and Local DE is absolutely correct, and we discuss it in the paper (lines 295-304). We had the same doubts as you about the name but agreed on StarSSE because it was originally motivated as a modification of SSE in our project. Generally, we are open to changing the name to, e.g., Star Ensemble or Star DE, especially if other reviewers also find the name StarSSE a bit misleading. We would also be happy to hear your suggestions =)\n\n**W2. Comparison of StarSSE and other local ensemble techniques.**\n\nWe initially focused on the cyclical methods (SSE, FGE, cSGLD) because they are the only semi-local methods and, at the same time, they show strong performance as local methods in the standard setup (see results of FGE in [1]). However, we agree that additional comparisons with other local methods would benefit the StarSSE part of the paper. Below we provide comparisons with KFAC-Laplace [2] and SPRO [3], both of which sample models from a local area near the usually trained network. Since SWAG is based on the SWA solution, its benefits are not only due to ensembling but also due to obtaining a better base model. Hence, for a fair comparison with SWAG, our method and baselines should be modified by incorporating SWA at the end of the training of each model. We were not able to conduct the experiments with SWA due to the time limitations of the rebuttal period, however, we will include them in the final revision of the paper. \n\nThe comparison of different 5-network ensembles on CIFAR-100 with SSL pre-training is shown in the table below (we also measure the diversity of networks in ensembles, see the answer to question 2 of Reviewer XbMH for the definition of the used diversity metric). We used grid search to choose an optimal sampling scale (1.0) for KFAC-Laplace and the optimal values of the learning rate (0.05) and training epochs (20) for each new vertex, as well as the regularization coefficient (1e-5) for SPRO.\t\n||Local DE|StarSSE (opt)|KFAC-Laplace|SPRO|\n|---|---|---|---|---|\n|Test acc|87.33±0.21|87.63 ± 0.12|86.01 ± 0.02|86.46 ± 0.12|\n|Diversity|66.08 ± 3.42|71.46 ± 2.01|35.63 ± 2.24|42.43 ± 18.27|\n\nKFAC-Laplace and SPRO results are inferior to StarSSE and, moreover, to Local DE, meaning these methods are extremely local. The results are consistent with the results for the standard setup from [1]. \n\n[1] Ashukha et. al. Pitfalls of in-domain uncertainty estimation and ensembling in deep learning. ICLR 2020.\\\n[2] Ritter et. al. A Scalable Laplace Approximation for Neural Networks. ICLR 2018.\\\n[3] Benton et. al. Loss surface simplexes for mode connecting volumes and fast ensembling. ICML 2021.\n\n**Q1. The origin of diversity among the ensemble members in StarSSE.**\n\nYes, the only origin of diversity is random mini-batch order and data augmentation. However, due to the separation between fine-tuning and exploration, we can utilize higher hyperparameter values (learning rate and number of epochs) in StarSSE, which results in higher diversity. We also provide the additional diversity analysis answering question 2 of Reviewer XbMH, which you may find interesting in connection to this question.\n\n**Q2. Other ways to regularize SSE solutions to stay in the pre-trained basin.**\n\nInspired by [1], we tried Elastic Weight Consolidation (EWC) towards the pre-trained solution (L2 regularization, reweighted using the approximation of the Fisher matrix on the source task) after the submission. Our motivation was to allow SSE to account for the source loss landscape shape when exploring the target loss landscape. We obtained a slight improvement of the optimal results: for ensembles of size 5 on CIFAR-100 with SSL pre-training, we get 87.22 ± 0.05 for SSE + EWC v.s. 87.11 ± 0.08 for SSE (for comparison, for StarSSE we get 87.63 ± 0.12). However, the optimal experiment still behaved as a local method, and EWC did not help SSE explore different target loss basins without losing the transfer benefits, most likely due to its Gaussian prior shape.\n\nFollowing your suggestion, we also tried L2 regularization towards the first fine-tuned network. It improves the optimal results too: 87.26 ± 0.05 in the same setup as above, which is still inferior to the StarSSE results. Hence, initialization with the starting point is more useful than the regularization towards the starting point for making SSE applicable in the transfer learning setup.\n\nWe will conduct a more thorough comparison of the discussed regularization techniques and include it in the final revision of the paper. \n\n[1] Shwartz-Ziv et. al. Pre-train your loss: Easy Bayesian transfer learning with informative priors. NeurIPS 2022. \n\n**Q3. Two-dimensional loss surface visualizations.**\n\nIn our connectivity analysis in Figures 2, 9, and 11, we compared three types of SSE/StarSSE behaviors, therefore, we decided to look only at one-dimensional visualizations since we can not look at four points (the first fine-tuned model and three final models) on one two-dimensional plain. However, we agree that looking at two-dimensional visualizations could be interesting. For example, a plot for the optimal and semi-local experiment pair may show how SSE with high hyperparameter values “go round the corner” in the train loss landscape and, at the same time, leave the high-quality zone in the test loss landscape. Would you like to suggest any other two-dimensional plains which we should check? \n\n\\\nWe hope we adequately addressed most of your concerns and that you would consider reflecting that in your score. We will also add more detailed experiments on local methods and regularizations in the final revision of the paper. We would be happy to hear your suggestions regarding W1 and Q3 and answer any additional questions during the discussion period =)"}, {"review_id": "cTWt7CQDu1", "reviewer": "Reviewer_rBpj", "summary": "This paper studies ensembling in the transfer learning setting. This is interesting because pre-training is important for strong predictive performance, yet has been shown to pre-determine the final basin in the loss-landscape after fine-tuning, reducing diversity. The authors explore the behavior of snapshot ensembles(SSE), which use different training iterates as ensemble members, and compare them to Global DE (fine-tune from independent pre-training runs) and Local DE (fine-tune from shared initialization), for both supervised and self-supervised pre-training. The authors find that Global DE is much superior to Local DE, but that Snapshot examples can start to close the gap. They also do some ablations and find that higher learning rates in SSE's increase diversity but decrease overall performance. The authors then propose StarSSE, where a single fine-tuning run seeds the initialization for multiple, parallel fine-tunings. The results of these parallel fine-tuning runs become ensemble members. StarSSE closes the gap between Local DE and Global DE further, and is more amenable to model souping than local DE.. \n", "questions": "Please see Weaknesses. \n\nQ1(Weakness 1): How does StarSSE perform on additional downstream datasets, especially non-natural ones?\n\nQ2 (Weakness 2): How does StarSSE compared to \"diversifed Local DE's\" (line 144)? \n\nWith adequate addressing of these questions, I'd be willing to bump my score. ", "limitations": "The authors have appropriate addressed limitations. ", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Originality:  The authors find that Snapshot Ensembles degrade in performance over subsequent snapshots, a key observation that inspires their \"Star SSE\". StarSSE is not a particularly large departure from SSE's, but it does seem to address a key limitation, \n\nQuality: The methodology and experimental setup seem sound, so given additional verification (see weaknesses) I would trust the result generalizes. \n\nClarity :The paper is clear and easy to follow. \n\nSignificance: Ensembles and pre-training are both crucial techniques in ML, any attempts to merge the strengths of both in easy-to-implement ways has strong impact. \n\n", "weaknesses": "Weakness 1: Though results for StarSSE's look promising, only a few datasets are considered. Because different methodologies can be overfit (even unintentionally) to specific datasets, it would be much better to demonstrate efficacy on a larger suite of downstream tasks, as is common (e.g. [1]). \n\nWeakness 2: The baseline alluded to in Line 144 is an important one; \"Several recent works (Mustafa et al., 2020; Wortsman et al.,\n143 2022; Rame et al., 2022) study another technique for improving the diversity of Local DEs by training\n144 networks using different hyperparameters (learning rate, weight decay, data augmentation, etc.).\" \n\nThe argument that it cannot close the gap between Local and Global DE is not convincing; the given constraints are that you have target data and a pretrained model, and the goal is the best possible performance on target test data. The \"diversivied local DE\" is a logical choice for this setup, and it's important to charactherize its performance compared to StarSSE. \n\n[1] How Well Do Self-Supervised Models Transfer? - Ericsson et. al., CVPR 2021. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful review of our work! Please allow us to address your concerns and answer the questions.\n\n**1. Experiments on additional downstream tasks.**\n\nWe started our project using the same standard benchmark suite (Aircraft, Caltech101, Cars, CIFARs, etc.) used in the work you mentioned [1]. However, due to the limited computational resources, we decided to focus on CIFARs and SUN397 since the gap between Local and Global DEs was the most prominent for these datasets. \n\nWe fully agree that experiments on more non-natural datasets, which you suggested, would be an interesting addition to the paper. In the table below we provide the results of such experiments on two datasets: ChestX dataset [2] containing medical images and a clipart image collection DomainNet Clipart [3]. Following [1], we drop all multi-label images from ChestX and solve a usual classification problem. We compare the accuracy of ensembles and model soups of 5 networks with SSL pre-training. \n|||Local DE|Global DE|StarSSE (opt)|\n|---|---|---|---|---|\n|Clipart|Ensemble|77.73 ± 0.23|78.57 ± 0.17|77.69 ± 0.09|\n||Soup|77.76 ± 0.22|-|77.79 ± 0.11|\n|ChestX|Ensemble|55.33 ± 0.22|55.46 ± 0.19|56.15 ± 0.35|\n||Soup|53.80 ± 0.27|-| 56.42 ± 0.12|\n\nOn Clipart, StarSSE and Local DE results are very close both for ensembles and model soups. This shows that in some cases Local DE models converge to the same convex basin (hence, the high soup quality), and in this case, StarSSE and Local DE explore almost the same area in the loss landscape. The situation with ChestX is quite surprising: StarSSE ensemble outperforms both Local and Global DEs. We analyzed the constituent models and found that the high performance of StarSSE on ChestX originate not from the stronger individual models (individual StarSSE models achieve 54.42 ± 0.50, which is very close to the individual Local/Global DE models with 54.32 ± 0.49) but from the higher diversity (46.00 ± 2.6 for StarSSE v.s. 38.13 ± 1.72 for Local DE and 40.32 ± 1.93 for Global DE). See the answer to question 2 of Reviewer XbMH for the definition of the used diversity metric. We will continue our investigation of the reasons for such positive behavior.\n\nWe will include a broader version of the results on non-natural datasets in the final revision of the paper. To further support the paper's conclusions, we will also add large-scale experiments following the suggestion of Reviewer 1VUP.\n\n[1] Ericsson et. al. How Well Do Self-Supervised Models Transfer? CVPR 2021.\\\n[2] Wang et. al. ChestXray8: Hospital-scale chest X-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. CVPR, 2017.\\\n[3] Peng et. al. Moment Matching for Multi-Source Domain Adaptation. ICCV 2019.\n\n**2. Improving network diversity by varying training hyperparameters.**\n\nVarying training hyperparameters of member networks is indeed a useful technique that should be used in practice to obtain the best possible ensembles. What we meant in the paper is that such diversification is known to be effective for both Local and Global DE and can not close the gap between them. However, we agree that experiments on diversified ensembles would be an interesting addition to the paper. We decided not only to compare \"diversified Local DE\" and StarSSE but also to check whether StarSSE can benefit from such diversification. Below we provide results for ensembles of size 5 on CIFAR-100 with SSL pre-training. We try two types of model diversification: \n* hyperparameter diversification — instead of training ensemble members with a fixed learning rate, weight decay, and number of epochs, we sample these hyperparameters from segments around the optimal values,\n* augmentation diversification — complementary to fixed data augmentation (random crop and random flip), we use four color augmentations (adjusting brightness, contrast, hue, and saturation) similar to [1], Appendix A.3. We sample minimal and maximal limits for the strength of these augmentations before each model training. During training, each augmentation is applied with 50% probability, and its strength is sampled within the corresponding limits. \n\nFor diversified StarSSE, we trained the first model with fixed optimal hyperparameters and augmentations as in the paper and then trained all the following models utilizing the diversification techniques.\n||Local DE|Global DE|StarSSE (opt)|\n|---|---|---|---|\n|No diversification|87.33 ± 0.21|88.10 ± 0.11|87.63 ± 0.12|\n|Hyperparameter diversification|87.60 ± 0.21|88.25 ± 0.05|87.39 ± 0.06|\n|Augmentation diversification|87.45 ± 0.15|88.20 ± 0.03|87.82 ± 0.04|\n\nThe results show that all the methods can benefit from the diversification of network training. While Global and Local DEs improve more from hyperparameter diversification, StarSSE is stronger in combination with augmentation diversification. We will include more thorough experiments with diversified ensembles in the final revision of the paper.\n\n[1] Mustafa et. al. Deep Ensembles for Low-Data Transfer Learning. 2020. \n\n\\\nWe hope we adequately addressed most of your concerns given the limited rebuttal period and that you would consider reflecting that in your score. We will also add more detailed experiments in the final revision of the paper. If you have any additional questions, we would be happy to answer them during the discussion period =)\n"}, {"review_id": "8G1NVqaOLa", "reviewer": "Reviewer_1VUP", "summary": "This paper studies an interesting dilemma -- literature reports that ensembles work better if you end up in different \"basins\",  but pre-training often confines you to a single \"basin\". The question is therefore, what should you do? This paper answers that you should start out with a pre-trained model and _stay_ in the basin, but do so using StarSSE -- fine-tune once then fine-tune more times \"branching\" out from that first fine-tuned model. The solutions can be ensembled and also souped (weight-averaged).", "questions": "- How do SSE and StarSSE compare when starting out with a large pre-trained network? For instance if the authors want to stick to image experiments than a pre-trained CLIP model is a good starting point. Here, you can already start out with a good \"zero-shot\" model.\n- How might the results change when using techniques to mitigate overfitting -- it seems like there is overfitting in some of the experiments.", "limitations": "I would have like to see a limitations Section which discusses that all experiments are quite small scale.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper tackles an interesting problem. I believe the community will find their results of interest. Some particular strenghts:\n- The proposed StarSSE method is well motivated and performs well on the datasets they examine. It seems like a practical modernization of SSE in the era of pre-trained networks.\n- I found the result particularly interesting: that if you leave the \"basin\" of the pre-trained model then you lose the benefits of pre-training.", "weaknesses": "There are a few weaknesses that the authors could address to improve the paper:\n- The experiments are all very small scale for a paper whose main topic is pre-training. It is understandable to not be able to compare Global DE on larger problems but surely the authors could start out with a pre-trained network when contrasting SSE and StarSSE.\n- It seems in Figure 2 / Figure 3 that the runs may be overfitting (much higher train acc then test acc) -- could this be mitigated with standard interventions like label smoothing / data aug / stochastic depth?\n- There are terms in the paper which are used frequently but not formalized including \"basin\". This is OK but then strong claims involving these terms should be avoided (e.g., leaving the basin hurts model performance -- what does this actually mean? that you can no longer interpolate?).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful review of our work! Please allow us to address your concerns and answer the questions.\n\n**1. Large-scale experiments.**\n\nAs you correctly pointed out, in the submission we focused on smaller problems to be able to accurately compare the results with both Local and Global DE baselines. Thank you for the reasonable suggestion to conduct larger experiments dropping the Global DE. We did not manage to make them during the limited rebuttal period, but we will add them in the final revision of the paper.\n\n**2. Overfitting and its mitigation.**\n\nWe chose fine-tuning procedures for our experiments based on the standard strategies for specific network architectures and datasets from the literature. For ResNet experiments, we used two types of regularization: weight decay and data augmentation (random crop and random flip), following, e.g., [1]. For Transformers, we additionally used more advanced augmentations (Color Jitter, Auto Augment, Random Erase, Mixup, and Cutmix) and label smoothing following [2]. The test accuracies achieved by our models are close to the results from the literature with similar training setups. A large accuracy gap between train and test metrics is very common for CIFAR datasets (see, e.g., [3]), even if some more advanced regularization techniques are used [4]. Hence, we argue that our experimental setup is adequate.\n\nHowever, we also think that a deeper investigation of fine-tuning with different data augmentations and regularizations is interesting, especially as a part of the study on the effectiveness of ensemble diversification by varying training hyperparameters for the constituent networks. You can find some preliminary results on this in the answer to question 2 of Reviewer rBpj. We will provide a more broad study in the final revision of the paper. \n\n[1] Grill et. al. Bootstrap your own latent - a new approach to self-supervised learning. NeurIPS 2020.\\\n[2] Liu et. al. Swin transformer: Hierarchical vision transformer using shifted windows. ICCV 2021.\\\n[3] Garipov et. al. Loss surfaces, mode connectivity, and fast ensembling of DNNs.NeurIPS 2018.\\\n[4] https://sebastianraschka.com/blog/2023/data-augmentation-pytorch.html\n\n**3. Formalization of the main terms.**\n\nWe give an intuitive definition of the term “basin” in lines 42-43: two models lay in the same basin if there is no high-loss barrier on the linear interpolation between them. The loss barrier is commonly defined as the highest difference between the loss occurring when linearly connecting two points and the linear interpolation of the loss values at each of them [1]. We indeed decided not to formalize our definition of the basin, mostly because we did not want to choose the specific threshold for the loss barrier (it can be exactly 0 or some epsilon>0). Instead, we focused on the loss behavior along the interpolation between the members of different SSE/StarSSE ensembles (Figures 2, 9, and 11) and its comparison to the interpolation of Local DE models. When we say that a model leaves the pre-train basin, we mean that there is a significant train loss barrier between this model and the first fine-tuned model. So, this loss barrier is not only higher than zero but also comparable to or higher than the loss barrier between Local DE models. Empirically we see that when the model leaves the pre-train basin, its test accuracy degrades, and the higher the train loss barrier between the two models, the more severe the test accuracy degradation. We see what you mean by the lack of formalization here and will reformulate this connectivity discussion more accurately in the final revision of the paper.\nCould you please specify which other terms seemed not properly defined to you so we would pay more attention to them in the revision?\n\n[1] Entezari et. al. The role of permutation invariance in linear mode connectivity of neural networks. ICLR 2022. \n\n\\\nWe hope we adequately addressed most of your concerns given the limited rebuttal period and that you would consider reflecting that in your score. We will also add more large-scale experiments and experiments with other regularizations in the final revision of the paper. If you have any additional questions, we would be happy to answer them during the discussion period =)"}]

Author rebuttal:
First of all, we want to thank all the reviewers for their constructive and valuable feedback! We have found all the suggestions very relevant and will definitely incorporate them into the paper.

We are happy that our analysis of the effectiveness of local and semi-local ensemble methods in the transfer learning setup and the resulting insights on how staying or leaving the pre-train basin influences the ensemble quality were found convincing, interesting, and important. We see this analysis and insights as the main contributions of our paper since they show which loss landscape exploration tactics are suitable for the transfer learning setup and can be useful in practice: while local ensemble methods can be beneficial, existing semi-local methods can not leave the pre-train basin without losing the transfer learning benefits.

Most of the reviewers' comments concerned the effectiveness of StarSSE, the proposed modification of the SSE method for transfer learning. We did not closely focus on this part in the submission because we looked at StarSSE more as an example of local ensemble methods that helps us to show that local methods, if used appropriately, can be beneficial in transfer learning too. However, we agree that some additional experiments and comparisons would improve this part of the paper. Given the tight rebuttal period, we do our best to address most of the concerns and provide additional results in the replies below. We will include more detailed additional experiments in the final revision of the paper. 


Meta-review: This paper studies improving an ensemble obtained from finetuning a single pretrained model. It is commonly believed an ensemble obtained from a single pretrained model is restricted to the same basin in the loss landscape and, thus, is not useful. Recent work on SnapShot Ensembling (SSE) suggests training an ensemble with a cyclic learning rate to help the original model escape the current basin. Although this semi-global approach can achieve better diversity compared to purely local methods, the authors show that, in the transfer learning setup, SSE could make the ensemble quality degrade. The ensemble degrades because ensemble members keep moving away from the pretrain basin, especially when a higher learning rate or a larger ensemble size is used. Thus, The paper proposes StarSSE, a parallel version of the original SSE method. It is shown that by converting the SSE from sequential to parallel, the new method effectively achieves better diversity without forcing the ensemble members to move too far from the pretrain basin.

The authors also provide a good rebuttal and improve the paper by providing a few extra experiments: (1) More clear quantitative comparison of SSE and StarSSE ensembles. (2) Showing StarSSE achieves better diversity than local deep ensemble methods. (3) Comparing StarSSE to other local ensembling methods regarding test accuracy. (4) StarSSE on more downstream tasks. (5) StarSSE and baselines with hyperparameter and augmentation diversification.
Decision: Accept (poster)