SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In the left pair of plots in Figure 2, we show how train and test accuracy behave along the line segment between the first and the last (5-th) network in the SSE. Metric values along the line segment between two random networks from the same Local DE are shown as baselines. From the loss landscape perspective, more local and more semi-local experiments indeed demonstrate the expected local and semi-local behavior, respectively. Interestingly, the behavior of optimal experiments is also local, without low-accuracy barriers. These results confirm that SSE can behave as both local and semi-local methods and that its local variant provides the best results. In Appendix G, we provide additional locality analysis based on the accuracy behavior on 2D planes in the weight space.
In the left pair of plots in Figure 3, we show how train and test accuracy of individual models in SSE changes with each cycle. In all the experiments, model test accuracy degrades with each cycle, even though the train accuracy increases or stays at the maximum value. From this, we can conclude that the further individual models walk away from the pre-train checkpoint, the more they overfit the training data of the target task and, as a result, lose the advantages of transfer learning.
SSE with optimal hyperparameters can improve over Local DE by better exploring the pre-train basin and collecting more diverse networks (see Appendix J for the detailed diversity analysis). The diversity of these networks even makes up for the gradual decrease in individual model quality. However, leaving the pre-train basin with SSE results in the most pronounced degradation of individual model test accuracy, which leads to decreasing ensemble quality. We also hypothesize that due to the decrease in the quality of individual models, SSE with any values of parameters is not able to effectively construct large ensembles in the transfer learning setup (any SSE plot will start to decrease if we grow the ensemble size further).
Whether staying in the pre-train basin is an essential condition to take advantage of knowledge transfer and obtain high-quality ensembles or the considered methods explore the vicinity of the pre-train basin in a non-effective way is an interesting question for future research. We suspect that the main reason why SSE can not leave the pre-train basin without losing the transfer learning advantage is the fact that SSE only looks at the target task loss landscape without taking into consideration the source task one. Shwartz-Ziv et al. (2022) show that accounting for the source loss landscape shape around the pre-trained checkpoint can improve fine-tuning accuracy of individual models and local ensembles. However, to do so, they fit a normal prior on the source task, centered in the pre-trained checkpoint, which most likely can be effective only locally in the pre-train basin (see Appendix L for additional experiments on SSE with similar regularizations). Hence, more advanced techniques should be developed to account for the source loss landscape shape outside of the pre-train basin.
5.4 StarSSE — a better version of Snapshot Ensemble for transfer learning
Our study of SSE behavior shows that, in the transfer learning setup, standard cyclical learning rate methods suffer from individual network quality degradation due to moving too far from the pre-train checkpoint and overfitting to the target training data, and this effect becomes more pronounced with each cycle. SSEs with long cycles and high learning rates are affected the most by this, but at the same time, they show high quality for small ensemble sizes (see results for ensembles of size 2 at the top row of Figure 1). So, even though the second network found by SSE is weaker as an individual model, it may be a better candidate for the ensemble with the first one than another independently fine-tuned network, as in Local DE.
Based on these observations, we propose a parallel version of SSE, StarSSE, which is more suitable for the transfer learning setup. StarSSE trains the first model with the standard fine-tuning schedule, as in Local DE or SSE. Then, all the following models are trained with the same schedules as in SSE cycles but in a parallel rather than sequential manner. Each of these models is initialized with the weights of the first fine-tuned model. As a result, we have a "star" with the first model in the center and all other models at the ends of the "rays". Due to the parallel structure, StarSSE does not move further from the pre-trained checkpoint with each training cycle as SSE, but at the same time, it explores the area with good candidates for ensembling around the first fine-tuned network.
We show how StarSSE with different cycle hyperparameters behaves in the bottom row of plots in Figure 1 (for consistency with SSE, we call hyperparameters for training all the networks except the first one as cycle hyperparameters). The results on other datasets are presented in Appendix E. StarSSE demonstrates more stable results for medium and high values of cycle hyperparameters than SSE, and as a result, it produces better larger size ensembles. Even though for the highest considered hyperparameter values, StarSSE quality still decreases with increasing ensemble size, its degradation is much less severe. As can be seen from Table 1, the optimal StarSSE outperforms the optimal SSE.
To better understand the difference between StarSSE and SSE behavior, we conduct the same analysis of linear connectivity and individual model quality for three distinct StarSSE experiments, as in the previous section for SSE. For easier comparison, we choose the same cycle hyperparameters for more local and more semi-local experiments as for their SSE counterparts. For the optimal experiment, we choose its own hyperparameter values to compare the best results of the methods. The exact values of hyperparameters can be found in Appendix B. In the right pair of plots in Figure 2, we show how train and test accuracy of StarSSE behave along the line segment between the first and any of the following models (it does not matter which model to choose because all of them are trained independently in the same manner). The more semi-local StarSSE experiment demonstrates much more local behavior from the loss landscape perspective than the SSE one, which confirms that StarSSE does not go too far from the pre-train basin, even with relatively high hyperparameter values. The optimal StarSSE experiment demonstrates local behavior; therefore, StarSSE, similarly to SSE, should be used only as a local method in the transfer learning setup. As shown in the right pair of plots in Figure 3, all the models in StarSSE, starting from the second one, have a very similar quality, which makes StarSSE more stable for larger ensemble sizes. For the optimal experiment, the quality degradation between the first and the following models is very low and can be compensated by the model diversity in StarSSE to achieve higher results than Local DE (see Appendix J for detailed diversity analysis).
In conclusion, StarSSE is a more effective alternative to cyclical learning rate methods for the transfer learning setup, which can work with higher values of cycle hyperparameters to obtain diverse models without gradual degradation of individual model quality. StarSSE is technically very similar to Local DE, however, there is an important conceptual difference between them. StarSSE separates the two processes: 1) model fine-tuning from the pre-train checkpoint to a high-quality area for the target task, and 2) exploration of this area for obtaining diverse models. On the contrary, Local DE tries to do both of them jointly each time it fine-tunes a network. The design of StarSSE allows it to make exploration with higher hyperparameter values, which are suboptimal for the initial fine-tuning process, and obtain a more diverse set of models. The fact that optimal hyperparameter values for StarSSE are higher than the optimal  $\times 1$  values for Local DE (see Figure 1) supports this reasoning.
6 Soups of SSE and StarSSE models
In the previous section, we show that SSE and StarSSE methods can be effective in transfer learning but only in constructing local ensembles. Even though the locality usually leads to limited model diversity, it also has its benefits: it allows to weight-average a set of models to obtain a single, more accurate, and robust model. To constitute a strong weight-averaged model, a set of models needs to lie in the same low-loss basin and have a convex loss landscape between the models. Diversifying the models in a set can improve the averaging results; therefore, model soups (Wortsman et al., 2022; Rame et al., 2022), which average independently fine-tuned models, show better results than SWA (Izmailov et al., 2018), which averages several checkpoints from the same training run. Moreover, for even higher diversity, model soups are usually constructed from the models fine-tuned with variable hyperparameters. However, the relation between the final quality and the diversity of the models is not as straightforward for weight-averaging as for ensembles because diversifying models too much may result in a non-convex loss between them. Because of this, averaging all fine-tuned models into a uniform model soup does not result in a high-quality model, and, to constitute an effective greedy model soup, a subset of models needs to be chosen using the validation data.
Linear connectivity analysis in Figure 2 shows that averaging two models from the Local DE does not lead to a strong model, while models from optimal SSE and StarSSE experiments lay in a more convex part of the pre-train basin and average much better. This observation motivates us to investigate the weight-averaging of models from SSE or StarSSE to construct uniform model soups. We compare ensembles and uniform model soups of models obtained with Local DE, SSE, and StarSSE in the top row in Figure 4 (see Appendix H for results with supervised pre-training). Firstly, weight-averaging
Figure 4: Results of ensembles (left plots) and model soups (right plots) of different sizes on ID (CIFAR-100 test set, top row) and OOD (CIFAR-100C, bottom row) for SSE and StarSSE with self-supervised pre-training. For OOD, we measure the average accuracy over all possible corruptions and severity values. Standard deviations are calculated over different pre-training checkpoints and/or fine-tuning random seeds.
Local DE models does not improve the quality over individual models. Model soups from more local and optimal SSE and StarSSE experiments show better results than an individual fine-tuned model. However, for the more local experiments, the improvement is minor due to the low model diversity. Weight-averaging networks from the more local SSE in our setup is very close to standard SWA. Models from more semi-local SSE and StarSSE experiments can not constitute adequate model soups due to the low-accuracy barriers between the models. Comparison between SSE and StarSSE optimal experiments shows the advantage of the latter: the soup of StarSSE models demonstrates better results than from SSE ones and matches the quality of the ensemble of the same models.
To conclude, StarSSE can collect a set of models in the pre-train basin, which are diverse enough to constitute a strong local ensemble and, at the same time, are located in a convex part of the basin and result in a strong uniform model soup after weight-averaging. In standard model soups, models are usually fine-tuned with varying hyperparameters for higher diversity, and StarSSE can also benefit from this hyperparameter diversification (see Appendix K). This makes the further analysis of their combination a promising direction for future research.
7 Robustness analysis
One of the main benefits of ensembles and model soups is their robustness to different types of test data modifications. In this section, we evaluate the robustness properties of the ensemble and soup variants of SSE and StarSSE on the corrupted version of the CIFAR-100 dataset. CIFAR-100C dataset (Hendrycks and Dietterich, 2019) includes 19 synthetic corruptions (e.g., Gaussian noise or JPEG compression), each having 5 different severity values. We compare the performance of all the methods on the in-distribution CIFAR-100 (ID) and the out-of-distribution CIFAR-100C (OOD) in Figure 4 (see Appendix I for results with supervised pre-training). For SSE and StarSSE, we consider the same three differently behaving experiments that are used in the previous sections.
First of all, we observe that more local ensembles lose their potential on OOD completely and have near-constant accuracy for all the considered ensemble sizes. In contrast, the behavior of more semi-local ensembles is very similar on ID and OOD in comparison to the Local DE. This difference indicates that model diversity is even more important under domain shifts. The quality of optimal
Model soups
Figure 5: Results of ensembles (left plots) and model soups (right plots) of CLIP models on ImageNet. Three differently behaving SSE and StarSSE experiments are shown with Local and Global DEs for comparison. Hyperparameters for more local and more semi-local experiments are the same for SSE and StarSSE, while hyperparameters for the optimal experiments may differ.
enssembles also decreases compared to a Local DE, agreeing with their local behavior. Overall, ensemble variants of SSE and StarSSE are less robust than DEs to the test data corruption. We discuss the reasons for such behavior in more detail in Appendix J.
For model soups, we observe similar changes in the behavior of more local and more semi-local experiments. The former can not provide a better quality model on OOD, while the results of the latter even improve in comparison to the soup of Local DE models. Thus, model diversity matters for the soups too. Interestingly, optimal soups significantly outperform optimal ensembles, while on the clean data, these two paradigms show almost the same results. This observation matches with the findings of Wortman et al. (2022), who claim the superiority of standard model soups over ensembles under distribution shifts. Overall, the best results on OOD data are achieved with an optimal soup of StarSSE models (of those trained from a single pre-trained checkpoint). It not only outperforms its ensemble variant but also provides considerably better results than Local DE. Thus, a soup of StarSSE models is a good option for fine-tuning a robust model given a single pre-trained checkpoint.
8 Large-scale experiments
Our main analysis of SSE and StarSSE methods is focused on moderate-size networks and smaller target tasks to be able to accurately compare the results with both Local and Global DE baselines. However, to confirm that the main conclusions of the paper stand in more practical large-scale setups, we experiment with the CLIP ViT-B/32 model pre-trained on the large dataset of the image-text pairs and the ImageNet target task. In these experiments, we drop the Global DE baseline due to its extremely high computational cost. In Figure 5, we compare the results of three differently behaving SSEs and StarSSEs (see Appendix B for specific hyperparameter values) and the Local DE baseline. SSE and StarSSE with optimal hyperparameter values significantly outperform Local DE both as ensembles and as uniform model soups. The comparison of StarSSE and SSE confirms previously discussed benefits of the parallel training strategy: StarSSE ensembles and model soups, trained with high hyperparameter values, show less quality degradation than their corresponding SSE counterparts. We conclude that StarSSE is effective for constructing better local ensembles in practical large-scale setups too.
In this work, we study the effectiveness of cyclical learning rate methods for training ensembles from a single pre-trained checkpoint in the transfer learning setup. We demonstrate that while better exploration of the pre-train basin with these methods can be beneficial, they can not leave the basin without losing the advantages of transfer learning and significant quality degradation. Based on our analysis, we also propose a parallel version of the SSE method for transfer learning, StarSSE. Models from StarSSE not only constitute a better ensemble but also lay in a more convex part of the pre-train basin and can be effectively combined into a high-quality model soup.
We would like to thank Maxim Kodryan, Arsenii Kuznetsov, and the anonymous reviewers for their valuable comments. Also, we are very grateful to Polina Kirichenko and Timur Garipov for presenting the paper at the NeurIPS poster session, which we could not attend, unfortunately. The results on ResNet with self-supervised pre-training were obtained by Ildus Sadrdinov with the support of the grant for research centers in the field of AI provided by the Analytical Center for the Government of the Russian Federation (ACRF) in accordance with the agreement on the provision of subsidies (identifier of the agreement 000000D730321P5Q0002) and the agreement with HSE University No70-2021-00139. The empirical results were supported in part through the computational resources of HPC facilities at HSE University (Kostenetskiy et al., 2021).
Ainsworth, S., Hayase, J., and Srinivasa, S. (2023). Git re-basin: Merging models modulo permutation symmetries. In International Conference on Learning Representations (ICLR).
Ashukha, A., Lyzhov, A., Molchanov, D., and Vetrov, D. (2020). Pitfalls of in-domain uncertainty estimation and ensembling in deep learning. In International Conference on Learning Representations (ICLR).
Benton, G. W., Maddox, W. J., Lotfi, S., and Wilson, A. G. (2021). Loss surface simplexes for mode connecting volumes and fast ensembling. In International Conference on Machine Learning (ICML).
da Costa, V. G. T., Fini, E., Nabi, M., Sebe, N., and Ricci, E. (2022). solo-learn: A library of self-supervised methods for visual representation learning. Journal of Machine Learning Research (JMLR), 23(56).
Entezari, R., Sedghi, H., Saukh, O., and Neyshabur, B. (2022). The role of permutation invariance in linear mode connectivity of neural networks. In International Conference on Learning Representations (ICLR).
Ericsson, L., Gouk, H., and Hospedales, T. M. (2021). How Well Do Self-Supervised Models Transfer? In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Fort, S., Hu, H., and Lakshminarayanan, B. (2019). Deep ensembles: A loss landscape perspective. arXiv preprint arXiv:1912.02757.
Garipov, T., Izmailov, P., Podoprikhin, D., Vetrov, D. P., and Wilson, A. G. (2018). Loss surfaces, mode connectivity, and fast ensembling of dnns. In Advances in Neural Information Processing Systems (NeurIPS).
Gontijo-Lopes, R., Dauphin, Y., and Cubuk, E. D. (2022). No one representation to rule them all: Overlapping features of training methods. In International Conference on Learning Representations (ICLR).
Grill, J.-B., Strub, F., Alché, F., Tallec, C., Richemond, P., Buchatskaya, E., Doersch, C., Avila Pires, B., Guo, Z., Gheshlaghi Azar, M., Piot, B., kavukcuoglu, k., Munos, R., and Valko, M. (2020). Bootstrap your own latent - a new approach to self-supervised learning. In Advances in Neural Information Processing Systems (NeurIPS).
Gustafsson, F. K., Danelljan, M., and Schon, T. B. (2020). Evaluating scalable bayesian deep learning methods for robust computer vision. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops.
He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Hendrycks, D. and Dietterich, T. (2019). Benchmarking neural network robustness to common corruptions and perturbations. International Conference on Learning Representations (ICLR).
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. (2021). Learning transferable visual models from natural language supervision. In Meila, M. and Zhang, T., editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8748-8763. PMLR.
Rame, A., Kirchmeyer, M., Rahier, T., Rakotomamonjy, A., patrick gallinari, and Cord, M. (2022). Diverse weight averaging for out-of-distribution generalization. In Advances in Neural Information Processing Systems (NeurIPS).
Ritter, H., Botev, A., and Barber, D. (2018). A scalable laplace approximation for neural networks. In International Conference on Learning Representations (ICLR).
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., and Fei-Fei, L. (2015). Imagenet large scale visual recognition challenge. In International Journal of Computer Vision (IJCV).
Shwartz-Ziv, R., Goldblum, M., Souri, H., Kapoor, S., Zhu, C., LeCun, Y., and Wilson, A. G. (2022). Pre-train your loss: Easy bayesian transfer learning with informative priors. In Advances in Neural Information Processing Systems (NeurIPS).
Strauss, T., Hanselmann, M., Junginger, A., and Ulmer, H. (2017). Ensemble methods as a defense to adversarial perturbations against deep neural networks. arXiv preprint arXiv:1709.03423.
Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., and Summers, R. M. (2017). ChestX-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Wortsman, M., Ilharco, G., Gadre, S. Y., Roelofs, R., Gontijo-Lopes, R., Morcos, A. S., Namkoong, H., Farhadi, A., Carmon, Y., Kornblith, S., and Schmidt, L. (2022). Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning (ICML).
Xiao, J., Hays, J., Ehinger, K. A., Oliva, A., and Torralba, A. (2010). Sun database: Large-scale scene recognition from abbey to zoo. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR).
Zhang, R., Li, C., Zhang, J., Chen, C., and Wilson, A. G. (2020). Cyclical stochastic gradient MCMC for bayesian deep learning. In International Conference on Learning Representations (ICLR).
A Limitations and societal impact
The main limitation of our work is choosing the hyperparameter values for Local and Global DEs based on the validation accuracy of individual models. Even though it is standard practice for DEs, such hyperparameters may lead to suboptimal ensemble quality. However, we used the same hyperparameters for the training of the first SSE and StarSSE models, so their quality may also be suboptimal. Also, an extension of the study to other domains may be an interesting addition to our work.
To the best of our knowledge, our work does not have any negative societal impact, except the substantial computational costs of the conducted study. However, our work improves understanding of the ensembles and model soups construction using only one pre-trained checkpoint, which can help to reduce computational costs in future projects.
B Experimental setup details
All the described experiments were carried out using the PyTorch framework (Paszke et al., 2019).
Pre-training of ResNets. For the supervised case, we use available ResNet-50 checkpoints<sup>2</sup> pretrained by Ashukha et al. (2020). We take five of them trained from different random initializations with the same hyperparameters. For the self-supervised case, we also pre-train five checkpoints from scratch for 1000 epochs using the hyperparameters from the original BYOL paper. We use the solo-learn library for BYOL pre-training (da Costa et al., 2022).
Swin-T. We pre-train five Swin-T checkpoints from scratch in a supervised manner on ImageNet for 300 epochs. We use the official code and hyperparameters from the original Swin paper. For fine-tuning of individual models for Local and Global DEs, we use AdamW (Loshchilov and Hutter, 2019) and linear learning rate warm-up for  $10\%$  of the fine-tuning epochs. For SSE and StarSSE, we also use AdamW for each cycle but the learning rate warm-up is applied only in the first cycle.