SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 5: Accuracy of individual models and ensemble diversity for in-distribution (ID, CIFAR-100 test set) and out-of-distribution (OOD, CIFAR-100C) data for supervised (SV) and self-supervised (SSL) pre-training. The OOD results are averaged over different corruptions and values of severity. Standard deviations for both ID and OOD are shown over different pre-training checkpoints and training seeds.
Pre-train.	Method	accuracy (single)	diversity
ID	OOD	ID	OOD
SV	Local DE	84.47±0.15	57.46±0.81	60.21±1.37	58.72±2.33
Global DE	82.58±1.70	79.05±2.32
SSE	more local	84.24±0.23	56.30±0.99	64.01±8.94	60.80±8.32
optimal	84.18±0.49	56.38±1.27	70.50±8.78	67.79±9.06
more semi-local	82.03±1.45	54.38±2.52	81.35±12.32	78.40±11.63
StarSSE	more local	84.35±0.20	57.06±0.69	55.30±1.74	53.62±2.29
optimal	84.55±0.17	56.77±1.03	73.28±4.42	69.07±4.23
more semi-local	82.95±0.75	55.31±1.80	82.87±9.47	78.63±9.30
SSL	Local DE	85.96±0.28	58.53±0.82	66.25±3.56	62.75±3.38
Global DE	80.21±1.62	75.93±2.36
SSE	more local	85.87±0.24	57.83±0.85	47.46±7.23	44.61±7.64
optimal	85.27±0.49	57.56±1.26	71.57±5.94	66.98±6.36
more semi-local	82.52±1.96	54.20±2.94	80.93±9.57	78.68±10.37
StarSSE	more local	85.95±0.32	58.13±0.76	38.94±1.57	36.12±1.80
optimal	85.79±0.28	57.91±0.97	71.51±2.08	67.37±3.12
more semi-local	83.65±1.20	55.48±2.18	82.42±8.59	79.22±8.24
Table 6: Accuracy of individual models and ensembles of size 5 for Local DE, Global DE and optimal StarSSE, trained with and without hyperparameter and augmentation diversification of ensemble members. Self-supervised pre-training.
Dataset	Diversity	Local DE (single)	Local DE (ensemble)	Global DE (ensemble)	StarSSE (single)	StarSSE (ensemble)
hyp.	aug.
CIFAR-100	-	-	85.96±0.28	87.33±0.21	88.10±0.11	85.79±0.28	87.63±0.12
-	+	85.95±0.27	87.45±0.15	88.20±0.03	85.91±0.17	87.82±0.04
+	-	85.85±0.28	87.60±0.21	88.25±0.05	85.69±0.27	87.39±0.06
+	+	85.24±0.38	87.54±0.14	88.24±0.17	85.81±0.24	87.74±0.08
SUN-397	-	-	64.32±0.40	65.77±0.19	67.04±0.09	63.98±0.49	66.37±0.25
-	+	64.25±0.41	65.84±0.29	66.99±0.05	64.23±0.17	66.69±0.15
+	-	64.19±0.52	66.17±0.28	67.33±0.05	63.93±0.40	66.30±0.14
+	+	64.40±0.46	66.51±0.35	67.49±0.10	64.30±0.31	66.59±0.06
diversity on in-distribution CIFAR-100 (ID) and out-of-distribution CIFAR-100C (OOD) data for both types of pre-training.
Despite Local and Global DEs consisting of the models of the same quality level, the latter includes significantly more diverse networks, explaining the quality gap between the two. Generally, when we increase the hyperparameters of SSE/StarSSE, the networks become more diverse, but their ID accuracy decreases. The models of the optimal SSE/StarSSE experiments are much more diverse than the ones of the more local experiments, which makes up for lower ID accuracy when constructing ensembles. Moreover, the high diversity of models in optimal SSE/StarSSE exceeds the one of the Local DE models, allowing optimal StarSSE (and sometimes optimal SSE) to outperform the Local DE on clean data. The models of the more semi-local experiments achieve even higher diversity, though it is not enough to overcome their significant quality degradation.
Our robustness analysis in Section 7 (Figure 4) and Appendix I (Figure 16) shows that optimal SSE and StarSSE are less preferable under the distribution shifts than Local DE. The results in Table 5 confirm that the main reason for this is not the lower diversity but the lower quality of individual models. Generally, SSE and StarSSE models tend to overfit to target data — this is the main reason why semi-local methods do not show positive results in the transfer learning setup. Choosing the optimal hyperparameters for SSE/StarSSE results in low/almost no degradation of individual model quality on ID data (see Figures 3,13 and Table 5). However, overfitting becomes noticeable faster on OOD data, hence, individual SSE/StarSSE model quality degrades more significantly in this case.
K Diversifying ensembles
One of the most common techniques to improve ensemble quality is to diversify networks by training them in different ways (varying optimization algorithm, learning rate, weight decay, data augmentation, etc.). This technique is shown to be effective for both Local (Mustafa et al., 2020; Wortsman et al., 2022; Rame et al., 2022) and Global DE (Gontijo-Lopes et al., 2022). In this section, we show that such diversification can also improve StarSSE and thus does not change the results of methods comparison.
We consider ResNet-50 with self-supervised pre-training on CIFAR-100 and SUN-397 datasets and try two types of model diversification:
- hyperparameter diversification — instead of training ensemble members with a fixed learning rate, weight decay, and number of epochs, we sample these hyperparameters from the log-uniform distributions on segments around their optimal values:
- CIFAR-100: learning rate — [0.035, 0.07], number of epochs — [20, 50] for Local DE; learning rate — [0.08, 0.12], number of epochs — [18, 35] for StarSSE; weight decay —  $[5 \cdot 10^{-5}, 2 \cdot 10^{-4}]$  for both.
Table 7: Accuracy of individual models and ensembles of size 5 for three differently behaving SSEs without regularization, with EWC regularization to the pre-trained model (EWC-PT) and with L2 regularization to the first fine-tuned model (L2-FT) on CIFAR-100 for self-supervised pre-training. StarSSE results are given for comparison.
more local	optimal	more semi-local
(single)	(ensemble)	(single)	(ensemble)	(single)	(ensemble)
SSE	85.87±0.24	86.57±0.07	85.27±0.49	87.11±0.08	82.52±1.96	85.60±0.10
+ EWC-PT	85.83±0.21	86.55±0.05	85.57±0.45	87.22±0.05	82.73±1.75	85.69±0.04
+ L2-FT	85.94±0.16	86.62±0.04	85.79±0.22	87.26±0.05	85.65±0.21	87.26±0.04
StarSSE	85.95±0.32	86.41±0.30	85.79±0.28	87.63±0.12	83.65±1.20	86.20±0.05
- SUN-397: learning rate — [0.1, 0.15], number of epochs — [15, 30] for Local DE; learning rate — [0.1, 0.25], number of epochs — [15, 23] for StarSSE; weight decay —  $[5 \cdot 10^{-5}, 10^{-4}]$  for both.
- augmentation diversification — complementary to fixed data augmentation (random crop and random flip), we use four color augmentations (adjusting brightness, contrast, hue, and saturation) similar to Mustafa et al. (2020), Appendix A.3 (though we needed to change the brightness parameters because of a different parametrization used by torchvision (Paszke et al., 2019)). We sample minimal and maximal limits for the strength of these augmentations before each model training:
- brightness:  $s_{\min} \sim \text{Uniform}(0.25, 0.75)$ ,  $s_{\max} \sim \text{Uniform}(2 \cdot s_{\min}, 4 \cdot s_{\min})$
-hue:  $\delta_{\mathrm{max}}\sim$  Uniform(0.01,0.2),  $\delta_{\mathrm{min}} = -\delta_{\mathrm{max}}$
- saturation:  $s_{\min} \sim \text{Uniform}(0.25, 0.75)$ ,  $s_{\max} \sim \text{Uniform}(2 \cdot s_{\min}, 4 \cdot s_{\min})$
- contrast:  $s_{\mathrm{min}} \sim \text{Uniform}(0.25, 0.75)$ ,  $s_{\mathrm{max}} \sim \text{Uniform}(2 \cdot s_{\mathrm{min}}, 4 \cdot s_{\mathrm{min}})$
During training, each augmentation is applied with  $50\%$  probability, and its strength is sampled within the corresponding limits.
For diversified StarSSE, we train the first model with fixed optimal hyperparameters and augmentations and then train all the following models utilizing the diversification techniques.
In Table 6 we show how model diversification influences the average accuracy of individual models and ensembles of size 5 for Local DE, Global DE, and StarSSE. All considered ensemble types benefit from the diversification: while Global and Local DEs improve more from hyperparameter diversification, StarSSE is stronger in combination with augmentation diversification. The latter shows that proper augmentations are very important for StarSSE since longer fine-tuning of StarSSE networks makes them more prone to overfitting. In conclusion, network training diversification can improve the results of all ensemble types and thus can not close the quality gaps between them.
L Regularizing SSE to stay in the pre-trained basin
In this section, we experiment with different regularization techniques to prevent SSE from overfitting and losing the advantages of transfer learning. As a first technique, we consider Elastic Weight Consolidation (EWC, Kirkpatrick et al. (2017)), a popular continual learning method, which employs weighted L2 regularization to the pre-trained checkpoint. The weight of each parameter is a diagonal element of the Fisher information matrix (FIM) evaluated on the pre-training task. Hence, EWC not only regularizes the weights to not to go too far from the pre-train checkpoint, but also takes into account the pre-train loss landscape to control the direction of weights' changes. This makes EWC a very similar technique to the informative prior of Shwartz-Ziv et al. (2022), which also utilises the pre-train loss landscape to construct the Gaussian prior on network weights. We consider three values of the regularization coefficient  $\lambda_{EWC} \in [0.1, 1, 10]$ , choosing the one performing best on the test set  $\lambda_{EWC}^{*} = 10$ .
As a second technique, we consider regularization of SSE to the first fine-tuned model to force local behavior. In this case, we use isotropic L2 regularization. The reasons for such a simplification are two-fold: 1) we moved away from the pre-trained checkpoint, making it impossible to evaluate FIM
Figure 17: Results of ensembles (left plots) and model soups (right plots) of Swin-T models with supervised pre-training. Three differently behaving SSE and StarSSE experiments on CIFAR-100 and SUN-397 are shown with Local and Global DEs for comparison. Hyperparameters for more local and more semi-local experiments are the same for SSE and StarSSE on the same dataset, while hyperparameters for the optimal experiments may differ.
on the pre-train task without refitting the classifier head, and 2) there is no need to evaluate FIM on the target task, as we already have access to the target loss landscape during the optimization. We consider three values of the regularization coefficient  $\lambda_{L2} \in [10^{-5}, 10^{-4}, 10^{-3}]$ , choosing the one performing best on the test set  $\lambda_{L2}^{*} = 10^{-4}$ . In both cases (EWC to the pre-trained checkpoint and L2 to the first fine-tuned model), the linear head is not regularized.
In Table 7, we consider three differently behaving SSEs and add regularization to each of them, preserving the original hyperparameter values (i. e., the maximum learning rate and the number of epochs for each cycle). Overall, the EWC regularization has little impact on the accuracy of individual models and the ensemble. In contrast, L2 regularization increases both metrics with the most pronounced improvements for the more semi-local experiment, which becomes similar to the optimal one. Generally, both regularization techniques are able to improve the SSE results by promoting a more local behavior and preventing SSE from leaving the pre-train basin, however they can not help to explore the vicinity of the pre-train basin more effectively. Moreover, regularized variants of SSE underperform the optimal StarSSE, making our parallel method the most effective for the local exploration of the pre-train basin.
M Additional experiments with Swin-T
In this section, we experiment with Swin-T models pre-trained in a supervised manner. We consider only CIFAR-100 and SUN-397 target tasks because the gap between Local DE and Global DE on CIFAR-10 is too small. For more experimental details see Appendix B. Figure 17 shows the results of ensembles and soups of models from three differently behaving SSE and StarSSE experiments. Generally, we observe the same types of behavior as for ResNet-50. Optimal ensemble variants of SSE and StarSSE show similar results to Local DE, while soups of models from optimal SSE and StarSSE are much more accurate than the ones from Local DE models.
Figure 18: Results of ensembles (left plots) and model soups (right plots) of different sizes on non-natural image classification datasets Clipart and ChestX for SSE and StarSSE with supervised and self-supervised pre-training. Hyperparameters for more local and more semi-local experiments are the same for SSE and StarSSE on the same dataset, while hyperparameters for the optimal experiments may differ.
N Additional experiments on non-natural image datasets
In this section, we experiment with two non-natural datasets: ChestX dataset (Wang et al., 2017) containing medical images and a clipart image collection DomainNet Clipart (Peng et al., 2019). We consider the ResNet-50 network with supervised and self-supervised pre-training on ImageNet. Following Ericsson et al. (2021), we drop all multi-label images from ChestX and solve a usual classification problem. For more experimental details, see Appendix B.
Figure 18 shows the results of ensembles and soups of models from three differently behaving SSE and StarSSE experiments. We also provide the quality and diversity of ensemble members in Table 8 for a deeper analysis of StarSSE results. Generally, we observe more extreme results, most likely due to higher discrepancy between source and target tasks.
On Clipart, StarSSE and SSE either show very close or even inferior results compared to the Local DE. Analyzing the quality and diversity of ensemble members, we conclude that StarSSE does not find a balance between the two for this task. In the supervised case, StarSSE finds stronger individual models with lower diversity than the ones of a Local DE. Hence, the ensemble quality of StarSSE
Table 8: Accuracy of individual models and ensemble diversity for experiment on non-natural image classification datasets with supervised (SV) and self-supervised (SSL) pre-training.
Pre-train.	Method	Clipart	Chestx
acc. (single)	diversity	acc. (single)	diversity
SV	Local DE	74.58±0.16	66.74±1.65	51.86±0.62	42.42±1.30
Global DE	75.56±1.70	43.81±1.45
StarSSE (opt.)	75.21±0.21	64.28±1.55	53.93±0.47	46.06±4.41
SSL	Local DE	75.28±0.31	56.61±1.83	54.32±0.49	38.62±1.62
Global DE	68.46±1.46	40.05±1.76
StarSSE (opt.)	74.58±0.16	67.24±1.29	54.42±0.5	46.03±2.70
is lower, but at the same time, the soup of StarSSE models improves over the Local DE one. In the self-supervised case, models from StarSSE are more diverse but are of lower quality than the Local DE ones. As a result, StarSSE and Local DE show very similar results in this case.
On the contrary, the results on ChestX are very positive: SSE and StarSSE outperform both Local and Global DEs. Interestingly, Local and Global DEs work very similarly on this task. In both supervised and self-supervised cases, StarSSE finds a much more diverse set of models in a more convex part of the loss landscape, hence, resulting in better ensembles and models soups. Additionally, in the supervised case, StarSSE fine-tunes individual models more effectively than Local and Global DEs, improving the results even further.