Year: 2021

Abstract:
Despite being able to capture a range of features of the data, high accuracy models trained with supervision tend to make similar predictions. This seemingly implies that high-performing models share similar biases regardless of training methodology, which would limit ensembling benefits and render low-accuracy models as having little practical use. Against this backdrop, recent work has made very different training techniques, such as large-scale contrastive learning, yield competitively-high accuracy on generalization and robustness benchmarks. This motivates us to revisit the assumption that models necessarily learn similar functions. We conduct a large-scale empirical study of models across hyper-parameters, architectures, frameworks, and datasets. We find that model pairs that diverge more in training methodology display categorically different generalization behavior, producing increasingly uncorrelated errors. We show these models specialize in subdomains of the data, leading to higher ensemble performance: with just 2 models (each with ImageNet accuracy 76.5%), we can create ensembles with 83.4% (+7% boost). Surprisingly, we find that even significantly low-accuracy models can be used to improve high-accuracy models. Finally, we show diverging training methodology yield representations that capture overlapping (but not supersetting) feature sets which, when combined, lead to increased downstream performance.

heading: 1 INTRODUCTION
text: Over the years, the machine learning field has developed myriad techniques for training neural networks. In image classification, these include data augmentation, regularization, architectures, losses, pre-training schemes, and more. Such techniques have highlighted the ability of networks to capture diverse features of the data: textures/shapes (Geirhos et al., 2018), robust/non-robust features (Ilyas et al., 2019), and even features that fit a random, pre-determined classifier (Hoffer et al., 2018).
Despite this representation-learning power, methods that yield high generalization performance seem to produce networks with little behavior diversity: models make similar predictions, with high-accuracy models rarely making mistakes that low-accuracy models predict correctly (Mania et al., 2019). Additionally, the quality of features learned (e.g.: for downstream tasks) seems dictated by upstream performance (Kornblith et al., 2019). Finally, training on subsets of the data yields low-accuracy models that don’t make performant ensembles (Nixon et al., 2020). This seemingly suggests that high-performing models share similar biases, regardless of training methodology.
Without behavior diversity, ensemble benefits are limited to reducing noise, since models make correlated errors (Perrone & Cooper, 1992; Opitz & Maclin, 1999). Without feature diversity, representations might not capture important features for downstream tasks, since feature reuse has been shown to be crucial for transfer learning (Neyshabur et al., 2020). Without knowing the effect of training methodology, one might conclude that low-accuracy models have no practical use, since their predictions would be dominated by high-accuracy ones.
One open question is if these findings faced unavoidable selection bias, since the highest-performing models have historically been trained with similar supervised objectives on IID datasets. Up until recently, this hypothesis was difficult to test. That changed with the recent success of large-scale contrastive learning, which produces competitively-high accuracy on standard generalization and robustness benchmarks (Radford et al., 2021; Jia et al., 2021). This motivates revisiting the question:
How does training methodology affect learned representations and prediction behavior?
In this paper, we conduct a systematic empirical study of 82 models, which we train or collect, across hyper-parameters, architectures, objective functions, and datasets, including the latest high performing models CLIP, ALIGN, SimCLR, BiT, ViT-G/14, and MPL. In addition to using different techniques, these new models were trained on data collected very differently, allowing us to probe the effect of both training objective, as well as pre-training data. We categorize these models based on how their training methodologies diverge from a typical, base model and show:
1. Model pairs that diverge more in training methodology (reinitializations, hyper-parameters, architectures, frameworks, → dataset scales) produce increasingly uncorrelated errors.
2. Ensemble performance increases as error correlation decreases, due to higher ensemble efficiency. The most typical ImageNet model (ResNet-50, 76.5%), and its most different counterpart (ALIGN-ZS, 75.5%) yield 83.4% accuracy when ensembled, a +7% boost.
3. Contrastively-learned models display categorically different generalization behavior, specializing in subdomains of the data, which explains the higher ensembling efficiency. We show CLIP-S specializes in antropogenic images, whereas ResNet-50 excells in nature images.
4. Surprisingly, we find that low-accuracy models can be useful if they are trained differently enough. By combining a high-accuracy model (BiT-1k, 82.9%) with only low-accuracy models (max individual acc. 77.4%), we can create ensembles that yield as much as 86.7%.
5. Diverging training methodology yield representations that capture overlapping (but not supersetting) feature sets which, when combined, lead to increased downstream performance (91.4% on Pascal VOC, using models with max individual accuracy 90.7%).

heading: 2 RELATED WORK
text: Diversity in Ensembles. It is widely understood that good ensembles are made of models that are both accurate and make independent errors (Perrone & Cooper, 1992; Opitz & Maclin, 1999; Wen et al., 2020). Beyond improving ensemble performance, finding diverse solutions that equally well explain the observations can help quantify model uncertainty (also known as epistemic uncertainty) – what the model does not know because training data was not appropriate (Kendall & Gal, 2017; Fort et al., 2019). Many works have explored ways of finding such solutions (Izmailov et al., 2018). Boostrapping (Freund et al., 1996) (ensembling models trained on subsets of the data) was found not to produce deep ensembles with higher accuracy than a single model trained on the entire dataset (Nixon et al., 2020). This emphasizes how much data deep neural networks need to achieve high performance. Another work has examined the effect of augmentation-induced prediction diversity on adversarial robustness (Liu et al., 2019). More relevant to us, Wenzel et al. (2020) has explored the effect of random hyper-parameters, finding best ensembles when combining models that are both hyperparameter and weight-diverse, albeit still considering similar frameworks and architectures.
Model Behavior Similarity. These attempts were hindered as many high-performing techniques seem to produce similar prediction behavior. Mania et al. (2019) demonstrates, via “dominance probabilities”, that high-accuracy models rarely make mistakes that low-accuracy models predict correctly. This indicates that, within the models studied, high-accuracy models “dominate” the predictions of low-accuracy ones. Recht et al. (2019) shows that out-of-distribution robustness seems correlated with in-distribution performance. Relatedly, Kornblith et al. (2019) shows that upstream and downstream performance are very correlated. These jointly indicate that high-accuracy models learn strictly better representations, diminishing the importance of low-accuracy solutions (even if they are diverse). Finally, Fort et al. (2019) shows that subspace-sampling methods for ensembling generate solutions that, while different in weight space, remain similar in function space, which gives rise to an insufficiently diverse set of predictions.
Contrastive-Learning Models; Different Large-Scale Datasets. This model behavior similarity might be explained by the fact that the training techniques that yield high performance on image classification tasks have been relatively similar, mostly relying on supervised learning on ImageNet, optionally pre-training on a dataset with similar distribution. Recently, various works have demonstrated the effectiveness of learning from large-scale data using contrastive learning (Radford et al., 2021; Jia et al., 2021). They report impressive results on out-of-distribution benchmarks, and have been shown to have higher dominance probabilities (Andreassen et al., 2021). These represent some of the first models to deviate from standard supervised training (or finetuning) on downstream data,
while still yielding competitive accuracy. They add to the set of high-performing training techniques (which include data augmentation (Cubuk et al., 2018; 2020), regularization (Srivastava et al., 2014; Szegedy et al., 2016; Ghiasi et al., 2018), architectures (Tan & Le, 2019; Dosovitskiy et al., 2020; Hu et al., 2018; Iandola et al., 2014; Li et al., 2019; Szegedy et al., 2016; Simonyan & Zisserman, 2014; Sandler et al., 2018), losses (Chen et al., 2020; Radford et al., 2021; Jia et al., 2021), pre-training schemes (Kolesnikov et al., 2020; Pham et al., 2021), etc), and provide the motivation for revisiting the question of whether training methodology can yield different model behavior.

heading: 3 METHOD

heading: 3.1 MODEL CATEGORIZATION
text: In order to evaluate the performance of learned representations as a function of training methodology, we define the following categories, which classify model pairs based on their training differences:
1. Reinits: identical models, just different in reinitialization. 2. Hyper-parameters: models of the same architecture trained with different hyper parameters
(e.g.: weight decay, learning rate, initialization algorithm, etc). 3. Architectures: models with different architectures, but still trained within the same frame-
work and dataset (e.g.: ResNet and ViT, both with ImageNet supervision). 4. Frameworks: models trained with different optimization objectives, but on same dataset
(e.g.: ResNet and SimCLR, respectively supervised and contrastive learning on ImageNet). 5. Datasets: models trained on large-scale data (e.g.: CLIP or BiT – trained on WIT or JFT).
In some sense, model categories can be supersets of one another: when we change a model architecture, we may also change the hyper-parameters used to train such architecture, to make sure that they are optimal for training this new setting. Unless stated otherwise, all ensembles are comprised of a fixed base model, and another model belonging to one of the categories above. This way, each category is defined relative to the base model: model pairs in a given category will vary because Model 2 is different than the base model along that axis. The result is that as we navigate along model categories (Reinit → ... → Dataset), we will naturally be measuring the effect of increasingly dissimilar training methodology. See Appendix Table 1 for details.

heading: 3.2 MODEL SELECTION
text: We collect representations and predictions for 82 models, across the many categories above. We fix ResNet-50, trained with RandAugment, as our base model. ResNet is a good candidate for a base model since it is one of the most typical ImageNet classification models, and the de-facto standard baseline for this task. In total, we train/collect models in the categories: 1) Reinit; 2) Hyperparameters (51): varying dropout, dropblock, learning rate, and weight decay, sometimes jointly; 3) Architectures (17): including EfficientNet, ViT, DenseNet, VGG; 4) Framework (2): including SimCLR, and models trained with distillation; and 5) Dataset (12): including CLIP, ALIGN, BiT, and more, trained on WIT (Radford et al., 2021), the ALIGN dataset, JFT (Sun et al., 2017), etc. We additionally collect high-performing models MPL, ALIGN (L-BFGS), ViT-G/14, BiT-1k, CLIP-L, EfficientNet-B3 for some of our analyses. These are some of the latest, highest-performing models for ImageNet classification.
We found it necessary to calibrate all models using temperature scaling (Roelofs et al., 2020; Guo et al., 2017) to maximize ensemble performance. Finally, unless stated otherwise, we only use models in the same narrow accuracy range (74-78% accuracy on ImageNet), which guarantees that the effects observed are indeed a function of diverging training methodology, and not of any given model being intrinsically more performant than another. A complete list of models can be found in the Appendix.

heading: 3.3 ENSEMBLING
text: In our paper, we use ensembling as a tool for understanding: as such, our goal is not to find methods to ensemble the highest accuracy models and reach state-of-the-art. Instead, we wish to use ensembling to probe whether/when training methodology yields uncorrelated (and therefore useful) predictions.

heading: 4 RESULTS
text: In order to understand whether model similarity reported in literature varies as a function of training methodology, we evaluate error correlation by measuring the number of test-set examples where one model predicts the correct class, and the other predicts an incorrect class. We call this Error Inconsistency, as it complements the error consistency measure, defined in Geirhos et al. (2020).
We choose error inconsistency to measure error correlation because it allows us to connect the ensemble prediction diversity (which we are interested in quantifying) directly into performance. That is, when errors are consistent (i.e.: both models make a correct prediction, or both models make an incorrect prediction), this agreement directly translates into higher/lower accuracy after ensembling. When errors are inconsistent (i.e.: when one model is correct and another is incorrect), the ensemble prediction will be determined by the models’ confidences. If a model is sufficiently confident, its prediction will “win” the ensembling procedure. If this model’s prediction was correct, we say that this prediction was “converted” into a correct ensemble prediction.

heading: 4.1 AS TRAINING METHODOLOGY DIVERGES, ERRORS BECOME MORE UNCORRELATED
text: In Fig. 1 we see that, as we compare increasingly different training methodologies (“Reinit” → ... → “Dataset”), error inconsistency increases – the number of examples where only one model makes a correct prediction. This indicates that as training methodology diverges, model predictions become dissimilar and, as a result, errors become uncorrelated.
This also represents an opportunity for these uncorrelated mistakes to be converted into a correct prediction, when we ensemble the models. Such an opportunity will be beneficial if the conversion rate of these examples is higher than the decrease in number of examples where both models made correct predictions. As we will see in the following section, this indeed happens.

heading: 4.2 AS UNCORRELATED ERRORS INCREASE, SO DOES ENSEMBLE EFFICIENCY
text: In order to assess whether these increased uncorrelated errors can be beneficial, we measure how models in the various categories ensemble. That is, we ensemble our base model with models of the various categories, and measure the ensemble’s accuracy, relative to its member models.
Per Fig. 2, as uncorrelated errors increase, so does ensemble performance (left). Additionally, because we restrict our models to ones within the narrow accuracy range 74-78%, we guarantee that this relative improvement translates into absolute improvement (center), and is not due to any individual ensemble member being intrinsically better than another. Relative to our base model ResNet-50 (76.5% on ImageNet), the most differently-trained model ALIGN-ZeroShot (75.5%) yields 83.4% top-1 accuracy when ensembled, a boost of nearly 7% accuracy.
Taken together, these results mean that combining models with diverse training methodologies can yield performance improvements that offset the decrease in examples where both models are correct. To explain this relative boost, we also measure the conversion rate of each ensemble – the rate at which inconsistent errors are converted into correct ensemble predictions. Surprisingly, this rate also increases as training methods diverge (Fig. 2, right). This means that different training methods not only yield more opportunities for ensemble benefits, but also more efficient ensembles.

heading: 4.3 DISSIMILAR TRAINING METHODOLOGIES CREATE SPECIALIZED MODELS
text: To understand how diverging training setups create efficient ensembles, we measure the specialization of ensemble member models. We first note that, for a model’s prediction to “win” the ensembling process, it only needs to be sufficiently more confident than the other model’s prediction. This means that we can measure specialization by the relative confidence of two models. We do this by measuring the angle distance θ in the confidence-confidence plot (see Fig. 3, top left). When θ is high, Model 1 is more confident (relative to Model 2), and vice-versa.
In Fig. 3, we use θ to understand the specialization of different models categories. To simplify our analysis, we compare three different ensembles, from the categories Reinit (Model 2: ResNet-50), Architecture (EfficientNet-B3), and Dataset (CLIP-S), as they are representative of the spread of error correlation. When we plot a histogram of examples as a function of θ, we find that: 1) As observed in Fig. 1, when training methods diverge, the models make mistakes in different sets of examples 2) As training methods diverge, Model 1 tends to be more confident in examples where only Model 1 is correct, and vice-versa for Model 2. This effect is most striking when we look at the Dataset category, where Model 2 (CLIP-S) is significantly more confident than Model 1 (ResNet-50) in examples where only CLIP-S predicts correctly, and vice-versa. This is in contrast with Reinit, where models don’t seem to be significantly more confident than each other at all.
These results show: as training methodology diverges, model specialize to different data subdomains.

heading: 4.4 MODEL SPECIALIZATION DEPENDS ON TRAINING METHODOLOGY
text: Next, we want to investigate what kind of data each model category specializes to. In Fig. 4, we plot the same data as Fig. 3, but we divide the examples along their ImageNet class IDs. This allows us to inspect whether a given model is more/less specialized in a given class.
As before, we find that, while models in the Architecture category demonstrate higher specialization (Model 1 tends to be more confident in examples where it is correct, and vice versa for Model 2),
this specialization does not seem to correlate with specific classes. In contrast, not only do models of the Dataset category display more specialization, but this specialization seems to be correlated with groups of classes. In particular, CLIP-S seems to specialize to anthropogenic images, whereas ResNet-50 seems to specialize to nature images. We suspect this phenomena is a result of CLIP-S being trained on a dataset that was collected independently, and much differently than the one used to train our base model, ResNet-50.

heading: 4.5 DIFFERENT ENOUGH LOWER-ACCURACY MODELS CAN IMPROVE ACCURACY
text: In classic machine learning, a common way to create high-performing ensembles is to create specialized models with bootstrapping (Freund et al., 1996), where models are trained on subsets of the data and subsequently ensembled. In deep learning, bootstrapping has not been found to work well, since training on IID subsets of the data (with similar methodology) does not seem to yield ensembles that perform better than an individual model trained on the entire dataset (Nixon et al., 2020). This seems to indicate that lower-accuracy models would have little practical benefit for performance.
In order to investigate this, and encouraged by the finding that differently-trained models can be experts in subdomains of the data, we ask whether lower-accuracy models can be beneficial. After all, some of the models in the Dataset category were trained on much larger-scale datasets (JFT, WIT), which are collected independently.
To do this, we first combine our base model, ResNet-50 (76.2%), with a lower-accuracy model trained very differently, CLIP-S-ZeroShot (63.3%, “Dataset” category), and observe a performance boost (77.7%). To push this idea further, we combine a high-accuracy model (BiT-1k, 82.85%) with only lower-accuracy models (max individual accuracy 77.44%), by greedily selecting ensemble members that maximize accuracy. With this procedure, we can create ensembles that yield as much as 86.66%. In Fig. 5, we repeat this procedure for 3 additional high-accuracy models and find that lower-accuracy models can consistently improve the accuracy of high-accuracy models. Importantly, the low-accuracy models need to be different enough in their training methodology (in terms of Sec. 3.1 categories): for a given high-accuracy model, the most beneficial lower-accuracy model to ensemble is one trained with a different loss and/or dataset, often both. See table in Fig. 5 for details.
This result calls into question the idea that the information learned by high accuracy models is a strict superset of those learned by low accuracy models, which we explore further in sections below.

heading: 4.6 SPECIALIZATION COMES FROM OVERLAPPING (BUT NOT SUPERSETTING) REPRESENTATIONS
text: In order to explain why differently-trained models specialize, we posit that training methodology changes the features learned by individual models. In this view, models trained differently would have an overlapping (but not supersetting) set of features which, when combined, provide more information for classification than any individual model’s representation could.
We test this directly by concatenating the features of our base model (ResNet-50) with those of the models above (ResNet-50, EfficientNet-B0 and CLIP-S), and linearly evaluating these combined features on ImageNet classification. More specifically, we first randomly select features from the base model and each other model at inversely proportional rates. For example, we concatenate 25% of ResNet-50 features with 75% of CLIP-S features, yielding a final embedding that is at most the same dimensionality of the ResNet-50 features. This guarantees that any performance boost is not due to a higher number of dimensions being available, but by the quality of the features represented.
In Fig. 6 (center), we see that the best performance is obtained when features are combined. Additionally, we find that combining features yields higher performance as training methodology diverges, with the best performing combination being ResNet-50 + CLIP-S. Concurrent work similarly finds that different training objectives yield different final-layer feature embeddings (Grigg et al., 2021). This seems to confirm the idea that the methods used to train networks can generate diverse features, which capture information that neither embedding alone has captured.
To push this further, we ask how efficiently these representations capture important features of the data. To test this, we first compute the covariance of each dimension of embeddings from both models (e.g.: ResNet-50 and CLIP-S). This gives us a ranking of highest to lowest covariance dimensions, which we use as a measure of diversity. We then select features in order of most diverse to least diverse, and linearly evaluate them on ImageNet classification.
Fig. 6 (right) shows how divergently-trained models yield more diverse (and therefore more compressible) features. We tested multiples of 256 features, and found that Reinit models needed 2304 features on average to achieve 76% accuracy. In constrast, Hyperparameter models required 2057, Architecture 1392, and Dataset only required 512 features.

heading: 4.7 DOWNSTREAM TASK PERFORMANCE
text: With the knowledge that diverse training leads to diverse feature embeddings, we ask whether these can transfer better to downstream tasks. In order to maximize performance, we pick three of the highest accuracy models on ImageNet, that are also representative of very diverse training setups. Meta-Pseudo Labels (MPL) was the previous SOTA on ImageNet, reporting 90.24% top-1 accuracy (Pham et al., 2021). It is trained with a teacher network that generate pseudo labels on unlabeled data to teach a student network, whose performance on the labeled set is then used to adapt the teacher. It is trained on the JFT-300M dataset. ViT-G/14 is the current SOTA, with 90.45% top-1 accuracy (Zhai et al., 2021), when measured with EMA (Polyak & Juditsky, 1992). We obtained our image embeddings without the use of EMA, so we find effective accuracy of 88.93% It is trained supervisedly, on JFT-3B, a larger version of the JFT-300M dataset. Finally, CLIP-S is a constrastive learning model, yielding 75.7% top-1 accuracy (with L-BFGS linear evaluation of its features on ImageNet) (Radford et al., 2021). It is trained with constrastive learning, on the WIT dataset. Despite being a lower accuracy model, as we will see, it is a useful model.
In order to test the downstream generalization ability of these models’ learned representations, we linearly evaluate them on Pascal VOC (Everingham et al., 2010) using the same evaluation method as Kornblith et al. (2019). Pascal VOC is a multi-label image classification benchmark, with 20 diverse classes ranging from vehicles, animals, household objects and people. This diversity of scenes make it an interesting downstream task to study. Performance on this benchmark is measured by 11-point mAP. We measure it according to the definition in Everingham et al. (2010).
In Fig.7 we find that, surprisingly, the highest performing model on ImageNet (MPL) is not the best performing model on VOC. We also see that the contrastive model CLIP-S deviates from the linear trend described in Kornblith et al. (2019). Indeed, when compared with the other models, CLIP-S yields the most prediction diversity on ImageNet. We additionally test how the combined features perform by concatenating them pairwise and performing linear evaluation on top of these feature combinations – known as Stacked Generalization (Wolpert, 1992). The highest performing model combinations (on VOC) are ones which combine differently trained models. Further, the best combinations do not even include MPL, which indicates that diversity can be better indicator of downstream performance than any single model’s accuracy.
We posit that the reason diversely-trained model combinations yield highest performance on this downstream task is due to the diversity of features learned, which provide higher coverage of features captured that explain the data. This allows for better feature reuse, which is crucial for transfer learning (Neyshabur et al., 2020).

heading: 5 CONCLUSION
text: We have shown that diverse training methodologies, in particular, training with diverse optimization objectives on different large-scale datasets, can produce models that generate uncorrelated errors. These models ensemble more efficiently, attaining higher ensembling accuracy, since their different training setups allows them to specialize to different subdomains of the data. Due to this specialization, different-enough models can be useful for achieving high accuracies, even if they display low accuracies individually. Finally, we have also shown that they learn overlapping (but not supersetting) features, and that combining their embeddings can boost downstream performance.
The importance of behavior diversity has been highlighted in different fields. In signal processing, sensor fusion – the combination of signals from multiple sensors into a more accurate estimation – relies on these sensors making independent errors. In deep reinforcement learning, it has been hypothesized that maximizing the coverage of possible behaviors of an agent may help it acquire the skills that are useful (Eysenbach et al., 2018). In image classification, the model is our agent, its predictions the behavior, and ensembling our sensor fusion.
Our work demonstrates that this diversity of features is possible, and highlights a key question: why didn’t SGD learn it? Perhaps there exists an objective that can produce a single best/supersetting representation but, as we’ve shown, none of our existing training methodologies have found it.
If we look closely, our results may also provide clues for why this is the case. To find models with categorically different generalization behavior, we had to train them with objectives that do not directly maximize the classification accuracy we are ultimately interested in. In our collective search for novelty, we have stumbled upon the decade-old finding that some problems are best solved by methods that ignore the objective, since “the objective function does not necessarily reward the stepping stones in the search space that ultimately lead to the objective” (Lehman and Stanley, 2008).
Together, our results provide encouragement for machine learning practitioners to continue creating methodologies to train high-performing models, including new frameworks & datasets, in order to expand the types of features our models learn, behaviors they exhibit, and novel ways to train them.

heading: A APPENDIX
text: A.1 MODEL CATEGORIZATION
In some sense, model categories are supersets of one another: when we change a model architecture, we also change the hyper-parameters used to train such architecture, to make sure that they are optimal for training this new setting. In a similar fashion, when we change the training framework (supervised to contrastive), not only do we change hyper parameters, but the architecture also changes to best suit the new learning framework. Finally, when we change dataset scales (e.g.: ImageNet → WIT), we use the framework, architecture, and hyper parameters that allow the best performance possible on the new dataset.
A.2 MODEL LIST PER CATEGORY
In Table 1, we list all models we used in our main analysis, Figs. 1, 2, 3, 4, 6, along with their training methodologies, calibration temperatures, individual ImageNet accuracy, error inconsistency (relative to our base model ResNet-50), and ensemble accuracy (with our base model ResNet-50). We selected these models by controlling for their individual accuracy, which helps guarantee our analysis concerns training methodology, not any individual model being inherently better.
In table 2, we list the high-accuracy models used in Figs. 5 and 7. These models are higher accuracy, and provide a stronger base for ensembling.
Table 3 lists low accuracy models, but which are trained with very different methodologies (relative to the typical model). They are therefore still useful (as we show), and were used in Sec. 4.5 and Fig. 5.
Finally, Table 4 lists other models that we trained and analyzed, but which did not reach our target accuracy range 74-78%, so were not included in the main paper analysis.
A.3 ADDITIONAL ERROR ANALYSIS
In Fig. 8, we reproduce Fig. 1 (left), and additionally show how the number of examples where both models predict correct (center), as well as examples where both models predict incorrectly (right), both decrease. The added prediction diversity that comes with ensembling models trained differently will only be beneficial if the ensemble can efficiently convert the examples on the left, to compensate for the decreased number of examples in the center.
A.4 L-BFGS DETAILS
In order to train the linear classifiers, we use L-BFGS in the same setup/implementation described in Kornblith et al. (2019). We train the linear heads on top of the pre-logit representations without augmentation. We find it useful to normalize each representation before any operations (concatenation, subsampling, etc). In the first part of Section 4.6, the portions of each representation are picked randomly (e.g.: random 25% dimensions of ResNet, and random 75% dimensions of CLIP), and then concatenated. In the second part, the two representations are first concatenated then dimensions are selected based on the diversity criteria.
A.5 EXTRA FIGURES FOR REVIEWERS
0.0 0.2 0.4 0.6 0.8 1.0 Model 1 Confidence
0.0
0.2
0.4
0.6
0.8
1.0
M od
el 2
C on
fid en
ce
1.0
0.5
0.0
0.5
1.0
y-x
0.0 0.2 0.4 0.6 0.8 1.0 Model 1 Confidence
0.0
0.2
0.4
0.6
0.8
1.0
M od
el 2
C on
fid en
ce
0.75
0.50
0.25
0.00
0.25
0.50
0.75

Review summaries:
This paper empirically investigates how different models, trained with different methodologies should be ensembled to maximize the accuracy in image-net classification task. By carefully designed experiments, the authors look at different aspects of the trained models and provide guidelines for selecting models to be ensembled. The main takeaway: models trained with increased divergence in training methodologies are best suited for ensembling.

The authors conduct a large-scale study on when models are more likely to learn different representations, and how diversity in learned representations can help ensemble performance. They select 5 possible reasons for representation diversity: model reinitialization, hyperparameters, model architectures, model frameworks, and datasets. They define a measure of error correlation, and show that a pair of models with more uncorrelated errors results in higher ensemble performance - even if one or both of those models has "low accuracy". They show that models trained on different datasets are most likely to have uncorrelated errors, as opposed to the other sources of representation diversity. The authors study the effect of sampling and concatenating features from two different models, find that a mix of features from both models yields the best performance, and conclude that the models have learnt different features (yet likely overlapping). They also examine the categorical specializations between varying models, as well as the effect of training diversity for creating ensembles on downstream tasks.

This paper proposes an empirical study of how different modelling choices (eg, hyper-parameters, architecture, training algorithm, dataset) result in different and complementary data representations. The first half of the paper shows that those choices influence model predictions, leading to models that specialize to a subset of the data. The second half shows that those different representations are complementary and can be ensembled to yield better performing models. In particular, the authors show that ensembling models that are most diverse gives the largest improvement when transferring to a downstream task.