No One Representation to Rule Them All: Overlapping Features of Training Methods
Authors: Raphael Gontijo-Lopes
Yann Dauphin
Ekin Dogus Cubuk
Venue: ICLR 2022 Poster (2022)
Keywords: Representation Learning
Understanding Deep Learning
Deep Phenomena
Diversity
Novelty
Features
Training Methodologies
Contrastive Learning

Abstract: Despite being able to capture a range of features of the data, high accuracy models trained with supervision tend to make similar predictions. This seemingly implies that high-performing models share similar biases regardless of training methodology, which would limit ensembling benefits and render low-accuracy models as having little practical use. Against this backdrop, recent work has developed quite different training techniques, such as large-scale contrastive learning, yielding competitively high accuracy on generalization and robustness benchmarks. This motivates us to revisit the assumption that models necessarily learn similar functions. We conduct a large-scale empirical study of models across hyper-parameters, architectures, frameworks, and datasets. We find that model pairs that diverge more in training methodology display categorically different generalization behavior, producing increasingly uncorrelated errors. We show these models specialize in subdomains of the data, leading to higher ensemble performance: with just 2 models (each with ImageNet accuracy \~76.5\%), we can create ensembles with 83.4\% (+7\% boost). Surprisingly, we find that even significantly low-accuracy models can be used to improve high-accuracy models. Finally, we show diverging training methodology yield representations that capture overlapping (but not supersetting) feature sets which, when combined, lead to increased downstream performance.

Published as a conference paper at ICLR 2022
No ONE REPRESENTATION TO RULE THEM ALL: OVERLAPPING FEATURES OF TRAINING METHODS
Raphael Gontijo-Lopes, Yann Dauphin & Ekin D. Cubuk Google Research, Brain Team
{i fraphhel, ynd, cubuk ) @ @goog leg.ee.... com
ABSTRACT
Despite being able capture range of features of the data, high accuracy models trained with supervision tend to make similar predictions. This seemingly implies that nigh-performing models share similar biases regardless of training method- ology, which would limit ensembling benefits and render low-accuracy models as having little practical use. Against this backdrop, recent work has developed quite different training techniques, such as large-scale contrastive learning, yielding competitively high accuracy on generalization and robustness benchmarks. This motivates us revisit the assumption that models necessarily learn similar func- tions. We conduct a large-scale empirical study of models across pprem-arr,,eerss, architectures, frameworks, and datasets. We find that model pairs that diverge more in training methodology display categorically different generalization behavior, producing increasingly uncorrelated errors. We show these models specialize in subdomains of the data, leading to higher ensemble performance: with just 2 mod- els (each with ImageNet accuracy 76.5%), we can create ensembles with 83.4% (+7% boost). Surprisingly, we find that even significantly low-accuracy models can be used to improve high-accuracy models. Finally, we show diverging training methodology yield representations that capture overlapping (but not supersetting) feature sets which, when combined, lead to increased downstream performance.
INTRODUCTION
Over the years, the machine learning field has developed myriad techniques for training neural networks. In image classification, these include data augmentation, regularization, architectures, losses, pre-training schemes, and more. Such techniques have highlighted the ability of networks to capture diverse features of the data: textures/shapes (Geirhos et al. 2018), robust/non-robus features (Ilyas al. 2019), and even features that fit random, pre-determined classifier (Hoffer et 2018).
Despite this representation-llariio power, methods that yield high generalization performance seem produce networks with little behavior diversity: models make similar predictions, with high-accuracy models rarely making mistakes that low-accuracy models predict correctly (Mania al., 2019). Additionally, the quality features learned (c.g.: for downstream tasks) seems dictated by upstream performance (Kornblith et al., 2019). Finally, training on subsets of the data yields low-accuracy models that don't make performant ensembles (Nixon et al., 2020). This seemingly suggests that highhperforming models share similar biases, regardless of training methodology.
Without behavior diversity, ensemble benefits are limited to reducing noise, since models make correlated errors (Perrone & Cooper] 1992] Opitz & Maclin 1999). Without feature diversity, representations might not capture important features for downstream tasks, since feature reuse has been shown to be crucial for transfer learning (Neyshabur et al., 2020). Without knowing the effect of training methodology, one might conclude that low-accuracy models have no practical use, since their predictions would be dominated by high-accuracy ones.
One open question is if these findings faced unavoidable selection bias, since the highest- models have historically been trained with similar supervised objectives IID datasets. Up until recently, this hypothesis was difficult to test. That changed with the recent success of large-scale contrastive learning, which produces competitively-high accuracy standard generalization and robustness benchmarks (Radford et al.] 2021| Jia et al. 2021). This motivates revisiting the question:
How does training methodology affect learned representations and prediction behavior?

---

Published as a conference paper at ICLR 2022
To settle these questions. conduct a systematic empirical study of 82 models, which we train or collect, across high performing models CLIP, ALIGN, SimCLR, BiT, ViT-G/14, and MPL. In addition to using different techniques, these new models were trained on data collected very differently, allowing us to probe the effect of both training objective, as well as pre-training data. We categorize these models based on how their training methodologies diverge from a typical, base model and show:
architectures, objective functions, and datasets, including the latest
1. Model pairs that diverge more in training methodology (in order: reinitializations + hyper- parameters architectures + frameworks datasets) produce increasingly uncorrelated errors.
2. Ensemble performance increases as error correlation decreases, due to higher ensemble efficiency. The most typical ImageNet model (ResNet-50,, 76.5%), and its most different counterpart -ALIGN-ZS, 75.5%) yield 83.4% accuracy when ensembled, a +7% boost.
3. Contrastively-learred models display categorically different generalization behavior, special- izing in subdomains of the data, which explains the higher ensembling efficiency. We show CLIP-S specializes in antropogenic images. whereas ResNet-50 excels in nature images.
4. Surprisingly, we find that low- -cc-uccuryy models can be useful if they are trained differently enough. By combining high-accuracy model (BiT-Ik, 82.9%) with only low-accuracy models (max individual acc. 77.4%), we can create ensembles that yield as much as 86.7%.
5. Diverging training methodology yield representations that capture overlapping (but not super- setting) feature sets which, when concatenated, lead to increased downstream performance (91.4% on Pascal VOC, using models with max individual accuracy 90.7%).
2 RELATED WORK
Diversity in Ensembles. It is widely understood that good ensembles are made of models that are both accurate and make independent errors (Perrone & Cooper 1992 Opitz & Maclin 1999: Wen
ett al. 2020). Beyond improving ensemble performance, finding diverse solutions that equally well explain the observations can help quantify model uncertainty (also known epistemic uncertainty)
what the model does not know because training data was not appropriate (Kendall & Gal 2017
Fort et al. 2019). Many works have explored ways of finding such solutions (Izmailov et al. 201
Boostrapping (Freund et al., 1996) (ensembling models trained on subsets of the data) was found not to produce deep ensembles with higher accuracy than single model trained on the entire dataset (Nixon et al. 2020). Another work has examined the effect of ingmentation-inducee prediction diversity on adversarial robustness (Liu et al., 2019). More relevant to us, Wenzel et al. (2020) and and architectures respectively.
Zaidi et al. (2021) have explored the effect of random finding best ensembles when combining diverse models, albeit still considering similar frameworks.
Model Behavior Similarity. These attempts were hindered as many high-performing techniques seem to produce similar prediction behavior. Mania et al. (2019) demonstrates, via "dominance probabilities", that high-accuracy models rarely make mistakes that low-accuracy models predict correctly. This indicates that, within the models studied, high-accuracy models "dominate" the predictions of low-accuracy ones. Recht al. (2019) shows that correlated with in-distribution performance. Relatedly, Kornblith et al. (2019) shows that upstream and downstream performance are very correlated. These jointly indicate that high-accuracy models learn strictly better representations, diminishing the importance of low-accuracy solutions (even if they are diverse). Finally, Fort al. (2019) shows that aubspace-sampling methods for ensembling generate solutions that, while different in weight space, remain similar in function space, which gives rise to an insufficiently diverse set of predictions.
robustness seems
Contrastive-earrnin Models; Different Large-Scale Datasets. This model behavior similarity might be explained by the fact that the training techniques that yield high performance on image classification tasks have been relatively similar, mostly relying on supervised learning on ImageNet, optionally pre-training on dataset with similar distribution. Recently, various works have demon- strated the effectiveness of learning from large-scale data using contrastive learning (Radford et al.) 2021 Jia et al. 202T). They report impressive results on out-of-distribbn di-dstributiin benchmarks, and have been shown to have higher dominance probabilities (Andreassen et al.) 2021). These represent some of the first models to deviate from standard supervised training (or finetuning) on downstream data, while still yielding competitive accuracy. They add to the set of high-performing training techniques,
2

---

Published as a conference paper at ICLR 2022
which include data augmentation (Cubuk al.
2018 2020), regularization (Srivastava al. 2014
Szegedy et al. [0166] Ghiasi et al. 2018). architectures (Tn & Le 2019 Dosovitskiy et al. 2020 Hu
et al. [018]] landola la et al. [014 Li al.
2019
Szegedy et al. 2016: Simonyan & Zisserman 2014
Sandler et al. al.]|] 2018), losses (Chen et al.
2020; Radford et al. 2021 Jia et al. (021). pre-training
schemes (Kolesnikov al. 2020; Pham et al. 2021). I) and provide the motivation for revisiting the
question of whether training methodology can yield different model behavior.
3 METHOD
3.1 MODEL CATEGORIZATION
In order to evaluate the performance of learned representations as a function of training methodology. we define the following categories, which classify model pairs based on their training differences:
1. Reinits: identical models, just different in reinitialization.
2. Hyper-parameters: models of the same architecture trained with different hyper parameters (e.g.: weight decay, learning rate, initialization scheme, etc).
3. Architectures: models with different architectures, but still trained within the same frame- work and dataset (c.g.: ResNet and ViT, both with ImageNet supervision).
4. Frameworks:
models trained with different optimization objectives, but same dataset (c.g.: ResNet and SimCLR, respectively supervised and contrastive learning on ImageNet).
5. Datasets:
models trained on large-scale data (e.g.: CLIP or BiT trained on WIT or JFT).
In some sense, model categories can be supersets of one another: when we change a model archi- tecture, we may also change the hyper-parametent used to train such architecture, make sure that they are optimal for training this new setting. Unless stated otherwise, all ensembles are comprised of fixed base model, and another model belonging to one of the categories above. This way, each category is defined relative to the base model: model pairs in a given category will vary because Model 2 is different than the base model along that axis. The result is that as navigate along model categories (Reinit + ... + Dataset), we will naturally be measuring the effect of increasingly dissimilar training methodology. See Appendix Table for details.
3.2 MODEL SELECTION
We collect representations and predictions for 82 models, across the many categories above. We fix ResNet-50, trained with RandAugment, as our base model. ResNet a good candidate for a base model since is one of the most typical ImageNet classification models, and the de-facto standard baseline for this task. In total, we train/collect models in the categories: I) Reinit; 2) Hyper- parameters (51): varying dropout, dropblock, learning rate, and weight decay, sometimes jointly; 3) Architectures (17): including EfficientNet, ViT, DenseNet, VGG; 4) Framework (2): including SimCLR, and models trained with distillation; and 5) Dataset (12): including CLIP, ALIGN, BiT, and more, trained on WIT (Radford et al.) 2021), the ALIGN dataset, JFT (Sun et al., (017). etc. We additionally collect high-performing models MPL, ALIGN (L-BFGS), ViT-G/14, BiT-Ik, CLIP-L, EfffciienNNetB33 for some of our analyses. These are some of the latest, highest-perfomme models for ImageNet classification.
We found it necessary to calibrate all models using temperature scaling (Roelof' et al. 2020 Guo ctt al. 2017) to maximize ensemble performance. Finally, unless stated otherwise, we only use models in the same narrow accuracy range (74-78% accuracy on ImageNet), which guarantees that the effects observed are indeed function of diverging training methodology, and not of any given model being intrinsically more performant than another. complete list of models be found in the Appendix.
3.3 ENSEMBLING
In our paper, we ensembling tool for understanding: such, our goal is not to find methods to ensemble the highest accuracy models and reach state-of ttate-of-the-art. Instead, we wish to use ensembling probe whether/when training methodology yields uncorrelated (and therefore useful) predictions.
3

---

Published as a conference paper at ICLR 2022
Model
Correct
Incorrect
0.22
Error Inconsistenc
Model 2 is Correct Incorrect
0.20
0.18
0.16
0.14
0.12
0.10
Observed Inconsistency
As training methodology diverges (Reinit Dataset), errors become uncorrelated. Left: 'Observed error inconsistency" is the fraction examples where only one model in the pair makes correct prediction. Higher error inconsistency indicates the model errors uncorrelated. Right: As models are trained with increasingly different methodologies, their error inconsistency grows, providing opportunity for converting these examples into correct ensemble predictions.
Figure 1:
RESULTS
In order to understand whether model similarity reported in literature varies as a function training methodology, evaluate error correlation by measuring the number of test-set examples where one model predicts the correct class, and the other predicts an incorrect class. We call this Error Inconsistency, as it complements the consistency measure, defined in Geirhos al. (2020).
We choose error inconsistency to measure error correlation because it allows us to connect the ensemble prediction diversity (which are interested in quantifying) directly into performance. That is, when errors are consistent (i.e.: both models make a correct prediction, or both models make an incorrect prediction), this agreement directly translates into higher/lower aceuracy after ensembling, When errors are inconsistent (i.e.: when one model is correct and another is incorrect), the ensemble prediction will be determined by the models' confidences. a model is sufficiently confident, its prediction will "winn" the ensembling procedure. If this model's prediction was correct, we say that this prediction was "converted" into correct ensemble prediction.
4.1 As TRAINING METHODOLOGY DIVERGES, ERRORS BECOME MORE UNCORRELATED
In Fig. we see that, as we compare increasingly different training methodologies ("Reinit" + "Dataset"), " error inconsistency increases the number of examples where only one model makes correct prediction. This indicates that as training methodology diverges, model predictions become dissimilar and, result, enrors become uncorrelated. As the framework and dataset categories be orthogonal, we note that models with the most error inconsistency tend to modify both.
This also represents opportunity for these uncorrelated mistakes to be converted into correct prediction, when we ensemble the models. Such an opportunity will be beneficial if the conversion rate of these examples is higher than the decrease in number of examples where both models made correct predictions. As we will see in the following section, this indeed happens.
4.2
AS UNCORRELATED ERRORS INCREASE, DOES ENSEMBLE EFFICIENCY
In order to assess whether these increased uncorrelated errors can be beneficial, we measure how models in the various categories ensemble. That is, we ensemble our base model with models the various categories, and measure the ensemble's 's accuracy, relative to its member models.
Per Fig. 2 as uncorrelated errors increase, so does ensemble performance (left). Additionally, because we restrict our models to ones within the narrow accuracy range 74-78%, we guarantee that this relative improvement translates into absolute improvement (center), and not due to any individual ensemble member being intrinsically better than another. Relative to our base model ResNet-50 (76.5% on ImageNet). the most differently-traie -trinne model ALIGN- ZeroShot (75.5%) yields 83.4% top-1 accuracy when ensembled, boost of nearly 7% accuracy.
4

---

Published as a conference paper at ICLR 2022
Ensemble Accuracy (%)
(Rel, to Model Average
8
Ensemble Accuracy (%)
(Rel. to Model Max)
IPARAN!
Conversion Rate (%)
AACH FRAHEWORK
75
DATASET
4-
REINT MPARAM ARCH FRANIMORK DATASET
65
HPARAM
RENET
2
ARCH
FRANTWORK
60
DATAGET
0.15
0.20
9.25
0.15
0.20
0.2
9.30
0.20
Error Inconsistency
Error Inconsistency
Error Inconsistency
Figure 2: As uncorrelated errors increase, does ensemble performance (left, center), and so does ensemble efficiency (right). Left: Error inconsistency is linearly correlated with the ensemble performance improvement, relative to the mean accuracy of the models in the ensemble. Stars represent averages over models in each category (w/ error bars). Center: Because we limited our analysis to models in the same 74-78% accuracy range, the increase in relative accuracy translates into absolute accuracy boost, with best performing ensembles comprising of models whose training methodologies are most different. Right: Surprisingly, the conversion rate the rate at which these examples are converted into correct predictions by the ensemble also increases. This indicates that the benefits of combining divergently-traiie models go beyond increasing the number of examples that can become correct predictions, to also increasing how efficiently these examples do become correct predictions.
Taken together, these results mean that combining models with diverse training methodologies can yield performance improvements that offset the decrease in examples where both models are correct. To explain this relative boost, we also measure the conversion rate of each ensemble the rate at which inconsistent errors are converted into correct ensemble predictions. Surprisingly, this rate also increases as training methods diverge (Fig. 2 right). This means that different training methods not only yield more opportunities for ensemble benefits, but also more efficient ensembles.
4.3 DISSIMILAR TRAINING METHODOLOGIES CREATE SPECIALIZED MODELS
To understand how diverging training setups create efficient ensembles, we measure the specialization of ensemble member models. We first note that, for model's prediction to "win" the ensembling process, only needs to be sufficiently more confident than the other model's 's prediction. This means that measure specialization by the relative confidence of two models. We do this by measuring the angle distance 0 in the confidence-confidence plot (see Fig. top left). When 0 is high, Model more confident (relative Model 2), and vice-versa.
In Fig- we use 0 to understand the specialization of different models categories. To simplify our analysis, we compare three different ensembles, from the categories Reinit (Model ResNet-50), Architecture (Efficien(NettB33), and Dataset (CLIP-S), as they are representative of the spread of error correlation. When we plot a histogram of examples as a function of we find that: As observed in Fig. when training methods diverge, the models make mistakes in different sets of examples 2) As training methods diverge, Model tends to be more confident in examples where only Model 1 is correct, and vice-versa for Model 2. This effect is most striking when we look at the Dataset category, where Model (CLIP-S) is significantly more confident than Model (ResNet-50) in examples where only CLIP-S predicts correctly, and vice-versa. This is in contrast with Reinit, where models don't 't seem to be significantly more confident than each other at all.
These results show: training methodology diverges, model specialize to different data subdomains.
4.4
MODEL SPECIALIZATION DEPENDS ON TRAINING METHODOLOGY
Next, we want to investigate what kind of data each model category specializes to. In Fig. we plot the same data as Fig. 3 but we divide the examples along their ImageNet class IDs. This allows us to inspect whether given model is more/less specialized in given class.
As before, we find that, while models in the Architecture category demonstrate higher specialization (Model tends be more confident in examples where it is correct, and vice versa for Model 2),
5

---

Published as a conference paper at ICLR 2022
1.0
Both Models Correct
Nelther Model Correct
400
09:461 Conrdarrs
100
0.8
+
80
300
0.4
100
0.2
20
6.25
0.00
0.0
0.2
Model1 1. Confidence
0.4
3.6
0.8
1.0
Confidant
Conficent
Confidbont
0
Confident
Only
Only Correct
Medel 1's top prediction always
100
prediction might win
depencing logits
80
daw might onother
100
ospensling on other loghts
40
Model 3h prediction wwwyy conversbon
50
20
0.75 0.50 -0.25
0.00
0.25
0.50
0.75
-0.75
-0.00
0225
0.50
0.25
0.50
0.75
DATASET
Model
Confident
Conticent
Modal
Model
Eontidenn
Conf dent
Figure Differently-trained models specialize: We plot histograms of examples where least one ensemble produced error inconsistency, as a function of specialization measure 9, the angle distance the confidence-confidenne plot (upper left). As we saw Fig. I when model training setups diverge (Reinit + Arch > Dataset), the fraction of consistent errors decreases (upper center & right), in favor of more error inconsistency (lower center & right). This added error inconsistency comes with specialization of the models in ensemble: when only Model makes correct prediction, it is often more confident (lower right), and vice-versa for Model 2 (lower center). Faint dotted lines indicate values of & for which model's top-1 prediction is likely prevail ensemble time.
REINIT
ARCH
DATASET
Modelt
500
500
250
250
250
0-
0.5
0.5 Corf dent
0.0
Model2 Confident
Modd1 Contident
θ
Model
Reshwidt0 CLP
Condident
Confideent
Zonfident
1000
1000
1000
Moded Cornect
750
750
500
0.5
0.5
0.0
Model 2
0
Model 1
Model
0 Model
Model
Model
ResNet
Figure 4: Specialization type depends on training setup: When models differ in dataset (right plot), not only is specialization highest (see Fig. 3 but also this specialization happens in different classes CLIP is better at anthropogenic classes (cids 500-900) than ResNet-50, which is better at nature classes (cids 0-300; Right detail). When models differ in their architecture (Center plot), they are more specialized than reinitializations (Left plot), but such specialization does not correlate with specific classes.
this specialization does not seem to correlate with specific classes. In contrast, not only do models of the Dataset category display more specialization, but this specialization seems to be correlated with groups of classes. In particular, CLIP-S seems to specialize to anthropogenic images, whereas ResNet-50 seems specialize nature images. We suspect this phenomena is result of CLIP-S being trained on dataset that was collected independently, and much differently than the one used to train base model, ResNet-50.

---

Published as a conference paper at ICLR 2022
Ersemble Accuracy
ALIO 041 PMG
DUP1L 2511
Model
Nethod
Model
Nethod
Ensemble Accurscy
Dff 182.7%
CUPP (85.
Cont. /ITT
SEfteshiett-
(27.6%)
Sup. /
85.7%
TF nocetton A01
18ftcali and
Ineefton- 1D (17,5%)
(84. 7%)
Incopdien
AL (75.5%)
bit
Bffkienshee-BB 833
(80 9%)
Sup. IN
ALIGN-25 (75.5%)
/ FT
24.8%
Number of Models in Ensemble
VIT 4016 (74,9))
CLP 175.7%
(82 9%)
SJF.
ALIGN (75, (5..56)
Conk.
83.2%
Figure 5: Lower-accuracy models can benefit high-aceuracy ensembles: Left: Starting with high-accuracy models (colored markers; CLIP-L, ALIGN, BiT-Ik, EfficientNet-B3), we greedily select the best lower accuracy models (cach with max individual accuracy 77.9%. indicated in the legend) to ensemble, and plot the ensemble's s accuracy. Colors indicate which high-accuracy model the ensemble begins with, shapes indicate which models are added. By adding only lower-accuracy models to a high-accuracy one, are able to create ensembles that reach high as 86.7%. This shows that lower-accuracy models can be made useful, if they are diverse enough. Right: In each case, the best first model to add is one that complements the base model's training methodology.
4.5 DIFFERENT ENOUGH NOWER-ACCURAS MODELS CAN IMPROVE ACCURACY
In classic machine learning, common way to create nigh-performing ensembles create special- ized models with bootstrapping (Freund et al. 1996), where models are trained on subsets of the data and subsequently ensembled. In deep learning, bootstrapping has not been found work well, since training on IID subsets of the data (with similar methodology) does not seem to yield ensembles that perform better than an individual model trained on the entire dataset (Nixon et al. 2020). This seems to indicate that lower-accuracy models would have little practical benefit for performance.
In order to investigate this, and encouraged by the finding that differently-trainen models can be experts in subdomains of the data, we ask whether lower-accuracy models can be beneficial. After all, some of the models in the Dataset category were trained on much larger-scale datasets (JFT, WIT), which are collected independently.
To do this, we first combine our base model, ResNet-50 (76.2%), with a lower-accurayy model trained very differently, CLIP- S-.P----ZeroSott (63.3%, "Dataset" category), and observe performance boost (77.7%). To push this idea further, we combine a high-accuracy model (BiT-1k, 82.85%) with only lower-accuracy models (max individual accuracy 77. 77.44%), by greedily selecting ensemble members that maximize accuracy. With this procedure, we can create ensembles that yield as much as 86.66%. In Fig. [5] we repeat this procedure for 3 additional high-accuracy models and find that lower-accuracy models can consistently improve the accuracy of high-accuracy models. Importantly, the low-accuracy models need to be different enough in their training methodology (in terms of Sec. 3. categories: for a given high-accuracy model, the most beneficial lower- model to ensemble is one trained with different loss and/or dataset, often both. See table in Fig. 5 for details.
This result calls into question the idea that the information learned by high accuracy models is strict superset of those learned by low accuracy models, which we explore further in sections below.
4.6 SPECIALIZATION COMES FROM OVERLAPPING (BUT NOT SUPERSETTING) REPRESENTATIONS
In order to explain why differently-traine models specialize, we posit that training methodology changes the features learned by individual models. In this view, models trained differently would have an overlapping (but not supersetting) set of features which, when combined, provide more information for classification than any individual model's representation could.
We test this directly by concatenating the features of our base model (ResNet-50) with those of the models above (ResNet-50,, EfficientNet-BB -O and CLIP-S), and linearly evaluating these combined features ImageNet classification. More specifically, we first randomly select features from the base model and each other model at inversely proportional rates. For example, we concatenate 25% of ResNet-50 features with 75% of CLIP-S S features, yielding a final embedding that is at most the same dimensionality of the ResNet-50 features. This guarantees that any performance boost is not due to higher number of dimensions being available, but by the quality of the features represented.
7

---

Published as a conference paper at ICLR 2022
Recket
CLI
REINT ARCH DATASET
0.775
0.750
0.725
RIITT
HRARAM
ARCII
DATASET
Frall Embeeding
50.50
25775
1050
30:00
4000
Base//ther Emrbedo Used
Represemtallon mimaiin
afterd wwerddvrsttpg..eee comporasioon
Figure 6: Specialization comes from overlapping (not supersetting) representations. When we combine the representations of two models different rates (Left, 25/75 using 25% of the ResNet embedding and 75% of the other model's embedding). we find that: 1) despite being smaller number of dimensions, the concatenated embeddings (Center, e.g.: 50/50) yield higher LBFGS accuracy than the complete embedding of any of the two models (100/0 or 0/100). This indicates that each model has learned overlapping. but not supersetting, features, which when combined maximize performance. 2) when we combine embeddings of models that are more differently -triined (Center, "Reinit" + "Dataset"), we find bigger gains, indicating that divergent training induces the learning of different features. When we compress the concatenated representations using a diversity heuristic, we find that dufferently-traineed embeddings are also more efficiently compressible, reaching higher accuracies with fewer dimensions (Right, see text for details). See Appendix for L-BFGS implementation details.
In Fig. 6 (center), we see that the best performance is obtained when features are combined. Addi- tionally, find that combining features yields higher performance training methodology diverges, with the best performing combination being ResNet-50 + CLIP-S. Concurrent work similarly finds that different training objectives yield different final-layer feature embeddings (Grigg et al.) (021). This seems to confirm the idea that the methods used to train networks can generate diverse features, which capture information that neither embedding alone has captured.
To push this further, we ask how efficiently these representations capture important features of the data. To test this, first compute the covariance of each dimension of embeddings from both models (e.g.: ResNet-50 and CLIP-S). This gives us ranking of highest to lowest covariance dimensions, which we use as a measure of diversity. We then select features in order of most diverse to least diverse, and linearly evaluate them on ImageNet classification.
Fig. (right) shows how divergently- pressible) features. We tested multiples of 256 features, and found that Reinit models needed 2304 features on average to achieve 76% accuracy. In constrast, Hyperparameter models required 2057, Architecture 1392, and Dataset only required 512 features.
models yield more diverse (and therefore more com-
4.7 DOWNSTREAM TASK PERFORMANCE
With the knowledge that diverse training leads to diverse feature embeddings, we ask whether these transfer better downstream tasks. In order to maximize performance, pick three of the highest accuracy models on ImageNet, that are also representative of very diverse training setups. Meta-Pseudo I Labels (MPL. the previous SOTA on ImageNet, reporting 90.24% top- accuracy (Pham et al., (021). It is trained with a teacher network that generate pseudo labels on unlabeled data to teach a student network, whose performance on the labeled set is then used adapt the teacher. It is trained on the JFT-300M dataset. ViT-G/14 is the current SOTA, with 90.45% top-1 accuracy (Zhai al. 2021), when measured with EMA (Polyak & Juditsky 1992). We obtained our image embeddings without the use of EMA, so we find effective accuracy of 88 93% It is trained supervisedly, on JFT-3B, a larger version of the JFT-300M dataset. Finally, CLIP-S is a constrastive learning model trained on the WIT dataset, yielding 75.7 75.7% top-1 accuracy (with L-BFGS linear evaluation of its features on ImageNet) (Radford et al. 202Tp. Despite being a lower accuracy model, as we will see, it is useful model.
In order to test the downstream generalization ability of these models' learned representations, we linearly evaluate them on Pascal VOC (Everingham al. 2010) using the same evaluation method as Korbllith et al. (2019). Pascal VOC is a multi-label image classification benchmark, with 20 diverse classes ranging from vehicles, animals, household objects and people. This diversity of scenes make it an interesting downstream task to study. Performance on this benchmark is measured by 11 -piit mAP Everingham al. 2010).
8

---

Published as a conference paper at ICLR 2022
Pascal Voc 11 11-gaint mAp
92
0.20
Pascal VOc. 11 1--nt mAP
1.5
(Stacked Generalization
90-
VIT-G/14
Error Inconsistoncy
(ImageNeti
0.190
0.191
1.0
91.36
(L-8FGS)
0.15
0.5
MPL
G/14
88-
CLIP
MPL
0.0
0.10
0,080
-0.5
CLIP-S + CLIP-S
MPL + ViT-G/14
86
-1.0
0.05
89.74
84
-1.5
75
8.
90
VIT-G/14
MPL
VIT-G/14 CLIP-S
Cancatenated
ImageNet Accuracy
MPL
CLIP-S
Representations
Figure 7: Diverse training methodologies yield the best downstream performance. Left: The highest-aceuracy ImageNet model (MPL) is not the best pest-performing on Pascal VOC. Center: CLIP-S the model with the most prediction diversity, among the models analyzed. Right: Combining models that are most diverse seems to yield the biggest boost when performing stacked generalization on a downstream task (i.c.: training a linear layer on top of 2 models' concatenated embeddings).
In Fig we find that, surprisingly, the highest performing model on ImageNet (MPL) is not the best performing model on VOC. We also see that the contrastive model CLIP-S deviates from the linear trend described in Kornblith al. (2019). This is surprising since this linear trend previously reported with really high correlation. Indeed, when compared with the other models, CLIP-S yields the most prediction diversity ImageNet. We additionally test how the combined features perform by concatenating them pairwise and performing linear evaluation on top of these feature combinations knnww as Stacked Generalization (Wolpert) 1992). The highest performing model combinations (on VOC) are ones which combine differently trained models. Further, the best combinations do not even include MPL, which indicates that diversity can be better indicator of downstream performance than any single model's 's accuracy.
We posit that the reason diversely-trainee model combinations yield highest performance on this downstream task is due to the diversity of features learned, which provide higher coverage of features captured that explain the data. This allows for better feature reuse, which crucial for transfer learning (Neyshabur et al. 2020).
5 CONCLUSION
have shown that diverse training methodologies, in particular, training with diverse optimization objectives on different large-scale datasets, can produce models that generate uncorrelated errors. These models ensemble more efficiently, attaining higher ensembling accuracy, since their different training setups allows them to specialize to different subdomains of the data. Due to this specialization, different-enough models can be useful for achieving high accuracies, even if they display low accuracies individually. Finally, we have also shown that they learn overlapping (but not supersetting) features, and that combining their embeddings can boost downstream performance.
The importance of behavior diversity has been highlighted in different fields. In signal processing, sensor fusion the combination of signals from multiple sensors into a more accurate estimation relies on these sensors making independent errors. In deep reinforcement learning, it has been hypothesized that maximizing the coverage of possible behaviors of an agent may help it acquire the skills that are useful (Eysenbach al.) 2018). In image classification, the model is our agent, its predictions the behavior, and ensembling our sensor fusion.
Our work demonstrates that this diversity of features is possible, and highlights key question: why didn't 't SGD learn it? Perhaps there exists an objective that can produce a single best/ beptssupess representation but, as we' 'v shown, none our existing training methodologies have found
If we look closely, our results may also provide clues for why this is the case. To find models with categorically different generalization behavior, we had train them with objectives that do not directly maximize the classification accuracy are ultimately interested in. In our collective search for novelty, we have stumbled upon the decade-old finding that some problems are best solved by methods that ignore the objective, since "the objective function does necessarily reward the stepping stones in the search space that ultimately lead to the objective" (Lehman and Stanley 2008).
Together, results provide encouragement for machine learning practitioners to continue creating methodologics to train high gigh-perfoming models, including new frameworks & datasets, in order to expand the types of features our models learn, behaviors they exhibit, and novel ways train them.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
We would like thank the following people for their early feedback, thoughtful discussions, and help over the course of this project: Becca Roelofs, Ben Poole, Simon Kornblith, Niki Parmar, Ben Caine, Keren Gu, Ethan Dyer, Anders Andreassen, Katherine Heller, Chiyuan Zhang, Jon Shlens, Tyler Scott, Gamaleldin Elsayed, Rosanne Liu, Dan Hendrycks, Samy Bengio. Hieu Pham, Thomas Unterthiner, Chao Jia, Mostafa Dehghani, Neil Houlsby, Lucas Beyer, Josip Djolonga, Rodolphe Jenatton, Benham Neyshabur, Alec Radford, and Sam Altman.
REFERENCES
Samira Abnar, Mostafa Dehghani, Behnam Neyshabur, and Hanie Sedghi. Exploring the limits of large scale pre-training. arXiv preprint arXiv:2110.02095, 2021.
Anders Andreassen, Yasaman Bahri, Behnam Neyshabur, and Rebecca Roelofs. The evolution of out-of-distribution robustness throughout fine-tuning, arXiv preprint arXiv: 108.51,, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for
contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Yunpeng Chen, Jianan Li, Huaxin Xiao, Xiaojie Jin, Shuicheng Yan, and Jiashi Feng. Dual path networks. arXiv preprint arXiv: 1707701677 2017.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation policies from data. arXiv preprint (0X.:::800009000 2018.
Ekin Cubuk, Barret Zoph, Jonathon Shlens, and Quoe Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition Workshops, pp. 702-703, 2020.
Alexander D' Amour, Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Fisenstein, Matthew D) Hoffman, et al. Underspecification presents challenges for credibility in modern machine learning. arXiv preprint arXiv:2011.03395, 2020.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 11929, 2020.
Mark Everingham, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International, journal of computer vision, 88(2): 303-338, 2010.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv: 1.3i::18000000000 2018.
Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep ensembles: A loss landscape perspec- tive. arXiv preprint arXiv: /91.9//2..777 2019.
Yoav Freund, Robert E Schapire, et al. Experiments with new boosting algorithm. In icml, volume 96, pp. 148-156. Citeseer, 1996.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained enns are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv preprint arXiv: 11111111..2222 12231, 2018.
Robert Geirhos, Kristof Meding, and Felix A Wichmann. Beyond accuracy: Quantifying trial-by-trial behaviour of cnns and humans by measuring error consistency. arXiv preprint arXiv:2006. /6736. 2020.
Golnaz Ghiasi, Tsung- Yi Lin, and Quoc V Le. Dropblock: A regularization method for convolutional networks. arXiv preprint arXiv: 1810. /2890, 2018.
10

---

Published as a conference paper at ICLR 2022
Tom George Grigg, Dan Busbridge, Jason Ramapuram, and Russ Webb. Do self- self-supervised and supervised methods learn similar visual representations?, 2021.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International Conference Machine Learning, pp. 1321-1330. PMLR, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Elad Hoffer, Itay Hubara, and Daniel Soudry, Fix your classifier: the marginal value of training the last weight layer. arXiv preprint arXiv:1801.04544 2018.
Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132-7141, 2018.
Forrest landola, Matt Moskewicz, Sergey Karayev, Ross Girshick, Trevor Darrell, and Kurt Keutzer. Densenet: Implementing efficient convnet descriptor pyramids. arXiv preprint arXiv: /404. / 1869, 2014.
Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry, Adversarial examples not bugs, they are features. arXiv preprint S3Kiv:/905.0022 2019.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson, Av- eraging weights leads to wider optima and better generalization. arXiv preprint arXiv: /00.000070 2018.
Chao Jia, Yinfci Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hicu Pham, Quoc V Le, Yunhsuan Sung. Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. arXiv preprint 1rXiv:2102.05918, 2021.
Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? arXiv preprint arXiv.1703.04977 2017.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. Computer ECCV 2020: 16th European Conference, Glusgow, UK, August 23-28, 2020, Proceedings, Part V 16, pp. 491-507. Springer, 2020.
Simon Kornblith, Jonathon Shlens, and Quoe V Le. Do better imagenet models transfer better? In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 2661-2671, 2019.
Anders Krogh and John Hertz. simple weight decay can improve generalization. In Advances in neural information processing systems, pp. 950-957, 1992.
Alexey Kurakin, Ian Goodfellow, Samy Bengio, Yinpeng Dong, Fangzhou Liao, Ming Liang, Tianyu Pang, Jun Zhu, Xiaolin Hu, Cihang Xie, et al. Adversarial attacks and defences competition. In The NIPS 17 Competition: Building Intelligent Systems, pp. 195-231. Springer, 2018.
Youngwan Lee and Jongyoul Park. Centermask: Real-time anchor-free instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp-
13906-1399 13915, 2020.
Joel Lehman and Kenneth O Stanley. Exploiting open-endedness to solve problems through the search for novelty. In ALIFE, pp. 329-336. Citeseer, 2008
Xiang Li, Wenhai Wang, Xiaolin Hu, and Jian Yang. Selective kernel networks. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 510-519, 2019.
Ling Liu, Wenqi Wei, Ka-Ho Chow, Margaret Loper, Emre Gursoy, Stacey Truex, and Yanzhao Wu. Deep neural network ensembles against deception: Ensemble diversity, accuracy and robustness. In 2019 IEEE 16th international conference on mobile ad hoc and sensor systems (MASS), pp. 282. IEBE, 2019.
11

---

Published as a conference paper at ICLR 2022
Horia Mania, John Miller, Ludwig Schmidt, Moritz Hardt, and Benjamin Recht. Model similarity mitigates test set overuse. arXiv preprint arXiv:190S.125800 2019.
Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer learning? arXiv preprint arXiv:2008. 2020.
Jeremy Nixon, Balaji Lakshminarayanan, and Dustin Tran. Why are bootstrapped deep ensembles not better? In "I Can't I Believe It's Not Better!"Neurie NNurlPS 2020 workshop, 2020.
David Opitz. and Richard Maclin. Popular ensemble methods: An empirical study. Journal of artificial intelligence research, 11:169-198, 1999.
Michael P Perrone and Leon N Cooper. When networks disagree: Ensemble methods for hybrid neural networks. Technical report, BROWN UNIV PROVIDENCE RI INST FOR BRAIN AND NEURAL SYSTEMS, 1992.
Hieu Pham, Zihang Dai, Qizhe Xie, and Quoc V Le. Meta pseudo labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11557-11566 2021.
Boris T Polyak and Anatoli B Juditsky. Acceleration of stochastic approximation by averaging. SIAM journal on control and optimization, 30(44)838-855) 1992.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint 1rXiv:2103.0002: 2021.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning, pp. 5389-5400. PMI.,, 2019.
Rebecca Roelofs, Nicholas Cain, Jonathon Shlens, and Michael C Mozer. Mitigating bias in calibration eπror estimation. arXiv preprint arXiv:2012.08666 2020.
Abhijit Guha Roy, Jie Ren, Shekoofeh Azizi, Aaron Loh, Vivek Natarajan, Basil Mustafa, Nick Pawlowski, Jan Freyberg. Yuan Liu, Zach Beaver, et al. Does your dermatology classifier know what it doesn't know? detecting the long-tail of unseen conditions. Medical Imuge Analysis, 75: 102274, 2022.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Chieh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 4510-4520, 2018
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 1rXiv::449.111 2014.
Samarth Sinha, Homanga Bharadhwaj. Anirudh Goyal, Hugo Larochelle, Animesh Garg, and Florian Shkurti. Diversity inducing information bottleneck in model ensembles. arXiv preprint arXiv:2003.04514, 2020.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56):1929-199 1958, 2014. URL http: wttp://tttpppppp////////s//
srivastaval4e.htmll
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In Proceedings of the IEEE international conference on computer vision, pp. 843- 033, 2017.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 28 18-2826. 2016.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, pp. 6105-61 14, PMLR, 2019.
12

---

Published as a conference paper at ICLR 2022
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference Machine Learning, pp. 10347-10357. PMLR, 2021.
Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng. Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, et al. Deep high-resolution representation learning for visual recognition. IEEE transactions on pattern analysis and machine intelligence. 2020.
Yeming Wen, Dustin Tran, and Jimmy Ba. Batchensemble: an alternative approach to efficient ensemble and lifelong learning, arXiv preprini arXiv:2002.06715, 2020.
Florian Wenzel, Jasper Snoek, Dustin Tran, and Rodolphe Jenatton. Hyperparameter ensembles for robustness and uncertainty quantification. arXiv preprint urxiv:2006.13570) 2020.
David H Wolpert. Stacked generalization. Neural networks, 5(2))241-255) 1992.
Fisher Yu, Dequan Wang, Evan Shelhamer, and Trevor Darrell. Deep layer aggregation. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pp. 2403-2412, 2018.
Sheheryar Zaidi, Arber Zela, Thomas Elsken, Chris C Holmes, Frank Hutter, and Yee Teh. Neural ensemble search for uncertainty estimation and dataset shift. Advances Neural Information Processing Systems, 34, 2021.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. arXiv preprint arXiv:2106.04566 2021.
13

---

Published as a conference paper at ICLR 2022
A APPENDIX
A.1 CONCURRENT WORK
Many concurrent and recent papers have found related conclusions to Here, highlight few of them. Abnar et al. (202T) found that when training on the same objective, as we increase the upstream accuracy, the performance of downstream tasks saturates. This is in line with results in Sec. 4.7 that optimizing for a single objective might not capture diverse enough features for downstream performance. D D''Amuur al. (2020) show that, result of underspecification in training, models are treated as equivalent based on their training domain performance, but can behave very differently in deployment domains.
Additionally, other recent work also finds that optimizing for diversity is beneficial: Roy et al. (2022) found that using a diverse ensemble (mixing losses and datasets) improves OOD performance in dermatology tasks. Siinha et al. (2020) explicitly optimizes for diversity of predictions using an adversarial loss, leading improved OOD performance.
A.2 MODEL CATEGORIZATION
In some sense, model categories supersets of one another: when change model architecture, we also change the hyper-parameters used to train such architecture, to make sure that they are optimal for training this new setting. In similar fashion, when we change the training framework (supervised to contrastive), not only do we change hyper parameters, but the architecture also changes best suit the new learning framework. Finally, when we change dataset scales (e.g.: ImageNet > WIT), we use the framework, architecture, and hyper parameters that allow the best performance possible on the new dataset.
A.3 MODEL LIST PER CATEGORY
In Table we list all models used in main analysis, Figs. I 2 along with their training methodologies, calibration temperatures, individual ImageNet accuracy, error inconsistency (relative our base model ResNet-50), and ensemble accuracy (with our base model ResNet-50). We selected these models by controlling for their individual accuracy, which helps guarantee our analysis concerns training methodology, not any individual model being inherently better.
In table 2 we list the high-accuracy models used in Figs, 5 and These models are higher accuracy, and provide stronger base for ensembling.
Table 3 lists low accuracy models, but which are trained with very different methodologies (relative to the typical model). They are therefore still useful (as we show), and were used in Sec. 4.5 and Fig.
Finally, Table listts other models that we trained and analyzed, but which did not reach our target accuracy range 74-78%, so were not included in the main paper analysis.
A.4 ADDITIONAL ERROR ANALYSIS
In Fig. reproduce Fig. T (left), and additionally show how the number of examples where both models predict correct (center), as well as examples where both models predict incorrectly (right), both decrease. The added prediction diversity that comes with ensembling models trained differently will only be beneficial the ensemble can efficiently convert the examples on the left, to compensate for the decreased number of examples in the center.
A.5 L-BFGS DETAILS
In order to train the linear classifiers, we use L-BFGS in the same setup/imppeementtiion described in Korblith et al. (001)). We train the linear heads on top of the pre-logit representations without augmentation. We find useful to normalize each representation before any operations (concatenation, subsampling, etc). In the first part Section 4.6, the portions of each representation are picked randomly (e.g.: random 25% dimensions of ResNet, and random 75% dimensions of CLIP), and then
14

---

Published as a conference paper at ICLR 2022
Model
Ciasion
Mechod
Prediction Elead Type
Calibration Temp
ImageNett
Acc
Inconsistency ResNes-50)
Error
Calegury
(w/ ResNet
Ensemblle Aoc
Lc at 2016
Sup. INN
Traindd
1.10
76.27%
01650
REINIT
77.31%
10
IN
Triiee
100
74 27%
0.0072
HPARAM
76 73%
ResNer-50 (Droout (11)
Srivastonn
Sup.
Triined
1.10
76 90%
0.053
HPARAM
77.23%
ResNet-50 (Dropont 0.2)
Sup. IN
Traised
1.10
75
HPARAM
ResNer-50 (Depout 0.3)
Sup IN
Thined
1.10
75 431
0.0100
HPARAM
77. 17.125
ResNer.90 (Dropont (14)
IN
Triined
1.10
75. 37%
0.0928
HPARAM
77.12% 12%
ResNet50 (Droput
1.10
75.00%
0.965
IIPARAM
ResNer-90 (Drepont 0.6)
Sup IN
Triiedd
1:10
74 74-7
0.1006
HPARAM
77.015
ResNer-90 (Dropont 0.7)
Sup. IN
Triined
1.10
74 48-4
0.1058
HPARAM
76.80%
ResNett50 (Label Smoothing 0.1)
Sougedy 2016
IN
Triied
0.90
76 2214
0.0877
IIPARAM
77.30%
ResNer-50 (Jabeel Smonthing
IN
Triiedd
0.80
74%
00911
HPARAM
ResNet-50 (1.beel Smoothing 0.3)
Sup. IN
Truised
0.80
75. 75%744%
HPARAM
77.35 77.665
ResNet-50 500ee-500220 (Label Smoothing 0.4)
Sup IN
Trained
0.70
75. 709
0.0996
HIAARAM
77.31%
ResNerr90 90 (1 abel Smoothing 0.5)
IN
Trined
0.70
75. 51
0.1030
HPARAM
77. 36.36
ResNet-S0
Smuothing
IN
0.70
75.00%
0.1063
ResNet-50
Smoothing (.7)
Sup IN
Triied
0.70
74. 54%
0.1071
HPAAAA
76 30%
ResNerr90 (1 abel Smoothing 0.8)
Sup. IN
Trined
0.80
74 7855
0.1130
HPARAM
76 9899
ResNiet 50 (DnopBlock 34, 0.9)
Ghiasi 2018
IN
Trinnd
1.00
74.90 74.98%
0.8891
[IPARAM
76 76.84%
ResNet50 (DropBlock
0.9)
IN
1.00
87%
0.885
HPAAAM
ResNer-90 zaming Rote 0.05)
Sup
Trined
1:10
75. 5696
HPARAM
17.105
ResNiet-5 50 (1 (Leoming Rale 0.2)
Sup. IN
Traied
1.10
76. 26.07%
0.0658
IIAARAM
77. 77.35%
ResNiet-50 50 (Weight Decay 0 0.00001>
Krogh & Herty 1997 1992
IN
Triiel
1.20
74. 13%
0.1027
HPARAM
76 70%
ResNer-50 (Weight Decay 0,0000))
Sup. IN
Triined
1:10
0.0666
HPARAM
ResNeet-0 50 (Weight Decay 0,0002)
Sup. IN
Traindd
1.10
75.79%
0.065
IIPARAM
7666
ResNler-0 (LR 0.05, WD 0.0002)
Sup IN
Trined
1:10
75 58%
0.0666
HPARAM
77.08 08.9
ResNiet 50 (LR 0.2. WD 0,00005)
IN
Trained
1.10
76.36%
0.0644
EIAARAM
77.49%
ResNee-S0 (LR
0,0000)
IN
Traindd
1.20
75.71%
0,1011
IIPARAM
EtticsemNeet-D BB
Tas &
2019
Sup. 7IN
Inaned
0.90
70.38%
0.1136
ARCH
75.05%
SK-Re:Net.34
IN
Trained
0.90
76.99 76.91%%
0.109
ARCH
78.5256
MobileNet-V2
Sup.
Triined
0.90
0.1033
ARCH
78.48%
VGG11 RatchNorm
Sup IIN
Trined
1:10
235
ARCH
77.45%
Legacy-EEReSNat.k
Hi 21
2314
Sup. IN
Trained
1.10
74.81%
0.1206
ARCH
77.70%
Legacy St.ResNett50
2313
Sup. IN
Trained
0.90
77 64%
0.1144
ARCH
79.11%
SFResNexXi256
Triieed
0.90
60%
0.1187
ARCH
BenseNet-BBuu-122d
Sup. IN
Traised
0.90
76.59%
0.1082
ARCH
DesseNet-122
1ib33a 0 31
3014
Sup. IN
Trained
0.30
75.5 75.57%
0.1076
ARCH
77.70%
Incoption-VS
Sauncaly
33114
IN
Trined
1.00
77 44%%
0.1270
ARCH
79 49%
V3-Eocegtion--3
Sup.
Trained
1.00
86-6
0.1259
ARCH
03.5
Adv Inception-V3
Kurakmn
41
Sup IN
Trined
1.00
77. 58%
0.1252
ARCH
79.41%
HRNNr-WI--Smill
Waln 21 20
70121
Sup. IN
Tniied
1.00
75. 13%
0.1143
ARCH
77.78%
DPN-68
Lis
201
Sup. IN
Trained
1,20
76.31%
0.3141
ARCH
78.44%6
Trined
02.
0.1064
ARCH
75.35%
ENN-WWNNer
122 &
21120
Sup. IN
Trined
0
76 80%
ARCH
VIT-B/16
2ra1
2020
Sup. INN
Trained
1.70
74 74%55%
0.1472
ARCH
78.85%
SumCLR
20A1
Cont. /IN
Tuned
100
75 30%
0.1277
FWORK
75.35%
VIT Tiny
2021
Sup.
Distilld
1.00
14.50%
0.1247
VIT-3/16
2020
Sup. 7 /N--11
Trined
0.90
77 86%
0.1249
DATASET
79.73%
ALIN--S
Iir
Over, / ALLGN
Zem Shot
0.30
75. 50%
0,2295
DATASET
83.425
CLIP-S
2001
Cont, WTT
L-BFGS
0.90
75.67%
0.1687
DATASET
CLIP-L-ZS
Ridilond al
A001
Cont, / WIT
Zero Shot
1.10
76.57%
0.3140
DATASET
33.229%
Table 1: Models Used. These models are all within the 74-78% accuracy range, of which we base our main results. "Sup." indicates supervised learning, "Cont." " constrastive learning. "IN" indicates ImageNet.
Model
Cilstion
Method
Prediction Head Type
Calibration Temp
ImageNNt
Eruor Inconsistency ResNet-50)
Calegory
Eiseemlle Acc ResNes-50
FBiiianNNe-B3
Tin 1e
3019
Sup. IN
Trainod
0.90
80.89%
0.1147
ARCH
80.98%
ALIGN
2021
Constmasive ALIGN Datnset Sup.
L-BGGS
1.00
84.71%
0.1684
DATASET
85.50%
NIT-H/14
2020
Trainod
DATASET
87.48%
VIT-G/14
Zhan ot
Sup.
Trained
1.40
88.9
DATASET
88.40%
MPL
Phin of
202
Pscudo-Lahde Label/ IFT
Trained
0.90
90.24%
0.1741
DATASET
88.97%
CLIP-I
Kalloal
1
Contractive / WIT
L-BPGS
1.00
85.04%
0,1638
DATASET
85, 85.36%
R/T-IK
Sup.
Fine Taned
82.85%
0.1593
DATASET
84.28%
Table High Accuracy Models. These models are all above the 80% accuracy. "Sup." " indicates supervised learning, "Cont." constrastive learning. "IN" indicates ImageNet.
Model
Citation
Method
Prediction Head Type
Calibration Tomi
ImageNet Ace
Eruor Inconsistency (wi ResNeet- SKD)
Calegory
Ensemble (w/ ResNer-50)
CLIP-S-SS
Radfodd ot 3021
Cont. / WIT Sup. /FF
Zaro Shot
0.01
63. 25%25%
0.2675
DATASET
78.01%
20200
Class Mapping
0.02
68.63%
0.2744
Table Low Accuracy Models. These models are all below the 74% target accuracy, but they are trained with very different methodologies (relative to typical models), so we include them in a few analyses, as indicated main text. "Sup." indicates supervised learning, "Cont." " constrastive learning. "Class Mapping" indicates that the model's original JFT class vector was used, and mapping of ImT-tto-ImageNet classes was used to obtain the final prediction logits.
15

---

Published as a conference paper at ICLR 2022
Model
Method
Head Type
Prediction
ImageNett
Acc
Calegoey
ResNet-101
Sup.
75.4%%
HARAAM
ResNei.3002
Sup./
Trained
78.70%
HPARAM
ReoNet: 50 (Dropout 0.8)
Sup.
Trained
73.68%
IPARAM
ResNet-50 (Dropout
Sup. /IN
Traindd
72.48%
IPARAM
ResNct:50 Sanoohing
Sup.
ResNet:30 ResNe:330 0.1)
Sup.
Trained
29.58%
IPARAM
ResNet-50 (DroptBlack 34, 0.2)
Sup. INN
Trained
50.17%
HPAKAM
ResNet: (DropBBlock
Sup.
Trained
HPARAM
ResNet:30 (DropBBlock
Sup.
ResNet.30 [Droptilack 34, 0.5)
Sup./
Trained
66.19 19.1
HPARAM
ResNet 50 (DropBilack 34, 0.6)
Sup.
Traind
68.23%
HPARAM
ResNet-50 (DropBllock 34,
Sup./ IN
Traisd
70.95%
IPARAM
ResNet-50 (DropBBlack
Sup.
ResNer:30 ResNes:30.DtooBBleeek 1234, 0.13
Sup.
Trained
27.19%
HPARAN
ResNet-50 (DroptBlack 1234, 0.2)
Sup. IIN
Traind
47.27%
HPARAM
ResNet-30 (DroppBlock
Sup. IN
Trained
55.59%
HPARAM
ResNet:50 (DtopBBlcck
ResNet:50 [DropBlack 1234, 0.5)
Sup. /IN
Traised
64.57%
HPARAM
KesNet-50 (DroptBlack 1234, 0.6)
Sup. IN
Traindd
07.08%
HPAKAM
ResNet: 30 (DropBlock 1234,
Sup.
Traisdd
70.35%
HPARAM
ResNet-50 (DkopBlack
Stp./IN
ResNet.30 [Learning Kate (.01)
Sup./IN
Trained
69.24%
HPARAM
ResNet: 30 [Learning Rate 1.0)
Sup.
Traind
73.53%
HPARAM
ResNes-50 (Weight 0.001)
Sup./
Traised
ResNei-30 0.I, 0.001)
Sup.
HPARAM
Table 4: Other Models. These accuracy range, we did not use them in our main analyses. "Sup." indicates supervised learning, "IN" indicates ImageNet.
sweeps were trained, but did not fit our target 74-78%
concatenated. In the second part, the two represeniations are first concatenated then dimensions are selected based on the diversity criteria.
A.6 EXTRA FIGURES FOR REVIEWERS
Only One Model Correct
Examples Where
0.22
Examples Where
Both Models Coffect
0.72
% Examples Where Nelther Model Correct
0.70
0.20
0.08
0.38
0.66
0.64
0.62
0.12
0.00
0.58
RBIIT HP4NAM ARCH FRAME DATASET
REINT HAARAM ARCH
FRAME DATASET
KENIT HPARAM ARCH FRANEDATÁSET
Figure 8: As training methodology diverges, errors become uncorrelated. The number of test examples where only model predicts correctly increases (Left,
"Error Inconsistency" in Fig. D). Similarly, the number of examples where both models predict correctly (Center) and neither model predicts correctly (Right) decreases. For the most diverse training methodology ("Dataset", purple), we find more examples with inconsistent errors (left) than examples where both models predict correctly (right). This means that for an ensemble to perform better than another, it needs to resolve these inconsistent errors efficiently, to compensate for the decrease in the examples in the center plot.
16

---

Published as a conference paper at ICLR 2022
Model 2 Confidence
1.0
1.0
Model 2 Confidence
10
6.6
65
0.
6.6
6.
0.25
0.0
0.00
0.4
04
-025
62-
-05
02
0.0
04
52
Model Confidence
Model Confidence
Madels
Nelthar
500
200
-3.5
0.5
-0.5
0.0
Confident
Confident
Contidintt
Contidem::
Only Model Correct
Only Model Correct
n0
40
40-
20
20-
-0.5
Confident
Model
y-x
Confident
Modell
Model 2 Confident
Y-X
Confiient
Modwl
Figure 9: Justification for 0. The white dotted lines represent the thresholds where example's ensemble prediction will correspond to the nighee--ooffdence-moon measures distances in way that aligns with these thresholds, allowing top-1 prediction. Theta visualize the number of correctly-classified examples at a glance (by looking at the number of examples to the right/left of the dotted lines in Fig 3). Because y-x does align with this threshold, it's harder to visualize the effect of specialization on performance, as can be seen by the histograms.
Neither Correct
Neither Correct
Neither Correct
0.07
7000
7000
NNmm Converied Exomales
6000
Num Total Examples
6000
0.05
5000
5000
Convorsion
4000
4000
0.03
3000
3000
2000
2000
REINIT
ARCH
DATA
REINIT
ARCH
DATA
REINT
ARCH
Figure 10: Conversion rate of examples where "neither model" is correct individually. We find that, as training methodology diverges, the conversion rate of these examples increases: Reinit (94 converted out of 7154 examples), Architecture (190 6246). Dataset (355 5192). This is consistent with our finding that different training methodologies create more efficient ensembles. We note that despite the higher conversion rate, bigger effect performance is in the decrease of examples where neither is correct (7154 - 6246 - 5192).
17

---

Published as a conference paper at ICLR 2022
Ensembiee Accuracy
0.80
REIIT ARCH
Ensemble Accuracy
0.00
RERVIT
ARCH
DATA
6.70
DATA
0.70
6.76
0.76
00
022
0.
Weighted Average Parameter
Weighted Average Parameter
(Simple interpolation)
(Class- aware Interpolation)
We interpolate between the two models in the ensemble using single parameter, and find that the optimal interpolating parameter seems be at or near 0.5 (i.c.: it's just as good simply average the logits with equal weight). We suspect this is because the calibration performed on the models before ensembling guarantees that the confidences are appropriately scaled already. Right: order further investigate whether specialization can yield better-weighted ensembles, we also perform class-aware interpolation, where t=0 means nature classes get their logits from ResNet (and human classes get their logits from the other model), and vice versa for t=1. We find that for the ResNet+CLIP ensemble, t < 0.5 is optimal, which consistenct with our finding of ResNet/CLIP specialization. We note however that the boost is marginal (80.424% -> 80.452%, less than 15 images).
Figure 11:
Interpolating between ensemble members. Left:
Only SimCLR Correct
Only CUP-S Correct
100
150
100
60-
40
50
2..75-0.5500.25
SImCLR Confidert
0
CLP-S
SImCLR Confiden:
CLP Confidert
Corrident
CLI-S Correct
SimCLI Cornect
1000
1000
Irageeet Clan
400
400
-0.50 -0.25 0.50
9.25
6.50
0.75
-0.75 -0.0 -0.25
0.75
Contderr
ConfMtrt
Coliiont
Conficet
Figure 12: Specialization of CLIP-S and SimCLR. We find that these two models seem to also specialize in a way that is aligned with anthropogenichnature classes (cids 500-900, and 0-300 respectively). We suspect the distribution of CLIP's pre-training dataset has huge effect in making this happen. We also stress that these two models are still very distinct categories: when comparing against each other (and not against our base model), must note that they different in reinit, architecture, params, and dataset, even if framework kept the same.
18

---

Published as a conference paper at ICLR 2022
Error Consistency
08-
0.-
02
Figure 13: CKA vs Error Consistency. Here, CKA is measured between the two pre-logit representations of ResNet, and each of the models ResNet, CLIP-S-ZS, SimCLR, EfficientNet. V/T-B/16 and 121 CKA and Error Consistency are not well correlated, likely due to CKA being sensitive the geometry of representations.
19


Reviews:
[{"review_id": "pCAFp_J78CG", "reviewer": "Reviewer_yy7C", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is well-written and interesting, and the empirical analysis is impressive. The authors present their empirical observations as the main contribution, but I think this actually undersells the contributions -- this paper is also a tour de force in methodologies about how to evaluate and compare different models. Section 4 is very well-written and is a golden example of strong experimental design/scientific method; each sub-section presents a concrete hypothesis, an experiment to test the hypothesis, and discussion of empirical results. I wish every paper were written this well!\n\nMy only complaint is that some of the language is imprecise. As a concrete example, the phrase \"different enough\" or \"diverse enough\" appears several times throughout the manuscript. This phrase can be more precise (What does it mean to be \"different enough?\" Different architecture? Different dataset?). The results speak for themselves by the end of section 4, but the takeaways could be more precise for the reader (note that this may take more analysis/experiments -- but I'm sure the authors can knock that out of the park).\n\nI also think the paper could be improved by including some more intuition or examples of the types of divergences between different models. For example, what types of examples does CLIP get wrong but SimCLR gets right? This may be unwieldy for all 82 models, but a few spot instances for each framework would be very interesting and useful.", "summary_of_the_paper": "This paper presents an extensive empirical evaluation of what models learn when trained with different architectures, frameworks, and datasets. The authors discuss the effects of training methodology on the types of errors that models make, and show that ensembling models trained with diverse training methodologies can result in surprisingly strong performance (even when the models in the ensemble themselves are relatively weak).", "summary_of_the_review": "This paper is interesting and well-written, and it presents an impressive empirical evaluation of how training methodology affects representations and predictions. The paper does not claim novel methods, but the methods for empirical analysis themselves are interesting, and the community can learn a lot from them.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "hcCFpHh7MdL", "reviewer": "Reviewer_iemd", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nThe paper is well-written, and has clearly very thorough experiments. Notably, they include 82 different models (across their 5 categories of training diversity), which include recent high-performing works.\n\nEach point in their 5 listed contributions at the end of the introduction is interesting and valuable for future research (although I have some concerns listed below). Each point provides evidence for the authors' concluding statement, that no single training methodology should dominate the field, but rather research effort on a diverse set of approaches should continue. They also raise a very interesting question for theory as well as empirical studies - why doesn't SGD learn the more diverse representations that are shown to perform better?\n\nOne part of \"well-written\" - the authors repeat key definitions throughout the work, instead of defining them once, which I found helpful (e.g. repeating the definition of \"conversion rate\" in the Figure 2 caption).\n\nWeaknesses:\n\nUncorrelated errors seem like an intuitive metric to analyze representation diversity - however, there are other existing metrics which I imagine measure this more directly, such as mutual information. Why not measure representation diversity in that manner?\n\nI am also not sure why the authors chose to use the angle distance on the confidence-confidence plot in their analysis on model specialization. It works, but seems a little odd - why not just subtract one model confidence from another?\n\nIn Table 3 in the Appendix, the authors seem to define \"low accuracy\" models as those below the 74% accuracy (on ImageNet) threshold they use in their main analysis. However, Section 4.5 considers the 74%-78% range as \"low accuracy\" - which is it? In the current form, the claim \"low accuracy models can benefit high-accuracy ensembles\" is misleading.\n\nA small clarification question - why is the dataset star in Figure 2 marked with an additional cross? To highlight the best category? If so, that would be useful to mention (assuming I didn't miss the explanation somewhere).", "summary_of_the_paper": "The authors conduct a large-scale study on when models are more likely to learn different representations, and how diversity in learned representations can help ensemble performance. They select 5 possible reasons for representation diversity: model reinitialization, hyperparameters, model architectures, model frameworks, and datasets. They define a measure of error correlation, and show that a pair of models with more uncorrelated errors results in higher ensemble performance - even if one or both of those models has \"low accuracy\". They show that models trained on different datasets are most likely to have uncorrelated errors, as opposed to the other sources of representation diversity. The authors study the effect of sampling and concatenating features from two different models, find that a mix of features from both models yields the best performance, and conclude that the models have learnt different features (yet likely overlapping). They also examine the categorical specializations between varying models, as well as the effect of training diversity for creating ensembles on downstream tasks.", "summary_of_the_review": "The authors present a thorough evaluation into the effect of various differences in training procedures on resulting representation diversity. I have not specialized in this research area and cannot speak well to the novelty of each claim, but believe that the paper deserves acceptance due to the extensive experiments, clear presentation, and interesting findings which open questions for future research.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "e6xkqF9QA2C", "reviewer": "Reviewer_7RNN", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I find this paper well-motivated, with a clear focus on understanding whether different training methodologies learn similar representations, or not. The flow of the paper is well written, with each section addressing a one line of inquiry and naturally leading to the next. The (many) experiments are well designed, and the results clearly presented and interpreted. I especially enjoyed the first set of results (Figure 1), where the authors show that increasingly different training methodologies yield increasingly different representations — those results should be of interest to the ICLR community.\n\nI list below some questions and concerns, in no particular order.\n\n- If possible, please move Table 1 from the Appendix to the main text. I did not understand the experimental setup in Section 3.1 until I found this table.\n- On p. 3, the paper introduces categories of possible changes to their base learning setup (supervised learning of Resnet-50 on ImageNet), and argues that some changes are more important than others: eg, changing the architecture to ViT is more important than doubling the weight-decay coefficient. Could the authors justify why changing the dataset is more important than changing the learning framework? It seems reasonable the procedure to infer model parameters (ie, the framework) can be as or more important than the choice of datasets, provided those datasets are large enough. If that's the case, we'd expect error inconsistency to be higher for \"framework\" than for \"dataset\" in Table 1, thus more diverse models are obtained by changing the framework (not the dataset). This begs the question: how much of the subsequent analysis depends on \"dataset\" being more important than \"framework\"? Would we see that better ensembles are built from models trained on the same (large enough) dataset but with different frameworks?\n- The authors already provide experiments under many different conditions (see Table 1), and I do not wish to ask for more. Still, I question:\n    - Why not include a Resnet-50 model when changing the framework and the dataset? This way, one could see how much improvement is expected when changing framework / dataset but the architecture is kept constant.\n    - Are the (only) 2 different conditions enough to draw conclusions on the effect of changing the framework? One could imagine also including generative (auto-encoders, flows, etc) or few-shot training (protonet, metaoptnet, etc) as well.\n- One of the main weaknesses of the paper is the lack of driving hypothesis. The experiments show that different training conditions yield different representations (and that those representations become specialized, yield better ensembles), but we're never told why. The second-to-last paragraph (p. 9) is not illuminating in that regard.\n- Finally, there are a few moments of carelessness in the manuscript that need to be polished. For example, citations for WIT and JFT are never given; the \"LBFGS Accuracy\" of Section 4.6 is never properly defined (only mentioned in passing in Section 4.7); the section on bootstrapping (Section 4.5) doesn't discuss Freund and Schapire's work on weak learners; \"Stochastic Weight Averaging\" (Izmailov et al., UAI 2018) is never mentioned although it is a successful method to ensemble deep networks.\n\nSome typos:\n\n- p. 2: Use \\citep for Andreassen et al. (2021)?\n- p. 3: Is it 82 models since those in Table 4 are not included (looks more like 55-ish in Tables 1 and 2)?\n- p. 4: \"the the ensemble prediction\".\n- p. 8: \"imagenet\" not capitalized.\n- p. 9: \"CLIP-S is yields the most prediction diversity\".\n- p. 9: \"any single models' accuracy\" → \"model's\".\n- p. 11: The reference is just \"Lehman & Stanley\", no \"et al.\".", "summary_of_the_paper": "This paper proposes an empirical study of how different modelling choices (eg, hyper-parameters, architecture, training algorithm, dataset) result in different and complementary data representations. The first half of the paper shows that those choices influence model predictions, leading to models that specialize to a subset of the data. The second half shows that those different representations are complementary and can be ensembled to yield better performing models. In particular, the authors show that ensembling models that are most diverse gives the largest improvement when transferring to a downstream task.", "summary_of_the_review": "Overall I remain positive about this paper and vote for acceptance. My main concerns are points 2 and 3 from the main review.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Kkqc-_xp_jy", "reviewer": "Reviewer_ZeBH", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I find this paper very insightful and believe that the investigations done here would be useful for the community.  Ensembling is a prevalent way of achieving desired accuracies in practice and this study provides helpful pointers to do that effectively. I have following questions for the authors though:\n\n- It might just be me, but how is the training methodology \"Frameworks\" is incorporated in this work? I could not find any details or explanation about it.\n- In the frameworks section, SimCLR is used. This model is trained with a self-supervised objective with no prediction head for labels. How do you attain prediction confidences on a test set for such models? or how do you incorporate predictions for such a model?\n- Based on the previous question, is it possible to use different self-supervised or even unsupervised methods?\n- The dataset difference is a part of the study, but is it fair to compare the accuracies on image-net for the models which were not trained on image-net? Wouldn't the methods which aren't trained on image-net suffer from domain generalization problem?\n- In CLIP-S method, it is unclear to me whether it's the training dataset that introduces a huge divergence in the methodology or the contrastive objective. I think there could be a sub-categorization of the training methodology \"Dataset\" into \"optimization objectives\" and \"datasets\" for a more controlled study. I will expand on this in the following point.\n- Figure 4, demonstrates some clear differences in the specialization of CLIP-S and resnet-50. Since the models are trained on different datasets and then tested on image-net. I wonder how much important the bias of the training dataset is. More importantly, what will happen if the ResNet is trained on the [somewhat similar] dataset used for training CLIP-S and tested on image-net? would the variation in architecture and frameworks still matter?\n- In the light of the above, supervised contrastive learning [1] might be a useful model to try as it uses contrastive training objective and is trained on image-net.\n- Have you quantified the conversion rate where both ensembling models predict incorrectly. I wonder if the ensembled prediction confidences might make the incorrect prediction correct. It will be interesting to see how various training methodologies influence that?\n- Is there an opportunity for weighted ensembling? Based on the analysis of training methodology can one further improve the ensembling by rescaling prediction confidences of certain models for certain test datasets? One can probably use specialization criteria to do that?\n- In section 4.6, do you train a linear classifier (via L-BFGS) on top of concatenated representations? How do you pick the portions of each representation to be concatenated? is it selected randomly?\n- In section 4.6 (later half), do you select diverse features from both models at the same time and concatenate these features or you do that separately for both models?\n\n[1] [https://arxiv.org/abs/2004.11362](https://arxiv.org/abs/2004.11362)", "summary_of_the_paper": "This paper empirically investigates how different models, trained with different methodologies should be ensembled to maximize the accuracy in image-net classification task. By carefully designed experiments, the authors look at different aspects of the trained models and provide guidelines for selecting models to be ensembled. The main takeaway: models trained with increased divergence in training methodologies are best suited for ensembling.", "summary_of_the_review": "I liked the overall study and believe that it will be useful for the community. I would appreciate it if the authors could address my concerns/ questions above.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This well written and well motivated paper has been independently reviewed by four expert reviewers. They all voted for the acceptance with three straight accepts and one marginal. The feedback provided to authors was constructive and the authors responded comprehensively. I recommend acceptance of this work for ICLR.
Decision: Accept (Poster)