No One Representation to Rule Them All: Overlapping Features of Training Methods

1 INTRODUCTION . Over the years , the machine learning field has developed myriad techniques for training neural networks . In image classification , these include data augmentation , regularization , architectures , losses , pre-training schemes , and more . Such techniques have highlighted the ability of networks to capture diverse features of the data : textures/shapes ( Geirhos et al. , 2018 ) , robust/non-robust features ( Ilyas et al. , 2019 ) , and even features that fit a random , pre-determined classifier ( Hoffer et al. , 2018 ) . Despite this representation-learning power , methods that yield high generalization performance seem to produce networks with little behavior diversity : models make similar predictions , with high-accuracy models rarely making mistakes that low-accuracy models predict correctly ( Mania et al. , 2019 ) . Additionally , the quality of features learned ( e.g . : for downstream tasks ) seems dictated by upstream performance ( Kornblith et al. , 2019 ) . Finally , training on subsets of the data yields low-accuracy models that don ’ t make performant ensembles ( Nixon et al. , 2020 ) . This seemingly suggests that high-performing models share similar biases , regardless of training methodology . Without behavior diversity , ensemble benefits are limited to reducing noise , since models make correlated errors ( Perrone & Cooper , 1992 ; Opitz & Maclin , 1999 ) . Without feature diversity , representations might not capture important features for downstream tasks , since feature reuse has been shown to be crucial for transfer learning ( Neyshabur et al. , 2020 ) . Without knowing the effect of training methodology , one might conclude that low-accuracy models have no practical use , since their predictions would be dominated by high-accuracy ones . One open question is if these findings faced unavoidable selection bias , since the highest-performing models have historically been trained with similar supervised objectives on IID datasets . Up until recently , this hypothesis was difficult to test . That changed with the recent success of large-scale contrastive learning , which produces competitively-high accuracy on standard generalization and robustness benchmarks ( Radford et al. , 2021 ; Jia et al. , 2021 ) . This motivates revisiting the question : How does training methodology affect learned representations and prediction behavior ? In this paper , we conduct a systematic empirical study of 82 models , which we train or collect , across hyper-parameters , architectures , objective functions , and datasets , including the latest high performing models CLIP , ALIGN , SimCLR , BiT , ViT-G/14 , and MPL . In addition to using different techniques , these new models were trained on data collected very differently , allowing us to probe the effect of both training objective , as well as pre-training data . We categorize these models based on how their training methodologies diverge from a typical , base model and show : 1 . Model pairs that diverge more in training methodology ( reinitializations , hyper-parameters , architectures , frameworks , → dataset scales ) produce increasingly uncorrelated errors . 2 . Ensemble performance increases as error correlation decreases , due to higher ensemble efficiency . The most typical ImageNet model ( ResNet-50 , 76.5 % ) , and its most different counterpart ( ALIGN-ZS , 75.5 % ) yield 83.4 % accuracy when ensembled , a +7 % boost . 3 . Contrastively-learned models display categorically different generalization behavior , specializing in subdomains of the data , which explains the higher ensembling efficiency . We show CLIP-S specializes in antropogenic images , whereas ResNet-50 excells in nature images . 4 . Surprisingly , we find that low-accuracy models can be useful if they are trained differently enough . By combining a high-accuracy model ( BiT-1k , 82.9 % ) with only low-accuracy models ( max individual acc . 77.4 % ) , we can create ensembles that yield as much as 86.7 % . 5 . Diverging training methodology yield representations that capture overlapping ( but not supersetting ) feature sets which , when combined , lead to increased downstream performance ( 91.4 % on Pascal VOC , using models with max individual accuracy 90.7 % ) . 2 RELATED WORK . Diversity in Ensembles . It is widely understood that good ensembles are made of models that are both accurate and make independent errors ( Perrone & Cooper , 1992 ; Opitz & Maclin , 1999 ; Wen et al. , 2020 ) . Beyond improving ensemble performance , finding diverse solutions that equally well explain the observations can help quantify model uncertainty ( also known as epistemic uncertainty ) – what the model does not know because training data was not appropriate ( Kendall & Gal , 2017 ; Fort et al. , 2019 ) . Many works have explored ways of finding such solutions ( Izmailov et al. , 2018 ) . Boostrapping ( Freund et al. , 1996 ) ( ensembling models trained on subsets of the data ) was found not to produce deep ensembles with higher accuracy than a single model trained on the entire dataset ( Nixon et al. , 2020 ) . This emphasizes how much data deep neural networks need to achieve high performance . Another work has examined the effect of augmentation-induced prediction diversity on adversarial robustness ( Liu et al. , 2019 ) . More relevant to us , Wenzel et al . ( 2020 ) has explored the effect of random hyper-parameters , finding best ensembles when combining models that are both hyperparameter and weight-diverse , albeit still considering similar frameworks and architectures . Model Behavior Similarity . These attempts were hindered as many high-performing techniques seem to produce similar prediction behavior . Mania et al . ( 2019 ) demonstrates , via “ dominance probabilities ” , that high-accuracy models rarely make mistakes that low-accuracy models predict correctly . This indicates that , within the models studied , high-accuracy models “ dominate ” the predictions of low-accuracy ones . Recht et al . ( 2019 ) shows that out-of-distribution robustness seems correlated with in-distribution performance . Relatedly , Kornblith et al . ( 2019 ) shows that upstream and downstream performance are very correlated . These jointly indicate that high-accuracy models learn strictly better representations , diminishing the importance of low-accuracy solutions ( even if they are diverse ) . Finally , Fort et al . ( 2019 ) shows that subspace-sampling methods for ensembling generate solutions that , while different in weight space , remain similar in function space , which gives rise to an insufficiently diverse set of predictions . Contrastive-Learning Models ; Different Large-Scale Datasets . This model behavior similarity might be explained by the fact that the training techniques that yield high performance on image classification tasks have been relatively similar , mostly relying on supervised learning on ImageNet , optionally pre-training on a dataset with similar distribution . Recently , various works have demonstrated the effectiveness of learning from large-scale data using contrastive learning ( Radford et al. , 2021 ; Jia et al. , 2021 ) . They report impressive results on out-of-distribution benchmarks , and have been shown to have higher dominance probabilities ( Andreassen et al. , 2021 ) . These represent some of the first models to deviate from standard supervised training ( or finetuning ) on downstream data , while still yielding competitive accuracy . They add to the set of high-performing training techniques ( which include data augmentation ( Cubuk et al. , 2018 ; 2020 ) , regularization ( Srivastava et al. , 2014 ; Szegedy et al. , 2016 ; Ghiasi et al. , 2018 ) , architectures ( Tan & Le , 2019 ; Dosovitskiy et al. , 2020 ; Hu et al. , 2018 ; Iandola et al. , 2014 ; Li et al. , 2019 ; Szegedy et al. , 2016 ; Simonyan & Zisserman , 2014 ; Sandler et al. , 2018 ) , losses ( Chen et al. , 2020 ; Radford et al. , 2021 ; Jia et al. , 2021 ) , pre-training schemes ( Kolesnikov et al. , 2020 ; Pham et al. , 2021 ) , etc ) , and provide the motivation for revisiting the question of whether training methodology can yield different model behavior . 3 METHOD . 3.1 MODEL CATEGORIZATION . In order to evaluate the performance of learned representations as a function of training methodology , we define the following categories , which classify model pairs based on their training differences : 1 . Reinits : identical models , just different in reinitialization . 2 . Hyper-parameters : models of the same architecture trained with different hyper parameters ( e.g . : weight decay , learning rate , initialization algorithm , etc ) . 3 . Architectures : models with different architectures , but still trained within the same frame- work and dataset ( e.g . : ResNet and ViT , both with ImageNet supervision ) . 4 . Frameworks : models trained with different optimization objectives , but on same dataset ( e.g . : ResNet and SimCLR , respectively supervised and contrastive learning on ImageNet ) . 5 . Datasets : models trained on large-scale data ( e.g . : CLIP or BiT – trained on WIT or JFT ) . In some sense , model categories can be supersets of one another : when we change a model architecture , we may also change the hyper-parameters used to train such architecture , to make sure that they are optimal for training this new setting . Unless stated otherwise , all ensembles are comprised of a fixed base model , and another model belonging to one of the categories above . This way , each category is defined relative to the base model : model pairs in a given category will vary because Model 2 is different than the base model along that axis . The result is that as we navigate along model categories ( Reinit → ... → Dataset ) , we will naturally be measuring the effect of increasingly dissimilar training methodology . See Appendix Table 1 for details . 3.2 MODEL SELECTION . We collect representations and predictions for 82 models , across the many categories above . We fix ResNet-50 , trained with RandAugment , as our base model . ResNet is a good candidate for a base model since it is one of the most typical ImageNet classification models , and the de-facto standard baseline for this task . In total , we train/collect models in the categories : 1 ) Reinit ; 2 ) Hyperparameters ( 51 ) : varying dropout , dropblock , learning rate , and weight decay , sometimes jointly ; 3 ) Architectures ( 17 ) : including EfficientNet , ViT , DenseNet , VGG ; 4 ) Framework ( 2 ) : including SimCLR , and models trained with distillation ; and 5 ) Dataset ( 12 ) : including CLIP , ALIGN , BiT , and more , trained on WIT ( Radford et al. , 2021 ) , the ALIGN dataset , JFT ( Sun et al. , 2017 ) , etc . We additionally collect high-performing models MPL , ALIGN ( L-BFGS ) , ViT-G/14 , BiT-1k , CLIP-L , EfficientNet-B3 for some of our analyses . These are some of the latest , highest-performing models for ImageNet classification . We found it necessary to calibrate all models using temperature scaling ( Roelofs et al. , 2020 ; Guo et al. , 2017 ) to maximize ensemble performance . Finally , unless stated otherwise , we only use models in the same narrow accuracy range ( 74-78 % accuracy on ImageNet ) , which guarantees that the effects observed are indeed a function of diverging training methodology , and not of any given model being intrinsically more performant than another . A complete list of models can be found in the Appendix .

Summary: This paper proposes an empirical study of how different modelling choices (eg, hyper-parameters, architecture, training algorithm, dataset) result in different and complementary data representations. The first half of the paper shows that those choices influence model predictions, leading to models that specialize to a subset of the data. The second half shows that those different representations are complementary and can be ensembled to yield better performing models. In particular, the authors show that ensembling models that are most diverse gives the largest improvement when transferring to a downstream task.