No One Representation to Rule Them All: Overlapping Features of Training Methods

1 INTRODUCTION . Over the years , the machine learning field has developed myriad techniques for training neural networks . In image classification , these include data augmentation , regularization , architectures , losses , pre-training schemes , and more . Such techniques have highlighted the ability of networks to capture diverse features of the data : textures/shapes ( Geirhos et al. , 2018 ) , robust/non-robust features ( Ilyas et al. , 2019 ) , and even features that fit a random , pre-determined classifier ( Hoffer et al. , 2018 ) . Despite this representation-learning power , methods that yield high generalization performance seem to produce networks with little behavior diversity : models make similar predictions , with high-accuracy models rarely making mistakes that low-accuracy models predict correctly ( Mania et al. , 2019 ) . Additionally , the quality of features learned ( e.g . : for downstream tasks ) seems dictated by upstream performance ( Kornblith et al. , 2019 ) . Finally , training on subsets of the data yields low-accuracy models that don ’ t make performant ensembles ( Nixon et al. , 2020 ) . This seemingly suggests that high-performing models share similar biases , regardless of training methodology . Without behavior diversity , ensemble benefits are limited to reducing noise , since models make correlated errors ( Perrone & Cooper , 1992 ; Opitz & Maclin , 1999 ) . Without feature diversity , representations might not capture important features for downstream tasks , since feature reuse has been shown to be crucial for transfer learning ( Neyshabur et al. , 2020 ) . Without knowing the effect of training methodology , one might conclude that low-accuracy models have no practical use , since their predictions would be dominated by high-accuracy ones . One open question is if these findings faced unavoidable selection bias , since the highest-performing models have historically been trained with similar supervised objectives on IID datasets . Up until recently , this hypothesis was difficult to test . That changed with the recent success of large-scale contrastive learning , which produces competitively-high accuracy on standard generalization and robustness benchmarks ( Radford et al. , 2021 ; Jia et al. , 2021 ) . This motivates revisiting the question : How does training methodology affect learned representations and prediction behavior ? In this paper , we conduct a systematic empirical study of 82 models , which we train or collect , across hyper-parameters , architectures , objective functions , and datasets , including the latest high performing models CLIP , ALIGN , SimCLR , BiT , ViT-G/14 , and MPL . In addition to using different techniques , these new models were trained on data collected very differently , allowing us to probe the effect of both training objective , as well as pre-training data . We categorize these models based on how their training methodologies diverge from a typical , base model and show : 1 . Model pairs that diverge more in training methodology ( reinitializations , hyper-parameters , architectures , frameworks , → dataset scales ) produce increasingly uncorrelated errors . 2 . Ensemble performance increases as error correlation decreases , due to higher ensemble efficiency . The most typical ImageNet model ( ResNet-50 , 76.5 % ) , and its most different counterpart ( ALIGN-ZS , 75.5 % ) yield 83.4 % accuracy when ensembled , a +7 % boost . 3 . Contrastively-learned models display categorically different generalization behavior , specializing in subdomains of the data , which explains the higher ensembling efficiency . We show CLIP-S specializes in antropogenic images , whereas ResNet-50 excells in nature images . 4 . Surprisingly , we find that low-accuracy models can be useful if they are trained differently enough . By combining a high-accuracy model ( BiT-1k , 82.9 % ) with only low-accuracy models ( max individual acc . 77.4 % ) , we can create ensembles that yield as much as 86.7 % . 5 . Diverging training methodology yield representations that capture overlapping ( but not supersetting ) feature sets which , when combined , lead to increased downstream performance ( 91.4 % on Pascal VOC , using models with max individual accuracy 90.7 % ) . 2 RELATED WORK . Diversity in Ensembles . It is widely understood that good ensembles are made of models that are both accurate and make independent errors ( Perrone & Cooper , 1992 ; Opitz & Maclin , 1999 ; Wen et al. , 2020 ) . Beyond improving ensemble performance , finding diverse solutions that equally well explain the observations can help quantify model uncertainty ( also known as epistemic uncertainty ) – what the model does not know because training data was not appropriate ( Kendall & Gal , 2017 ; Fort et al. , 2019 ) . Many works have explored ways of finding such solutions ( Izmailov et al. , 2018 ) . Boostrapping ( Freund et al. , 1996 ) ( ensembling models trained on subsets of the data ) was found not to produce deep ensembles with higher accuracy than a single model trained on the entire dataset ( Nixon et al. , 2020 ) . This emphasizes how much data deep neural networks need to achieve high performance . Another work has examined the effect of augmentation-induced prediction diversity on adversarial robustness ( Liu et al. , 2019 ) . More relevant to us , Wenzel et al . ( 2020 ) has explored the effect of random hyper-parameters , finding best ensembles when combining models that are both hyperparameter and weight-diverse , albeit still considering similar frameworks and architectures . Model Behavior Similarity . These attempts were hindered as many high-performing techniques seem to produce similar prediction behavior . Mania et al . ( 2019 ) demonstrates , via “ dominance probabilities ” , that high-accuracy models rarely make mistakes that low-accuracy models predict correctly . This indicates that , within the models studied , high-accuracy models “ dominate ” the predictions of low-accuracy ones . Recht et al . ( 2019 ) shows that out-of-distribution robustness seems correlated with in-distribution performance . Relatedly , Kornblith et al . ( 2019 ) shows that upstream and downstream performance are very correlated . These jointly indicate that high-accuracy models learn strictly better representations , diminishing the importance of low-accuracy solutions ( even if they are diverse ) . Finally , Fort et al . ( 2019 ) shows that subspace-sampling methods for ensembling generate solutions that , while different in weight space , remain similar in function space , which gives rise to an insufficiently diverse set of predictions . Contrastive-Learning Models ; Different Large-Scale Datasets . This model behavior similarity might be explained by the fact that the training techniques that yield high performance on image classification tasks have been relatively similar , mostly relying on supervised learning on ImageNet , optionally pre-training on a dataset with similar distribution . Recently , various works have demonstrated the effectiveness of learning from large-scale data using contrastive learning ( Radford et al. , 2021 ; Jia et al. , 2021 ) . They report impressive results on out-of-distribution benchmarks , and have been shown to have higher dominance probabilities ( Andreassen et al. , 2021 ) . These represent some of the first models to deviate from standard supervised training ( or finetuning ) on downstream data , while still yielding competitive accuracy . They add to the set of high-performing training techniques ( which include data augmentation ( Cubuk et al. , 2018 ; 2020 ) , regularization ( Srivastava et al. , 2014 ; Szegedy et al. , 2016 ; Ghiasi et al. , 2018 ) , architectures ( Tan & Le , 2019 ; Dosovitskiy et al. , 2020 ; Hu et al. , 2018 ; Iandola et al. , 2014 ; Li et al. , 2019 ; Szegedy et al. , 2016 ; Simonyan & Zisserman , 2014 ; Sandler et al. , 2018 ) , losses ( Chen et al. , 2020 ; Radford et al. , 2021 ; Jia et al. , 2021 ) , pre-training schemes ( Kolesnikov et al. , 2020 ; Pham et al. , 2021 ) , etc ) , and provide the motivation for revisiting the question of whether training methodology can yield different model behavior . 3 METHOD . 3.1 MODEL CATEGORIZATION . In order to evaluate the performance of learned representations as a function of training methodology , we define the following categories , which classify model pairs based on their training differences : 1 . Reinits : identical models , just different in reinitialization . 2 . Hyper-parameters : models of the same architecture trained with different hyper parameters ( e.g . : weight decay , learning rate , initialization algorithm , etc ) . 3 . Architectures : models with different architectures , but still trained within the same frame- work and dataset ( e.g . : ResNet and ViT , both with ImageNet supervision ) . 4 . Frameworks : models trained with different optimization objectives , but on same dataset ( e.g . : ResNet and SimCLR , respectively supervised and contrastive learning on ImageNet ) . 5 . Datasets : models trained on large-scale data ( e.g . : CLIP or BiT – trained on WIT or JFT ) . In some sense , model categories can be supersets of one another : when we change a model architecture , we may also change the hyper-parameters used to train such architecture , to make sure that they are optimal for training this new setting . Unless stated otherwise , all ensembles are comprised of a fixed base model , and another model belonging to one of the categories above . This way , each category is defined relative to the base model : model pairs in a given category will vary because Model 2 is different than the base model along that axis . The result is that as we navigate along model categories ( Reinit → ... → Dataset ) , we will naturally be measuring the effect of increasingly dissimilar training methodology . See Appendix Table 1 for details . 3.2 MODEL SELECTION . We collect representations and predictions for 82 models , across the many categories above . We fix ResNet-50 , trained with RandAugment , as our base model . ResNet is a good candidate for a base model since it is one of the most typical ImageNet classification models , and the de-facto standard baseline for this task . In total , we train/collect models in the categories : 1 ) Reinit ; 2 ) Hyperparameters ( 51 ) : varying dropout , dropblock , learning rate , and weight decay , sometimes jointly ; 3 ) Architectures ( 17 ) : including EfficientNet , ViT , DenseNet , VGG ; 4 ) Framework ( 2 ) : including SimCLR , and models trained with distillation ; and 5 ) Dataset ( 12 ) : including CLIP , ALIGN , BiT , and more , trained on WIT ( Radford et al. , 2021 ) , the ALIGN dataset , JFT ( Sun et al. , 2017 ) , etc . We additionally collect high-performing models MPL , ALIGN ( L-BFGS ) , ViT-G/14 , BiT-1k , CLIP-L , EfficientNet-B3 for some of our analyses . These are some of the latest , highest-performing models for ImageNet classification . We found it necessary to calibrate all models using temperature scaling ( Roelofs et al. , 2020 ; Guo et al. , 2017 ) to maximize ensemble performance . Finally , unless stated otherwise , we only use models in the same narrow accuracy range ( 74-78 % accuracy on ImageNet ) , which guarantees that the effects observed are indeed a function of diverging training methodology , and not of any given model being intrinsically more performant than another . A complete list of models can be found in the Appendix .

Summary: This paper empirically investigates how different models, trained with different methodologies should be ensembled to maximize the accuracy in image-net classification task. By carefully designed experiments, the authors look at different aspects of the trained models and provide guidelines for selecting models to be ensembled. The main takeaway: models trained with increased divergence in training methodologies are best suited for ensembling.