Deep Ensembles for Low-Data Transfer Learning

1 INTRODUCTION . There are many ways to construct models with minimal data . It has been shown that fine-tuning pre-trained deep models is a compellingly simple and performant approach ( Dhillon et al. , 2020 ; Kolesnikov et al. , 2019 ) , and this is the paradigm our work operates in . It is common to use networks pre-trained on ImageNet ( Deng et al. , 2009 ) , but recent works show considerable improvements by careful , task-specific pre-trained model selection ( Ngiam et al. , 2018 ; Puigcerver et al. , 2020 ) . Ensembling multiple models is a powerful idea that often leads to better predictive performance . Its secret relies on combining different predictions . The source of diversity for deep networks has been studied ( Fort et al. , 2019 ; Wenzel et al. , 2020 ) , though not thoroughly in the low-data regime . Two of the most common approaches involve training independent models from scratch with ( a ) different random initialisations , ( b ) different random subsets of the training data . Neither of these are directly applicable downstream with minimal data , as we require a pre-trained initialisation to train competitive models1 , and data scarcity makes further data fragmentation impractical . We study some ways of encouraging model diversity in a supervised transfer-learning setup , but fundamentally argue that the nature of pre-training is itself an easily accessible and valuable form of diversity . Previous works consider the construction of ensembles from a set of candidate models ( Caruana et al. , 2004 ) . Services such as Tensorflow Hub ( Google , 2018 ) and PyTorch Hub ( FAIR , 2019 ) contain hundreds of pre-trained models for computer vision ; these could all be fine-tuned on a new task to generate candidates . Factoring in the cost of hyperparameter search , this may be prohibitively expensive . We would like to know how suited a pre-trained model is for our given task before training it . This need has given rise to cheap proxy metrics which assess this suitability ( Puigcerver et al. , 2020 ) . We use such metrics - leave-one-out nearest-neighbour ( kNN ) accuracy , in particular - as a way of selecting a subset of pre-trained models , suitable for creating diverse ensembles of task-specific experts . We show that our approach is capable of quickly narrowing large pools ( up to 2,000 ) of candidate pre-trained models down to manageable ( 15 models ) task-specific sets , yielding a practical algorithm in the common context of the availability of many pre-trained models . 1For an illustration of the importance of using pre-trained models in the low-data regime see Appendix C.1 . We first experiment with sources of downstream diversity ( induced only by hyperparameterisation , augmentation or random data ordering ) , giving significant performance boosts over single models . Using our algorithm on different pools of candidate pre-trained models , we show that various forms of upstream diversity produce ensembles that are more accurate and robust to domain shift than this . Figure 1 illustrates the different approaches studied in our work . Ultimately , this new form of diversity improves on the Visual Task Adaptation Benchmark ( Zhai et al. , 2019 ) SOTA by 1.8 % . The contributions of this paper can be summarized as follows : • We study ensembling in the context of transfer learning in the low data regime & propose a number of ways to induce advantageous ensemble diversity which best leverage pre-trained models . • We show that diversity from upstream pre-training achieves better accuracy than that from the downstream fine-tuning stage ( +1.2 absolute points on average across the 19 downstream classification VTAB tasks ) , and that it is more robust to distribution shift ( +2.2 absolute average accuracy increase on distribution shifted ImageNet variants ) . • We show that they also surpass the accuracy of large SOTA models ( 76.2 % vs. 77.6 % ) at a much lower inference cost , and achieve equal performance with less than a sixth of the FLOPS . • We extend the work from Puigcerver et al . ( 2020 ) and demonstrate the efficacy of kNN accuracy as a cheap proxy metric for selecting a subset of candidate pre-trained models . 2 CREATING ENSEMBLES FROM PRE-TRAINED MODELS . We first formally introduce the technical problem we address in this paper . Next we discuss baseline approaches which use a single pre-trained model , and then we present our method that exploits using multiple pre-trained models as a source of diversity . 2.1 THE LEARNING SETUP : UPSTREAM , MODEL SELECTION , DOWNSTREAM . Transfer learning studies how models trained in one context boost learning in a different one . The most common approach pre-trains a single model on a large dataset such as ImageNet , to then tune the model weights to a downstream task . Despite algorithmic simplicity , this idea has been very successful . In a downstream low-data scenario , it is more difficult for a one-size-fits-all approach to triumph as specializing the initial representation becomes harder . As in Puigcerver et al . ( 2020 ) , we explore the scenario where a range of pre-trained models is available , and we can look at the target data to make a decision on which models to fine-tune . However , we generalize and improve it by simultaneously selected several models for fine-tuning , since downstream tasks may benefit from combining expert representations aimed at capturing different aspects of the learning task : for instance , on a natural scenery dataset one could merge different models that focus on animals , plants , food , or buildings . Fine-tuning all pre-trained models to pick the best one is a sensible strategy , but rarely feasible . To keep the algorithms practical , we identify two compute budgets that should be controlled for : The fine-tuning budget , i.e . the total number of models we can fine-tune on a downstream task ; and the inference budget , the maximum size of the final model . 2.2 BASELINES : DIVERSITY FROM DOWNSTREAM TRAINING . The baselines we propose leverage transfer learning by requiring a pre-trained model - this is crucial , see Appendix C.1 . We use a strong generalist model ( BiT-ResNet 50s from Kolesnikov et al . ( 2019 ) , trained on all upstream data ) and consider three methods to create a model set for ensemble selection . Random Seeds . Fine-tuning a generalist model multiple times with fixed hyperparameters will yield different classifiers , analagous to the DeepEnsembles of Lakshminarayanan et al . ( 2017 ) . Note , here we can only take advantage of randomised data ordering/augmentation , which Fort et al . ( 2019 ) showed , though useful , was not as beneficial as diversity from random initalisation . HyperEnsembles . Hyperparameter diversity was recently shown to further improve DeepEnsembles ( Wenzel et al. , 2020 ) . We define a hyperparameter search space , randomly sample as many configurations as we have fine-tuning budget , and fine-tune the generalist on downstream data with each of those configurations . Further details on training are given in Appendix A.2 . AugEnsembles . We generate a set of models by fine-tuning the generalist on each task with randomly sampled families of augmentation ( but fixed hyperparameters ) . Details are in Appendix A.3 . 2.3 OUR METHOD : DIVERSITY FROM UPSTREAM PRE-TRAINING . Fort et al . ( 2019 ) explain the strong performance of classical ensembling approaches – independently training randomly initialised deep networks – by showing that each constituent model explores a different mode in the function space . For transfer learning , Neyshabur et al . ( 2020 ) show that with pre-trained weights , fine-tuned models stay in a local ‘ basin ’ in the loss landscape . Combining both gives a compelling reasoning for the use of multiple pre-trained networks for transfer with ensembles , as we propose here . Instead of diversity from downstream fine-tuning , we show that in the low data regime , better ensembles can be created using diversity from pre-training . We consider three sources of upstream diversity . First , we consider generalists that were pre-trained with different random seeds on the same architecture and data . Second , we consider experts , specialist models which were pre-trained on different subsets of the large upstream dataset . Lastly , we exploit diversity in scale – pre-trained models with architectures of different sizes . Given a pool of candidate models containing such diversity , we propose the following algorithm ( Figure 1 ) : 1 . Pre-trained model selection . Fine-tuning all experts on the new task would be prohibitively expensive . Following Puigcerver et al . ( 2020 ) , we rank all the models by their kNN leave-one-out accuracy as a proxy for final fine-tuned accuracy , instead keeping the K best models ( rather than 1 ) . 2 . Fine-tuning . We add a fully connected layer to each model ’ s final representation , and then train the whole model by minimising categorical cross-entropy via SGD . Given a pool of K pre-trained models from stage 1 , we tune each with 4 learning rate schedules , yielding a total of L = 4K models for the step 3 ( Usually K = 15 and L = 60 ) . See Appendix A.1.1 for more details . 3 . Ensemble construction . This is shared among all presented ensembles . We use the greedy algorithm introduced by Caruana et al . ( 2004 ) . At each step , we greedily pick the next model which minimises cross-entropy on the validation set when it is ensembled with already chosen models . These steps are independently applied to each task ; each step makes use of the downstream dataset , so each dataset gets a tailored set of pre-trained models to create the ensemble pool and therefore very different final ensembles result . We also considered a greedy ensembling algorithm in kNN space which aims to sequentially pick complementary models which will likely ensemble well together ( Appendix C.6 ) , but picking top-K was generally better . 2.3.1 COMBINED APPROACHES . The diversity induced by different upstream models and distinct downstream hyperparameters should be complementary . Given a fine-tuning budget of L , we can set the number of pre-trained models K in advance , providing each of them with a random hyperparameter sweep of size L/K . However , for some tasks it may be more beneficial to have fewer different pre-trained models and a wider sweep , or vice versa . We aim to dynamically set this balance per-dataset using a heuristic based on the kNN accuracies ; namely , we keep all pre-trained models within some threshold percentage τ % of the top kNN accuracy , up to a maximum of K = 15 . Ideally , this would adaptively discard experts poorly suited to a given task , whose inclusion would likely harm ensemble performance . The saved compute budget is then used to squeeze more performance from available experts by testing more hyperparameters , and hopefully leading to greater useful diversity . We arbitrarily set τ = 15 % for our experiments , but this choice could likely be improved upon . Appendix C.5 shows how the number of models picked varies per task , and the gains with respect to having a fixed K . 2.3.2 PRE-TRAINING MODELS . We use BiT ResNets pre-trained on two large upstream datasets with hierarchical label spaces : JFT-300M ( Sun et al. , 2017 ) and ImageNet-21k ( Deng et al. , 2009 ) . We consider two types of pretrained models . Generalists are trained on the entire upstream dataset . In particular , we consider 15 JFT ResNet-50 generalists that were pre-trained with different random initalisations . Experts are generated by splitting the hierarchical label spaces into sub-trees and training independent models on the examples in each sub-tree . We pre-train 244 experts from JFT and 50 from ImageNet21k , following the protocol of ( Puigcerver et al. , 2020 ) ( see Appendix A.1 ) . For low-data downstream tasks , this is by far the most expensive stage of the process . It is however only incurred once , and its cost is amortized as new downstream tasks are served , since any downstream task can reuse them .

Summary: Paper proposed an ensemble learning approach for the low-data regime. Paper uses various sources of diversity - pre-training, fine-tuning and combined to create ensembles. It then uses nearest-neighbor accuracy to rank pre-trained models, fine-tune the best ones with a small hyper-parameter sweep, and greedily construct an ensemble to minimize validation cross-entropy. Paper claims to achieve state-of-the art performance with much lower inference budget.