Deep Ensembles for Low-Data Transfer Learning
Authors: Basil Mustafa
Carlos Riquelme Ruiz
Joan Puigcerver
André Susano Pinto
Daniel Keysers
Neil Houlsby
Venue:  (2021)
Keywords: transfer learning
representation learning
computer vision
ensembles

Abstract: In the low-data regime, it is difficult to train good supervised models from scratch.
Instead practitioners turn to pre-trained models, leveraging transfer learning. Ensembling is an empirically and theoretically appealing way to construct powerful predictive models, but the predominant approach of training multiple deep networks with different random initialisations collides with the need for transfer via pre-trained weights. In this work, we study different ways of creating ensembles from pre-trained models. We show that the nature of pre-training itself is a performant source of diversity, and propose a practical algorithm that efficiently identifies a subset of pre-trained models for any downstream dataset. The approach is simple: Use nearest-neighbour accuracy to rank pre-trained models, fine-tune the best ones with a small hyperparameter sweep, and greedily construct an ensemble to minimise validation cross-entropy. When evaluated together with strong baselines on 19 different downstream tasks (the Visual Task Adaptation Benchmark), this achieves state-of-the-art performance at a much lower inference budget, even when selecting from over 2,000 pre-trained models. We also assess our ensembles on ImageNet variants and show improved robustness to distribution shift.

Under review as a conference paper at ICLR 2021
DEEP ENSEMBLES FOR LOW-DATA TRANSFER LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
In the low-data regime, it is difficult to train good supervised models from scratch. Instead practitioners turn to pre-trained models, leveraging transfer learning. En- sembling is an empirically and theoretically appealing way to construct powerful predictive models, but the predominant approach of training multiple deep net- works with different random initialisations collides with the need for transfer via pre-trained weights. In this work, study different ways of creating ensembles from pre-trained models. We show that the nature of pre-training itself is perfor- mant source of diversity, and propose a practical algorithm that efficiently identi- fies a subset of pre-trained models for any downstream dataset. The approach is simple: Use nearest-neighhour accuracy to rank pre-trained models, fine-tune the best ones with a small hyperparameter sweep. and greedily construct an ensemble to minimise validation pross-entropy. When evaluated together with strong base- lines on 19 different downstream tasks (the Visual Task Adaptation Benchmark), this achieves state-of-the-arr performance a much lower inference budget, even when selecting from over 2,000 pre-trained models. We also assess our ensembles on ImageNet variants and show improved robustness to distribution shift.
INTRODUCTION
There are many ways to construct models with minimal data. It has been shown that fine-tur fine-tuning pre-trained deep models is a compellingly simple and performant approach (Dhillon et al.] 2020 Kolesnikov et al., 2019), and this is the paradigm pre-trained on ImageNet (Deng et al. 2009), but recent works show considerable improvements by careful, task-specific pre-trained model selection (Ngiam et al. 2018: Puigcerver al. 2020).
work operates in. is common to use networks
Ensembling multiple models is a powerful idea that often leads to better predictive performance. Its secret relies on combining different predictions. The source of diversity for deep networks has been studied (Fort et al.) 2019; Wenzel et al. 2020). though not thoroughly in the low-data regime, Two of the most common approaches involve training independent models from scratch with (a) different random initialisations, (b) different random subsets of the training data. Neither of these are directly applicable downstream with minimal data, as we require pre-trained initialisation to train compet- itive modelş"] and data scarcity makes further data fragmentation impractical. We study some ways of encouraging model diversity in a supervised transfer-learning setup, but fundamentally argue that the nature of pre-training is itself an easily accessible and valuable form of diversity.
Previous works consider the construction of ensembles from a set of candidate models (Caruana et al., 2004). Services such as Tensorflow Hub (Google) 2018) and PyTorch Hub (FAIR 2019) contain hundreds of pre-trained models for computer vision; these could all be fine-tuned on new task generate candidates. Factoring the cost of hyperparameter search, this may be prohibitively expensive. We would like to know how suited a pre-trained model is for our given task before training it. This need has given rise to cheap proxy metrics which assess this suitability (Puigcerver et al. 2020). We use such metrics lcave-one-out nearest-neighhour (kNN) accuracy, in particular as a way of selecting subset of pre-trained models, suitable for creating diverse ensembles of task-specific experts. We show that our approach is capable of quickly narrowing large pools (up to 2,000) of candidate pre-trained models down to manageable (15 models) task- ssecispciiiic sets, yielding a practical algorithm in the common context of the availability of many pre-trained models.
For an illustration of the importance of using pre- trained models in the low-data regime see Appendix C.I

---

Under review as a conference paper at ICLR 2021
upstream data
downstream data
Best Single Model
traning
val
generalist
generalist
greedy ensemblee
Downstream Ensemble
O
generalist
generalist
generalist
generalist
specialist
specialist H+1
greedy ensemble
Upstream Ensemble
specialist
specialist H+2
specialist
specialist
Upstream
Model
Finetuning
Ensembling
Flinal Model
Training
Selection
Figure I: Overview of the different ways of constructing diverse ensembles studied in this work. We propose algorithm that exploits diversity in large pool of pre-trained models, by using leave-one-out k-nearest--eigeee neighbst (kNN) accuracy to select subset to form the ensemble.
We first experiment with sources of downstream diversity (induced only by hyperparameterisation,, augmentation or random data ordering), giving significant performance boosts over single models. Using our algorithm on different pools of candidate pre-trained models, we show that various forms of upstream diversity produce ensembles that are more accurate and robust to domain shift than this. Figure T Iillustrates the different approaches studied in our work. Ultimately, this new form of diversity improves the Visual Task Adaptation Benchmark (Zhai al. 2019) SOTA by 1.8%.
The contributions of this paper can be summarized as follows:
We study ensembling the context of transfer learning in the low data regime propose a num- ber of ways to induce advantageous ensemble diversity which best leverage pre-trained models.
We show that diversity from upstream pre-training achieves better accuracy than that from the downstream fine-tuning stage (+1.2 absolute points on average across the 19 downstream classifi cation VTAB tasks), and that it is more robust to distribution shift (+2.2 absolute average accuracy increase on distribution shifted ImageNet variants).
We show that they also surpass the accuracy of large SOTA models (76.2% vs. 77.6%) at much lower inference cost, and achieve equal performance with less than sixth of the FLOPS.
We extend the work from Puigcerver et al. (2020) and demonstrate the efficacy of KNN accuracy as a cheap proxy metric for selecting subset of candidate pre-trained models.
CREATING ENSEMBLES FROM PRE-TRAINED MODELS
We first formally introduce the technical problem we address in this paper. Next we discuss baseline approaches which use single pre- tre-trained model, and then present our method that exploits using multiple pre-trained models as a source of diversity.
2.1 THE LEARNING SETUP: UPSTREAM, MODEL SELECTION, DOWNSTREAM
Transfer learning studies how models trained in one context boost learning in a different one. The most common approach pre-trains single model on a large dataset such as ImageNet, to then tune the model weights to a downstream task. Despite algorithmic simplicity, this idea has been very

---

Under review as a conference paper at ICLR 2021
successful, In downstream low-data scenario, more difficult for a one-size-fits-al approach to triumph as specializing the initial representation becomes harder. As in infuugeerr et al. 22020) we explore the scenario where a range of pre-trained models is available, and we can look at the larget data to make a decision which models to fine-tune. However, we generalize and improve it by simultancously selected several models for fine-tuning, since downstream tasks may benefit from combining expert representations aimed at capturing different aspects of the learning task: for instance, on a natural scenery dataset one could merge different models that focus on animals, plants, food, buildings. Fine-tuning all pre-trained models to pick the best one is a sensible strategy, but rarely feasible. To keep the algorithms practical, we identify two compute budgets that should be controlled for: The fine-tuning budget, i.c. the total number of models we can fine-tune on a downstream task; and the inference budget, the maximum size of the final model.
2.2 BASELINES: DIVERSITY FROM DOWNSTREAM TRAINING
The baselines we propose leverage transfer learning by requiring a pre-trained model this is crucial, see Appendix C.I We use strong generalist model (BiT-ResNet 50s from Kolesnikov et al. 2019))
trained on all upstream data) and consider three methods to create a model set for ensemble selection.
Random Seeds. Fine-tuning a generalist model multiple times with fixed hyperparameters will yield different classifiers, analagous to the DeepEnsembles of Laaakhmiinaaaaanan et al. (2017). Note, here can only take advantage of randomised data widering/augmentaine showed, though useful, was not as beneficial as diversity from random initalisation.
which Fort et al. (2019)
Hyperensembles. Hyperparameter diversity was recently shown to further improve DeepEnsem- bles (Wenzel et al., (2000.. We define a hyperparameter search space, randomly sample as many configurations as we have fine-tuning budget, and fine-tune the generalist downstream data with each of those configurations. Further details on training are given in Appendix A.2
AugEnsembles. We generate a of models by fine-tuning the generalist each task with ran- domly famile, of augmentation (but fixed hyperparameters) Details are in Appendix A.3
2.3
OUR METHOD: DIVERSITY FROM UPSTREAM TRA-TIG
Fort at. explain the strong performance of classical ensembling approaches - independently training randomly initialised deep networks by showing that each constituent model explores different mode in the function space. For transfer learning, Neyshabur et al. (2020) show that with pre-trained weights, fine-tuned models stay in local "basin' in the loss landscape. Combining both gives a compelling reasoning for the use of multiple pre-trained networks for transfer with ensembles, as we propose here. Instead of diversity from downstream tne-tuning, we show that in the low data regime, better ensembles can be created using diversity from pre- mre-training.
We consider three sources of upstream diversity. First, consider generalists that were pre-trained with different random seeds on the same architecture and data. Second, we consider experts, spe- cialist models which were pre-trained exploit diversity in scale pre-trained models with architectures of different sizes. Given pool of candidate models containing such diversity, we propose the following algorithm (Figure []::
different subsets of the large upstream dataset. Lastly, we
1. Pre-trained model selection. Fine-tuning all experts on the new task would be prohibitively expensive. Following Puigeerver al. (2020), we rank all the models by their ANN leave-one-oue accuracy as a proxy for final fine-tuned accuracy, instead keeping the K best models (rather than I).
2. Fine-tuning, We add fully connected layer to each model's final representation, and then train the whole model by minimising categorical cross-entropy via SGD. Given pool of K pre-trained models from stage 1, we tune each with 4 learning rate schedules, yielding total of I.= 4K models for the step 3 (Usually K = 15 and L. = 60). See Appendix A.1.1 for more details.
3. Ensemble construction. This shared among all presented ensembles. We use the greedy algorithm introduced by Caruana et al. (2004). At each step, we greedily pick the next model which minimises cross-entropy on the validation set when it is ensembled with already chosen models.
These steps are independently applied to each task; each step makes use of the downstream dataset, so each dataset gets a tailored set of pre-trained models to create the ensemble pool and therefore
3

---

Under review as a conference paper at ICLR 2021
very different final ensembles result. We also considered greedy ensembling algorithm ANN space which aims to sequentially pick complementary models which will likely ensemble well to- gether (Appendix C.6), but picking top- K was generally better.
2.3.1 COMBINED APPROACHES
The diversity induced by different upstream models and distinct downstream hyperparameters should be complementary. Given fine-tuning budget of L, we can set the number of preitrained models K in advance, providing each of them with a random hyperparameter sweep of size L/K, K. However, for some tasks it may be more beneficial to have fewer different pre-trained models and a wider sweep, or vice versa. We aim to dynamically set this balance per-dataset using a heuristic based the kNN accuracies; namely, centage r 5 of the top ANN accuracy, up to a maximum of K = 15. Ideally, this would adaptively discard experts poorly suited to given task, whose inclusion would likely harm ensemble perfor- mance. The saved compute budget is then used to squeeze more performance from available experts by testing more hyperparameters, and hopefully leading greater useful diversity. We arbitrarily set т 15% for our experiments, but this choice could likely be improved upon. Appendix C.5 shows how the number of models picked varies per task, and the gains with respect having a fixed K.
keep all pre-trained models within some threshold per-
2.3.2 PRE-TRAINING MODELS
We use BiT ResNets pre-trained on two large upstream datasets with hierarchical label spaces: JFT-300M (Sun et al. 2017) and ImageNet-2Ik (Deng et al. 2009). We consider two types of pre- trained models. Generalists are trained on the entire upstream dataset. In particular, we consider 15 JFT ResNet-50 generalists that were pre-trained with different random initalisations. Experts are generated by splitting the hierarchical label spaces into sub-trees and training independent models on the examples in each sub-tree. We pre-train 244 experts from JFT and 50 from ImageNet2 I k, following the protocol of (Puigcerver et al. 2020) (see Appendix AI.I). For low-data downstream tasks, this is by far the most expensive stage of the process. It is however only incurred once, and its cost amortized as new downstream tasks are served, since any downstream task can reuse them.
3 ENSEMBLE EVALUATION
Downstream Tasks. We evaluate models on the Visual Task Adaptation Benchmark (Zhai et al. 2019] 19 diverse downstream classification tasks, split into 'natural', 'specialised' and 'structured categories. As
primarily interested in low-data regimes, the tasks only have 1000 training datapoints (i.e., VTABBK) with a number of classes ranging from 2 to 397. We split data into 800 training examples and 200 validation examples. See Appendix B.I more information.
Test Performance. For our final competitive models, we first train all the individual models on the 800 training points. Then, use the 200 validation data points find the best ensemble (both running the greedy algorithm and choosing the overall ensemble size). For the resultant ensemble, we retrain constituent models on the full 1000 data points, and evaluate it on the test data.
Robustness. We train ExpertEnsembles and HyperEnsembles on ImageNet (Deng et al. 2009) While ImageNet does not match our low-data regime of interest, previous work and additional datasets allow us to conveniently measure robustness and uncertainty metrics. Thus, alongside re- porting the accuracy on the official validation split, we assess models on a number of ImageNet- based robustness benchmarks, aiming to quantify calibration and robustness to distribution shift (Djolonga et al. 2020p. More details on these variants are available in Appendices B.2 and A.4
4 EXPERIMENTAL RESULTS
Unless otherwise specified, all experiments use fine-tuning budget and inference budget of 60 and 15 models respectively. This was set arbitrarily; we experiment with both budgets to see the effect.
4

---

Under review as a conference paper at ICLR 2021
78
76
RI5234
74
R101x3
R201
72
72
Single SOTA
R101
R50k3
70
750
R50 IyperEneemble
Single-medel 5OTA
R50 ExpertEnsembie
R50 HyperEnsemble
R101 ExperCnnemble
70
R50
R50 Expe +nnembble
2
FLOPS R50)
FLOPS ix R50)
JFT pre-trained models.
(b) Imagenet2 pre-trained models.
Figure 2:
Inference cost vs. VTAB1K performance. State-of-the-arr generalist models of different scales are compared against ensembles with varying inference budgets.
4.1 THE VALUE OF ENSEMBLES
We first show that ensembles in low-data transfer regimes dramatically beat their single-model coun- terparts -which are often much larger networks. Figure 2 and Table [ compare our best ensembles (which all use upstream or combined diversity) on VTAB tasks. Our baselines are BiT models from (Kolesnikov et al.) 2019), which had until now satee-f-thh--e art performance.
JFT pre-trained models. The most standard approach -fine-tuning a single R50 model trained on all of JFT- leads to an average accuracy of 70.6%. It greatly lags behind compared to the ensemble- based algorithms; in particular, the difference is striking for structured and natural datasets. average, the ensembles selected between 9 and 10 downstream models -this number greatly varies depending on the task, c.g. 3 were selected for CalTech101 and 14 for Diabetic Retinopathy, Ac- cordingly, capacity-wise, it makes sense to compare the ensembles to larger ResNets. Table shows that the JFT R50 ensembles match or slightly beat the performance of a R152x4. In particular, the ensembles offer large advantage in settings where tasks diverge from single-object recognition, e.g. in the structured datasets. Even in natural domains, the experts have a better accuracy/LLog /FLP ratio than the R152x4, which has 40x more parameters than single R50. Even more significant is the difference in inference time, ensemble predictions can be easily parallelized.
ImageNet21k pre-trained models. The story fairly similar for the pool of ImageNet21k experts. The ensembles select on average between 7 and 8 models -again, strongly task-dependent. Ensem- bles' average performance improvement with respect is around absolute points (more than 7%). We also consider a much larger generalist baseline, in this case R101x3, which has more than 16x many parameters as a single R50. Still the ensembles far outperform it, especially in structured datasets.
single R50 trained on all of ImageNet2
Table 1: Test accuracy of our best ensembles against reproduced baselines from Kolesnikov et al. (2019) [*]. For each dataset, we take the median of three independent runs. Rows show the average over datasets. Bootstrapped confidence intervals at the 95% level. The source of diversity for ensem- bles shown: U upstream (during pre-traie training) and C combined (pre-training and fine-tuning).
Description
Diversity
VTAB
Natural
Specialised
Structured
JFT BÝT-L R50*
70.6 71.0]
77.8
83.6
57.9
JFT BiT-L R152x4*
76. 76.2
86.0
87.0
62.2
JFT R50 Experts Generalists
76.8 166.876.4-77 77.01
82.6
85.8
67.2
JFT R101 Experts
U
77.6
83.6
86.4
68.0
INet21k BiT-M R50"
70.0 70.0,69.6 69.6 70.51
77.0
84.7
56.6
INet21k BiT-M R101x3*
72.7 172.1 73.51
80.3
85.7
59.4
INet21k R50 Experts
U
75.3 17.55 75.6]
79.9
85.7
66.1
INet21k R50 HyperExperts
75.6 75.8)
79.9
85.5
67.0

---

Under review as a conference paper at ICLR 2021
Table 2: Upstream diversity gives better ensembles. Test accuracy of different ensembles. For each dataset, we take the median of three independent runs. Rows show the average over datasets. Bootstrapped confidence intervals at the 95% level. The source of diversity is noted as: D = down- stream (during tine-tunin), U upstream (during pre- (ee-rrainnng) and C combined (both).
Description
Diversity
VTABIK
Natural
Specialised
Structured
JFT R50 Seeds
D
74.9/74.5-75.1
78.5
85.9
66.2
JFT R50 Augs
D
73.7 (3.7 75.7]
80.6
86.8
61.2
JFT R50 Hypers
D
75. 75.6 75.7
80.1
85.6
66.6
JFT R50 Exxpess
U
76.8176.4 116.47-6-77700
82.6
85.8
67.2
JFT R50 Aug(Gen. Experts)
C
77.6 176.8-77.88 77.91
82.7
86.4
68.7
JFT R50 Experts
U
76.4)76.1 76.61
82.2
85.6
66.8
JFT R50 HyperExperts
C
76.6176.0 76.7]
82.5
85.8
66.8
JFT R50 AugExperts
C
76.8 76.6 76.91
82.6
86.0
67.2
INet21k R50 Hypers
D
74.2 74.81
79.6
86.2
63.5
INet21k R50 Experts
U
75.3 75.61
79.9
85.7
66.1
INet21k R50 HyperExperts
C
75.6 75.6 75.81
79.9
85.5
67.0
Table 3: Ablations. Test accuracy of different ensembles. For each dataset, we take the median of three independent Rows show the average datasets. Bootstrapped confidence intervals the 95% level. Pre-training done on JFT, except for "All Experts" that also used ImageNet21k.
Description
VTABIK
Natural
Specialised
Structured
Base
R50 Generalists+ Experts
76.8 1..8)76.-77.00 77.01
82.6
85.8
67.2
Specialists vs
R50 Experts
76. 76.4)76.1
82.2
85.6
66.8
generalists
R50 Generalists
76.5 76.71
81.3
86.2
67.7
Combine scales
R18/34/50 Experts
76.8/76.4 77.21
82.2
85.5
67.9
Stack with scale
R101 Experts
77.6
83.6
86.4
68.0
Massive pool
All Experts (JFT/INe(21K)
77. 77.66773.-77
83.6
86.1
68.1
Overall. A more complete story supporting the ensembles' value is depicted in Figure 2 The gray dashed line represents the previous Pareto frontier of VTAB average accuracy per FLOP The ensemble models dominate the state-of-the-an indicating their efficacy in the low-data regime.
4.2 THE VALUE OF UPSTREAM DIVERSITY
Results in Table suggest that upstream diversity improves downstream diversity. For both JFT and ImageNet21 k pre-training, ensembles benefiting from upstream sources of diversity outperform their downstream- based counterparts. Combining both gives a further small boost for expert ensembles.
Ablations on JFT pre-trained models are shown in Table we now discuss the learnings from that.
Experts help when pre-training is relevant: Results on Table 2 used kNN to pick from a pool of 15 generalists and 244 experts. We break these pools down separately. Experts give significant boost on Natural datasets; the ensembles take advantage of the relevance of experts pre-trained on the predominately 'natural' slices of the upstream data. For datasets without clear experts, there is less benefit to this approach, and the generalists shine.
Performance improvements stack with scale: shows performance gains stack somewhat with scale; it improves on R50 Experts by 1.2% absolute points accuracy, improving all categories. We explore this more thoroughly in Appendix C.7
The strong performance of the R101 Expert Ensemble
Combining upsiream and downstream diversity helps: As discussed in Section 2.3.1 we combine ex- perts with hyper or augmentation ensembles. The simple approach of thresholding by LNN accuracy works well, giving small boost in test performance (R50 Aug(Ger) Gen. Experts)) R50 HyperExperts, R50 AxgExter,)). More details on this are provided in Appendix C.5

---

Under review as a conference paper at ICLR 2021
4.3 THE VALUE OF NEAREST NEIGHBOUR SELECTION
kNN may possibly help: The greedy ensemble algorithm is not perfect, and with such small vali- dation set is prone to overfit. When all upstream JFT R50 experts are fine-tuned and passed to the greedy algorithm, test performance drops slightly. We further explore this in Appendices C.4
can compare models of different sizes: Overall, larger models perfom better at transfer (Kolesnikov et al. 2019). Per-dataset, this is not the case; e.g. found R34 experts were best on structured tasks. One may expect ANN selection or the greedy algorithm to be biased towards selecting larger architectures. The final ensembles instead improves on just R50s by 0.4%, indicating possible benefits; more discussion is in Appendix C.7
mix of scales. The R18/R34/R50 experts ensemble
Can filter very large pool of models: When selecting only 15 pre-trained models from over 2000 candidates (different architecture sizes and upstream datasets), the overall VTAB performance (All Experts in Table is similar to only selecting from ResNet-101s. This highlights the remarkable robustness of our model selection. These results are broken down further in Appendix C.8
Mirroring Puigcerver al. (2020), we have shown kNN be cheap yet successful way of selecting models. It is not perfect- when combining pools, one would hope for at least 'best of both performance. kNN selection wasn't needed for generalists (we had 15 pre-trained models), but when combining the generalists and experts in pool, specialised//srrutuuee performance drops slightly.
4.4 EFFECT OF FINE-TUNING BUDGET
In most experiments, the KNN picks K 15 experts. With the default 4 hyperparameters, this is fine-tuning budget of 60 models. This is the number of models trained for given task, and the majority of compute expenditure in- curred by practitioner, as the ANN selection/ensembingg comparatively cheap. The hyperensemble was run with the same budget. Figure shows how performance drops with reduced fine-tuning budget. Interestingly, the expert ensembles are actually more robust to a reduced budget, retaining higher performance when training fewer models, indicating the ANN's 's usefulness as pre-selection phase.
Final test accuracy
0
0.76
0.75
ensemble
0.74
experts
hyper
20
4D
60
Fine-tuning budget
Figure 3: Effect of fine-tuning budget ensemble VTAB performance.
4.5 ROBUSTNESS TO DISTRIBUTION SHIFT
Previous work has shown ensembles help with metrics relating to uncertainty and calibration (Laksh- minarayanan et al. 2017 Stickland & Murray 2020). To assess this, we train JFT R50 HyperEnsem- bles and Experttnnsembles for ImageNet cl classification. For the former, we use the BiT generalist; for the latter, we use the 244 experts, applying ANN to 50,000 examples from the training set to select experts. For both we use the validation set for greedy ensembling. Once the ensembles are trained and constructed, we assess them on a suite of datasets aiming to quantify robustness to distribution shift. Each dataset introduces some form of distribution shift (further details in Appendix B.2 what we assess is the accuracy on these datasets. AigureAssoww them. The expert ensembles offer slightly better accuracy the held out data; more importantly, they perform significantly better under distribution shift, improving over the HyperEnsembles by on average 2.2% across datasets.
5 RELATED WORK
We present literature related to the main aspects of this work. As well as previous highlighted nov- elties, we believe our contribution is distinguished from previous ensembling works by focusing on diverse datasets with production-scale deep models (instead of demonstrative smaller architectures and datasets), limiting the training data available, and formally assessing distribution shift.
Transfer Learning. Relating to a long history of research in manifold and representation learning Tan et al. 2018), the transfer of features learnt by deep neural networks aims to reuse the abstract features learnt on a source (upstream) dataset in order to improve performance or data efficiency on target (downstream) datasct. Bengio (201T) studied this in the context of unsupervised pre-training,
7

---

Under review as a conference paper at ICLR 2021
Figure 4: Expert ensembles re- tain higher accuracy under do- main shift. Aside from the first bar, which shows test accuracy, all other bars correspond to some form of induced distribution shift, either artificially or otherwise. In all but one, we get significant boosts in accuracy compared to the HyperEnsembles.
ImageNit
accuracy
accuracy
imageNiet C
accuracy
imageNit R
accuracy
imageNet V2
accuracy
imageNet
accuracy_dmo drop
VidRobust
accuracy_pmm pmk
anchor accuracy
ObjectNet
top accuracy
top 3 accuracy
accuracy drop
accuracy pmk
anchor catborrrcuraccyyc
-1%
0%
1%
2%
3%
4%
5%
6%
7%
Absolute improvement ExpertEnsemble over HyperEnsemble accuracy
proposing a number of ways to learn and re-use the features. Many works have shown benefits of transfer relating to convergence speed (Raghu et al. 2 generalisation (Yssinkki et al.) 2014), accuracy (Zhai et al.] 2019) and robustness (Djolonga al. 2020). with the latter two showing particular benefits in the low-data regime.
Ensemble Learning. Known for improving models, ensembling methods have been studied in depth in and out of deep learning academia (Seni & Elder (011)). There are few works which study ensembles in the context of transfer learning (Acharya et al., [012 Sun et al. 2013). Bachman et al. (2014) pre-trat pre-train entire ensembles on the source data and transfer. instead of transferring individual models. Work in the low-data regime is sparser. Using an ensemble of models from multiple train- ing checkpoints, Laine & Aila (2017) label unlabelled data then train individual models further, improving data efficiency for CIFAR 100/SVHN. For few-shot classification on new class, class,Dvornik Dvornik al. 2019) construct ensembles of mean-centroid classifiers from pre-trained ResNet18s.
Deep Ensembles. Lakshminarayanan et al. (0117) show that a simple approach of adversarially training multiple randomly initialised models from scratch and ensembling them yielded models with strong predictive uncertainty and calibration. Wenzel et al.20200) showed that hyperensembles, which vary random initialisations and hyperparameters, outperform these deep ensembles.
On Constructing Ensembles. A key part of our algorithm is the use of the KNN to narrow down candidate pre-trained models into a relevant subset. Caruana al. (2004) was arguably the seminal work studying how to select an optimal ensemble from a set of candidate models. A number of works extend AutoML frameworks (He et al. 2019) to explicitly optimise both the ensembling method and the members to maximise overall performance (Wistuba et al. 2017
Xavier-Júnior et al. 2018).
CONCLUSIONS
We have studied simple ways of creating performant ensembles with a limited amount of data. Over- all, ensembles dramatically outperform their single-model counterparts. We show that diversity from upstream pre-training results in better ensembles than diversity induced downstream, regardless of whether this upstream diversity comes from pre-training multiple generalist models with different initialisations, using different architectures specialisation via pre-training on different data. We demonstrate the efficacy of the nearest-neighhours classifier as an easily calculated discriminator between different pre-trained models, and even way decide how many models to try downstream task, leading to convenient ways to combine both upstream and downstream diversity.
These ensembles achieve SOTA performance on the Visual Task Adaptation Benchmark signifi- cantly smaller inference cost, while also outperforming ensemble approaches relying on downstream diversity. They also exhibit higher robustness to domain shift as assessed by ImageNet variants.
There are many interesting avenues for future work. All our considered models were pre-trained in a supervised fashion, and this should certainly be extended to include other forms of pre-training. This approach of combining different pre- rre-trained models is highly complimentary with efforts which train ensembles end-to-ae -onddto-dddddd with eerrurti-encuurainng losses, such as those in Lee et al. (2015) and Webb et al. (2019). Lastly, works such as Batch Ensembles (Wen et al. (020) and Parameter Superposition (Cheung et al. 2019) systematically decompose network parameters to compactly train ensembles. For pre-trained models with the same architecture, weights could be deconstructed to initialise those methods so as to benefit from transfer learning and make them feasible in the low-data regime.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Ayan Acharya, Eduardo R. Hruschka, Joydeep Ghosh, and Sreangsu Acharyya. Transfer learning with cluster ensembles. In ICML Workshop on Unsupervised and Transfer Learning, 2012.
Philip Bachman, Ouais Alsharif, and Doina Precup. Learning with pseudo-ensembles. In Neural Information Processing Systems (NeurIPS), 2014.
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In International Conf. Machine Learning (ICML), 2019.
Yoshua Bengio. Deep learning of representations for unsupervised and transfer learning. In ICML Unsupervised and Transfer Learning Workshop. 2011.
Rich Caruana, Alexandru Niculescu-Mizil, Geoff Crew, and Alex Ksikes. Ensemble selection from libraries of models. In International Conference on Machine Learning (ICML), 2004.
Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Bench- mark and state of the Proceedings of the IEEE, 18551091185--1883, Oct 2017.
Brian Cheung, Alex Terekhov, Yubei Chen, Pulkit Agrawal, and Bruno Olshausen. Superposition of many models into one. In Neural Information Processing Systems (NeurIPS), 2019.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In Computer Vision and Pattern Recognition (CVPR), 2014.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. ImageNet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition (CVPR), 2009.
Guneet S. Dhillon, Pratik Chaudhari, Avinash Ravichandran, and Stefano Soatto. A baseline for few-shot image classification. In International Conf. Learning Representations (ICLR), 2020.
Josip Djolonga, Jessica Yung, Michael Tschannen, Rob Romijnders, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Matthias Minderer, Alexander D' Amour, Dan Moldovan, Sylvain Gelly, Neil Houlsby, Xiaohua Zhai, and Mario Lucic. On robustness and transferability of convo- lutional neural networks. arXiv:2007.0855 2020.
Nikita Dvomikk Cordelia Schmid, and Julien Mairal. Diversity with cooperation: Ensemble methods for few-shot classification. In International Conference on Computer Vision (ICCV), 2019.
FAIR. PyTorch Hub, 2019. URL holLhttts::/pytrrc..co
Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep ensembles: A loss landscape per- spective. arXiv:19/2.0275,/ 2019.
Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are ready for autonomous driving? the kitti vision benchmark suite. In Computer Vision and Pattern Recognition (CVPR), 2012.
Google. TensorFlow Hub, 2018. URL htLlhttss///ttutut dev/
Xin He, Kaiyong Zhao, and Xiaowen Chu. arXiv:1908.00700) 2019.
AutoMI:: A survey of the state-of-the-art.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. EuroSAT: novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2226, 2019.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common ruptions and perturbations. In International Conf. on Learning Representations (ICLR), 2019.
Dan Hendryeks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. arXiv:1907.07111 2019.

---

Under review as a conference paper at ICLR 2021
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: critical analysis of out-of-distribution generalization.
10X4:22006.1641,,, 2020.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Fei-Fei Li. C Lawrence Zitnick, and Ross Girshick. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. Computer Vision and Pattern Recognition (CVPR), 2017.
Kaggle and HyePacs. Kaggle diabetic retinopathy detection, 2013. URL https:://// //www www.
kaggle. htt.e///...cccccaaaaeeeeeeeeeeee
data
Alexander Kolesmikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (BiT): General visual representation learning. arXiv:1912.113700 2019.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In International Conference on Learning Representations (ICLR), 2017.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predic- tive uncertainty estimation using deep ensembles. In Neural Information Processing Systems (NeurIPS), 2017.
Yann LeCun, Fu Jie Huang, and Léon Bottou. Learning methods for generic object recognition with invariance to pose and lighting. Computer Vision and Pattern Recognition (CVPR), 2004.
Stefan Lee, Senthil Purushwalkam, M. Cogswell, David J. Crandall, and Dhruv Batra. Why M heads are better than one: Training a diverse ensemble of deep networks. arXiv:1511 1166144 2015.
Fei-Fei Li, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental Bayesian approach tested on 101 object categories. Computer Vision and Puttern Recognition Workshop, 2004.
Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dSprites: Dis- entanglement testing sprites dataset, 2017. URL https://github.. com/ deepmind/
dsprites-dataset/
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011, 2011.
Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang, What is being transferred in transfer learn- ing? arXiv:2008. 2020.
Jiquan Ngiam, Daiyi Peng, Vijay Vasudevan, Simon Kornblith, Quoe V. Le, and Ruoming Pang. Domain adaptive transfer learning with specialist models. /xX1v:/811.070566 2018.
Maria-Elena Nilsback and Andrew Zisserman. A visual vocabulary for flower classification. In Computer Vision and Pattern Recognition (CVPR), 2006.
Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In Computer Vision and Pattern Recognition (CVPR), 2012.
Joan Puigcerver, Carlos Riquelme, Basil Mustafa, Cedric Renggli, André Susano Pinto. Syl- vain Gelly, Daniel Keysers, and Neil Houlsby. Scalable transfer learning with expert models. arXiv:2009. 13239, 2020.
Maithra Raghu, Chiyuan Zhang, Jon Kleinberg, and Samy Bengio. Transfusion: Understanding transfer learning for medical imaging. In Neural Information Processing Systems (NeurIPS), 2019.
10

---

Under review as a conference paper at ICLR 2021
Esteban Real, Jonathon Shlens, Stefano Mazzocchi, Xin Pan, and Vincent Vanhoucke. YouTube- BoundingBoxes: A large high-precision human-annotated data set for object detection in video. In Computer Vision and Pattern Recognition (CVPR), 2017.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet classifiers generalize to ImageNet? In International Conference Machine Learning (ICML), 2019.
Giovanni Seni and John F Elder. Ensemble methods in data mining: improving accuracy through combining predictions. Morgan & Claypool Publishers, 2010.
Vaishaal Shankar, Achal Dave, Rebecca Roelofs, Deva Ramanan, Benjamin Recht, and Ludwig Schmidt. Do image classifiers generalize across time? ICML Workshop on Deep Phenomena, 2019.
Asa Cooper Stickland and lain Murray. Diverse ensembles improve calibration. arXiv:2007.04206 2020.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable ef- fectiveness of data in deep learning era. In International Conf. on Computer Vision (ICCV), 2017.
Shiliang Sun, Zhijie Xu, and Mo Yang. Transfer learning with part-based ensembles. In Multiple Classifier Systems, 2013.
Chuanqi Tan, Fuchun Sun, Tao Kong, Wenchang Zhang, Chao Yang, and Chunfang Liu. A survey on deep transfer learning. In International Conf. on Artificial Neural Networks (ICANN), 2018.
Bastiaan S. Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling, Rotation equiv- ariant CNNs for digital pathology. In Medical Image Computing and Computer Assisted Inter- vention (MICCAI), 2018
Andrew M. Webb, Charles Reynolds, Wenlin Chen, Henry Reeve, Dan-Andrei Iliescu, Mikel Lu- jan, and Gavin Brown. To ensemble or not ensemble: When does end-to-end training fail? In Computer Vision and Pattern Recognition (CVPR), 2019.
Yeming Wen, Dustin Tran, and Jimmy Ba. Batchensemble: An alternative approach efficient ensemble and lifelong learning. In International Conf. on Learning Representations (ICLR), 2020.
Florian Wenzel, Jasper Snoek, Dustin Tran, and Rodolphe Jenatton. Hyperparameter ensembles for robustness and uncertainty quantification. urXiv:2006.13570 2020.
Martin Wistuba, Nicolas Schilling, and Lars Sehmidt-Thieme. Automatic Frankensteining: Creating complex ensembles autonomously. In International Conference on Data Mining, 2017.
João C. Xavier-Júnior, Alex A. Freitas, Antonio Feitosa-Neto, and Teresa B. Ludermir. A novel evo- lutionary algorithm for automated machine learning focusing on classifier ensembles. In Brazilian Conference on Intelligent Systems (BRACIS), 2018.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In Neural information processing systems (NeurIPS), 2014.
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, Lucas Beyer, Olivier Bachem, Michael Tschannen, Marcin Michalski, Olivier Bousquet, Sylvain Gelly, and Neil Houlsby. A large-scale study of representation learning with the visual task adaptation benchmark. 1wXiv:1910.04661 2019.
11


Reviews:
[{"review_id": "m4ZmWnuGnpk", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper does achieve good performance but its method is quite about engineering using very intuitive training tricks that everybody could be able to use given a lot of GPU machines. I would not like to encourage such work to be published as a research paper.\n\nPros:\n\n1. The proposed framework achieves a good performance compared to its related works.\n\n2. It is a good organization of a lot of training techniques, and a good reference for engineering.\n\nCons:\n\n1. No technique contribution. The main framework of this submission is very similar to the existing work [Scalable Transfer Learning with Expert Models] which has not been officially published but only on arXiv. Besides the common methods of pre-training and ensembling, it involves three \"new\" methods in its main framework: the first one is kNN selection on pre-trained models (referred to the same technique in the work [Scalable Transfer Learning with Expert Models]); the second is the hyperensembles by fine-tuning multiple diverse copies of the models (referred to the hyperparameter sets used in another related work [Big transfer (BiT): General visual representation learning]); and the last is greedy ensemble (referred to the third related word [Ensemble selection from libraries of models]). Not sure what is the contribution of this submission.\n\n2. The paper is quite about engineering tricks or combinations of tricks. In addition, in terms of engineering, it is not fair to compare to related methods under the condition of using the same numbers of pre-trained models. A better way may be based on the total computational COSTS such as the max running epochs, the network architectures, the total training time under the same usage of GPU machines.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "drXRBBkGeKk", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Update after author response: I appreciate the authors' efforts to address my concerns and to raise some interesting points I missed. I still find the paper's insights are lacking some novelty to be published, but I think that this line of research is worth it! \n\n\n------------------------\n\n\nIn the low data regime, the use of transfer learning techniques collides with a widely used strategy: training multiple models for different purposes, being the main obstacle the lack of a clear diversity source.\n\nThis paper proposes a simple method to circumvent this problem: identifying pre-training itself as an easily accessible and valuable form of diversity and proposing the greedy combination of several pre-trained models.  Experiments show that the proposed strategy can achieve state-of-the-art performance on 19 different tasks. One necessary assumption of the method is the availability of a large pool of related models and the ability to look at the target data to make a decision on which models to fine-tune.\n\nOne of the critical points of the method is the use of cheap proxy metrics which assess the suitability of a pre-trained model before training it.  To this end, the paper proposes the use of leave-one-out nearest-neighbour accuracy.\n\nPros:\n+ The paper takes one of the most important issues of deep learning: training high-performance models in the low data regime. \n+ The results section is well structured and experiments are convincing. The proposed method is evaluated from several points of view.\n\nCons:\n- The paper refers to a previous publication (Puigcerver et al., 2020) and from this point of view, the proposal represents only an incremental step, with a low level of novelty. \n- The description of the method is not very specific and it refers to other existing methods as the main steps (Puigcerver et al. (2020) and  Caruana et al. (2004))\n- The proposed method is based on heuristics and there are no hints about why it does work. Diversity is a generic concept and there is a large number of papers that have explored several measures of diversity in order to understand \"when\" and \"why\" it is helpful. I miss some references to this previous knowledge. See, for example: Bian, Yijun, and Huanhuan Chen. \"When does Diversity Help Generalization in Classification Ensembles?.\" arXiv (2019): arXiv-1910.\n\nMy main concern is not about the results, which I think are good, but about the level of novelty with respect to some existing publications (mainly Puigcerver et al. (2020)) and the lack of experiments devoted to understanding the role of diversity. It is a well-known fact that diversity per se is not sufficient to build strong multiple classifiers, and different kinds of diversity measures are helpful to diagnose it.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "_lKBjNTlPJs", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "[Summary] This paper presents different ways of creating ensembles from pre-trained models. Specifically, authors first utilize nearest-neighbor accuracy to to rank pre-trained models, then fine-tune the best ones with a small hyperparameter sweep, and finally greedily construct an ensemble to minimize validation cross-entropy. Experiments on the Visual Task Adaptation Benchmark show the efficacy of the approach in selecting few models within a computational budget.\n\n[Score] Overall, I found the paper is well-written with experiments using large-scale benchmarks such as JFT, ImageNet21K and VTAB datasets. I like the problem of model selection for transfer learning. However, my major concern is about the novelty of the paper including concerns regarding prior works. Given the lack of novelty and convincing experiments, I vote for rejecting the paper. Hopefully the authors can address my concerns in the rebuttal period. \n\n[Weaknesses] The technical novelty of the paper is very limited. Besides combining few prior methods (e.g., Puigcerver et al. (2020); Caruana et al. (2004)) and then performing large scale experiments on JFT/ImageNet21K datasets, what are the main contributions of the paper are not clear. Although I admit that papers on analysis or study of different methods are quite interesting, I failed to find any major insights from the study of different diverse ensemble techniques. Is the upstream pre-training achieves better accuracy than that from the downstream fine-tuning stage the major take away message of the paper? Authors should clearly explain the major contributions of the paper.\n\nThere are few recent papers which discuss model selection for transfer learning. E.g., Duality Diagram Similarity: a generic framework for initialization selection in task transfer learning, ECCV 2020; DEPARA: Deep Attribution Graph for Deep Knowledge Transferability, CVPR 2020. How is the proposed approach related to these prior works? These paper should be clearly discussed with proper comparison in the experiments. \n\nComparison with prior methods is not satisfactory. Authors should clearly discuss what are the different ways of selecting models and creating ensembles out of that in the experiments. Specifically, what are the different alternatives to KNN and greedy approach used to construct ensembles? What about the performance of those methods? How is the proposed simple approach comparable to them in terms of performance vs complexity. E.g., how is the proposed approach comparable to the pretrained model selection strategy based on Task2Vec: see TASK2VEC: Task Embedding for Meta-Learning?\n\nHow is the proposed method related to Leep: A new measure to evaluate transferability of learned representations? Furthermore, how is the current approach comparable to a simple baseline on fine-tuning with early stopping?\n\nFigure 1 is not clear and it is not described clearly anywhere in the paper. I would like the authors to clearly explain this figure either in the caption or text in the introduction section.", "extra_scores": {}, "extra_text": {}}, {"review_id": "WZdsAdbDuSs", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nPaper proposed an ensemble learning approach for the low-data regime. Paper uses various sources of diversity - pre-training, fine-tuning and combined to create ensembles. It then uses nearest-neighbor accuracy to rank pre-trained models, fine-tune the best ones with a small hyper-parameter sweep, and greedily construct an ensemble to minimize validation cross-entropy. Paper claims to achieve state-of-the art performance with much lower inference budget. \n\nRecommendation: Based on my understanding of the paper I recommend a clear rejection. Please look at the details below: \n\nStrength: \n1) Authors have tried to lot of experiments and give summary of conclusion/results in section 4. \n\n2) Experimental setup is clear and the motivation is valid. \n\nWeakness/Questions: \n1) Paper was very hard to read. I had to go back and forth between pages to make sense of what’s defined and make my own definitions in many cases. In some cases, terms are defined but never used and in other cases terns are never defined. For example, \na) AugEnsembles: Where is this used?\nb) ExpertEnsembles: Where is this defined? \nc) HyperExperts: Where is this defined?\nd) AugExperts: Where is this defined? \n\n2) In figure 2, Single-model SOTA has only one model. Do you have a graph for total cost (training + inference) vs VTAB_{1K} performance for all the models that are shown in figure 2? Only showing an inference budget may not tell the entire picture here. \n\n3) In Table 2, how is computational cost different for different sources of diversity (D, U and C)? If C needs more computational cost than U and D then is the comparison fair? \n\n4) Appendix A.2 mentions the hyper parameters used when using “hyper ensembles” and then there is a default hyper parameter sweep - “Default Hyper Parameter Sweep” in appendix A.1. \nDid you find any pattern in the hyperparameters with the best model?  How were the hyperparameters chosen for baselines in table 1? \n\n\nminor: \n1) VTAB should have been defined just before listing contributions - \n“new form of diversity improves on the Visual Task Adaptation Benchmark (VTAB) SOTA by 1.8% (Zhai et al., 2019).\n\n2) Paper repeatedly cites Puigcerver et al 2020 [1] to justify experimental framework or as a follow up paper which is also very similar to the current paper in terms of motivation. \n\n[1] Puigcerver, Joan, Carlos Riquelme, Basil Mustafa, Cedric Renggli, André Susano Pinto, Sylvain Gelly, Daniel Keysers, and Neil Houlsby. \"Scalable transfer learning with expert models.\" arXiv preprint arXiv:2009.13239 (2020).", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers recommend that the paper be rejected.  The reviewers appreciate the line of research and is worthwhile, but find that the paper lacks in technical novelty and insight.  The AC is in consensus with their reviews due to the concerns raised regarding novelty and insight and recommends rejection.
Decision: Reject