Which Model to Transfer? Finding the Needle in the Growing Haystack
Authors: Cedric Renggli
André Susano Pinto
Luka Rimanic
Joan Puigcerver
Carlos Riquelme Ruiz
Ce Zhang
Mario Lucic
Venue:  (2021)

Abstract: Transfer learning has been recently popularized as a data-efficient alternative to training models from scratch, in particular in vision and NLP where it provides a remarkably solid baseline. The emergence of rich model repositories, such as TensorFlow Hub, enables the practitioners and researchers to unleash the potential of these models across a wide range of downstream tasks. As these repositories keep growing exponentially, efficiently selecting a good model for the task at hand becomes paramount. We provide a formalization of this problem through a familiar notion of regret and introduce the predominant strategies, namely task-agnostic (e.g. picking the highest scoring ImageNet model) and task-aware search strategies (such as linear or kNN evaluation). We conduct a large-scale empirical study and show that both task-agnostic and task-aware methods can yield high regret. We then propose a simple and computationally efficient hybrid search strategy which outperforms the existing approaches. We highlight the practical benefits of the proposed solution on a set of 19 diverse vision tasks.

Under review as a conference paper at ICLR 2021
WHICH MODEL TO TRANSFER? FINDING THE NEEDLE IN THE GROWING HAYSTACK
Anonymous authors Paper under double-blind review
ABSTRACT
Transfer learning has been recently popularized as data-efficient alternative training models from scratch, in particular in vision and NLP where it provides a remarkably solid baseline. The emergence of rich model repositories, such as Ten- sorFlow Hub, enables the practitioners and researchers to unleash the potential of these models across a wide range of downstream tasks. As these repositories keep growing exponentially, efficiently selecting good model for the task at hand be- comes paramount. We provide formalization of this problem through a familiar notion of regref and introduce the predominant strategies, namely task-agnostic (e.g. picking the highest scoring ImageNet model) and task-aware search strate- gies (such as linear or kNN evaluation). We conduct large-scale empirical study and show that both task-agnostic and task-aware methods can yield high regret. We then propose simple and computationally efficient hybrid search strategy which outperforms the existing approaches. We highlight the practical benefits the proposed solution set of 19 diverse vision tasks.
INTRODUCTION
Services such as TensorFlow Hut or PyTorch Hub' offer a plethora of pre-trained models that often achieve state-of-the-ard performance specific tasks in the vision and NLP domains. The predominant approach, namely choosing a pre-trained model and fine-tuming it to the downstream task, remains very strong and data efficient baseline. This approach the pre-training task is similar to the target task, but also across tasks with seemingly differing char- acteristics, such as applying an ImageNet pre-trained model to medical applications like diabetic retinopathy classification (Oquab et al., 2014). Fine-tuning often entails adding several more layers to the pre-trained deep network and tuning all the parameters using a limited amount of down- stream data. Due to the fact that all parameters are being updated, this process can be extremely costly and intensive in terms of compute (Zhai et al.) 2019). Fine-tuning all models to find the best performing one is rapidly becoming computationally infeasible. A more efficient alternative is to simply train a linear classifier or a k-nearest neighbour (kNN) classifier on top of the learned repre- sentation (e.g. pre-logits). However, the performance gap with respect to fine-tuning be rather large (Kolesnikov et al. 2019 Kornblith et al. 2019).
only successful when
In this paper we study the application of computationally efficient methods for determining which model(s) one should fine-tune for given task at hand. We divide existing methods into two groups: (a) task-agnostic model search strategies which rank pre-trained models independently of the downstream task (e.g. sort by ImageNet accuracy, if available), and (b) task-aware model search strategies- which make use of the provided downstream dataset in order rank models (e.g. kNN classifier accuracy as a proxy for fine-tuning accuracy) (Kormblith et al. 2019 Meiseles & Rokach 2020 Puigcerver al., 2020).
Clearly, the performance of these strategies depends on the set of models considered and the com- putational constraints of the practitioner (e-g. the memory footprint, desired inference time, etc.). To this end, we define several model pools and study the performance and generalization of each strategy across different pools. In particular, we make sure that these pools contain both "gener- alist" models (e.g. models trained on ImageNet), but also "expert" models (e.g. models trained on domain-specific datasets, such as flowers, animals, etc.).
https://ttie dev and https:://ytorrchhogg////

---

Under review as a conference paper at ICLR 2021
UPSTREAM TRAINING
MODEL SEARCH
DOWNSTREAM TRAINING
DATASETS MODELS "0"
RESTRICTED POOL
MAX
acc
Number parameters
BEST MODEL
RETURNED MODELS
FINE-TUNING RETURNED MODELS
bby stotlgg)
PRE - TRAINING
LINEAR eval
KNN eval
FINE- TUNING
Init(O)
init(Wa)) init(W
initt)
Figure 1: Transfer learning setup: (1) Upstream models Pre-training of models from randomly initialized weights on the (large) upstream datasets; (2) Model search Either downstream task in- dependent or by running a proxy task, i.e. fixing the weights of all but the last layer and training a linear classifier or deploying a kNV classifier on the downstream dataset; (3) Downstream training Unfreezing all the weights, optimizing both the pre-defined ones and a new linear classification layer on the downstream dataset.
We formally define and motivate the model search problem through a notion of regret. We conduct the first study of this problem in a realistic setting focusing on heterogeneous model pools. (ii) We perform a large-scale experimental study by fine-tuning 19 downstream tasks on 46 models from a heterogeneous set of pre-trained models split into 5 meaningful and representa- tive pools. (iii) We highlight the dependence of the performance of each strategy on the constrained model pool, and show that, perhaps surprisingly, both task-aware and task agnostic proxies fail (i.e. suffer large regret) on a significant fraction of downstream tasks. (iv) Finally, we develop hybrid approach which generalizes across model pools as a practical alternative.
Our contributions. (i)
BACKGROUND AND RELATED WORK
We will now introduce the main concepts behind the considered transfer learning approach where the pre-trained model is adapted to the target task by learning mapping from the intermediate
2018 Weiss et al. representation to the target labels (Pan & Yang 2009 Tan et al. 2018 Wang
2016), as illustrated in Figure T
Upstream training, or pre-training, refers simply to a procedure which trains model given task. Given the variety of data sources, losses, neural architectures, and other design decisions, the set of upstream models provides a diverse set of learned representations which can be used for a downstream task. In general, the user is provided with these models, but can not control any of these dimensions, nor access the upstream training data. Previous work call the models in these pools specialist models (Ngiam al. 2018) or experts (Puigcerver al. 2020).
(I) Upstream models.
(II) Model search. Given no limits on computation, the problem is trivial - exhaustively fine-tune each model and pick the best performing one. In practice, however, one is often faced with stringent requirements on computation, and more efficient strategies are required. The aim of the second stage in Figure is therefore select a small number of models from the pool. so that they can be fine- tuned in the last step. The central research question of this paper is precisely how we can choose the most promising models for the task at hand. We divide existing related work and the methods we focus on later in this work into three categories as illustrated in Figure 2
(II A) Task -ansotannoticc search strategies. These strategies rank models without looking at the data of the downstream task (Kornblith et al. 2019). As consequence, given fixed pool of candidates, the same model is recommended for every task. We focus on the following popular task aask-agnostic approach: (i) Pick the highest test accuracy ImageNet model (if such model is in the pool), oth- erwise (ii) pick the one trained on the largest dataset. there is a tie, pick the biggest model in the pool (in terms of the number of parameters).
2

---

Under review as a conference paper at ICLR 2021
UPSTREAM TRAINING
METHODS for MODEL SEARCH
MODELS
(A) TASKAGNOSTIC
RANKING
(B) TASK-AWARE
USER'S DATASET
META
(C) META-LEARNED
META-DATA
ALGORITHM
TASK-AWARE
Model search methods: (A) aask-agnostic methods do not see the downstream task, pro- ducing the same ranking of models for all possible tasks (c.g.. using the highest ImageNet accuracy); (B) Task-aware methods deploy proxy (e.g.. linear evaluation) for each model on user's dataset, without referring to meta data used during pre-training; (C) Meta-learned task-aware methods de- ploy proxy for each model on user's dataset, together with meta data used during pre-training. For example, one could use some notion similarity between tasks, as in Achille al. (2019).
Figure 2:
(II B) Task-aware search strategies. In contrast to the task aank-agiostic approach, task-aware meth- ods use the downstream data, thus requiring additional computation. In this work first compute and freeze the representations extracted from the last layer, noting that this step requires a forward pass (i.e., running inference) for each example on every model in the pool. On top of that, one usually uses a classification method to solve the task with the new features provided by the pre- trained model. We choose two classifiers that are commonly used for this purpose: a linear and KNN classifier. We note that both are usually several orders of magnitude faster than fine-tuning the whole model, with &NN being non pana-armttric classifier that can capture non- non-linear relationships. Finally, we select the models that achieve highest accuracy in this simplified setup.
Alternative approaches such as LEEP (Nguyen et al., 2020). NCA (Tran al. 2019) or H- Score (Bao et al., 2019) replace the classifier's s accuracy by a somewhat cheaper estimator. These three approaches have clear limitation in supporting only those cases which a classification head for the original task is provided together with each pre-trained model. Furthermore, the theo- retical guarantees are given with respect to models trained on top of the frozen representations only. As observed with LEEP, albeit natural datasets only, fine-tuning achieves significantly better test accuracies than a linear classifier, which further invites for better understanding of using lin- accuracy proxy for fine-tuning, noting that linear correlations are not necessarily transitive. Furthermore, training& a single linear layer, or calculating the kNN accuracy on top of transformed representations, is typically much less computational demanding than running the inference itself, and both steps can easily be overlapped. Recently, Meiseles & Rokach (2020) introduced the Mean Silhouette Coefficient (MSC) to forecast model's performance after fine-tuning on time series data. We omit this approach due to its provable relation to a linear classifier proxy, whilst being similar ANN with respect to capturing non- non-linear relationships. Finally, Puigcerver al. (2020) utilize kNN as a cheap proxy task for searching models in a set of experts with the same architecture
C) Meta-learned task-aware search strategies. In the last category, we allow model search strategies to utilize meta-data about the pre-trained models in order to more efficiently rank models. Such strategies typically contain a meta-learning part that precedes the actual search component. The meta-learned dependencies between models can be used in computational (Achille et al.) 2019) or semantical (Zamir et al. 20188) Song et al. 20200 Dwivedi et al. 2020) way. Model2 Vec, generalization of Task2 Vec/chlll et al. 2019), learns embedding for each pre- pre-trained model as combination of a task vector and model bias. It minimizes a cross-entropy loss based on the accuracy achieved the source dataset by the corresponding models. Given part is performed by finding the nearest model embedding given the computed task embedding for the downstream task. This step is computationally cheap it requires training only single probe-network in order to compute the new task embedding. Therefore. the overall compute time resides with the owner of model repositories. We omit Model2Vee for the remainder of the paper due to limited understanding and computational burden of the incremental re-training when adding a new pre- trairrd model to the pool, and the less practical use case for the current users of available pre-trained model repositories. Having said that, in the final discussion section of this work we address how Model2Ves could benefit from our proposed strategies, hoping that this could open an interesting and fruitful followup research area.
task, the model search

---

Under review as a conference paper at ICLR 2021
(III) Downstream training. In this stage, the selected model is adapted to the downstream task (cf. Figure [T]. The predominant approach is to fully or partially apply the pre-trained neural network as feature extractor. The head (e.g. last linear layer) of the pre-trained model is replaced with new one, and the whole model is trained on the target data. This process is commonly referred to as fine-funing and it often outperforms other methods (Oquab et al.] 2014 Donahue et al. 2014
Sharif Razavian et al. (2114] Kornblith et al. 2019).
3 COMPUTATIONAL BUDGET AND REGRET
The main aim of this work is the study of simple methods to filter and search pre-trained models before stepping into the -more expensive- fine-tuning process. Formally, we define a search method m(M, D) with budget B as a function which takes a set of models M and downstream dataset DD as input, and outputs a number of distinct models Sm S M. with [mm] = B. Those B models are then all fine-tuned in order to obtain the best possible test accuracy on the downstream task D.
Budget and regret. Fine-tuning represents the largest computational cost; accordingly, we define the number of models that are fine-tuned as the computational complexity of a given method, Given any fixed budget B, we would like to return set S which includes the models resulting in good performance downstream. In particular, we define the notion of absolute regret of a search strategy m and pool of models M on dataset D as
max Elt(m;, D)J - E EM
max (85,, D)
(I)
S.
ORACLE
<(m)
where t(m, D) is the test accuracy achieved when fine-tuning model m on dataset D. The first expectation is taken over the randomness in the t(-) operator, that is, the randomness in fine-tuning and due a finite sampled test set. In addition the randomness in t(-), the second expectation also accounts for any potential randomization in the algorithm that computes Sn- We define s s(m) as the expected maximal test accuracy achieved by any model in the set Sm, the set of models returned by fixed strategy m. In our case, kNN is deterministic as all the downstream data is used, whereas the linear model depends on the randomness of stochastic gradient descent. To enable comparability between datasets of different difficulty as well as comparison between two selection strategies mi, and m2, we define their relative delta as
s(m1) s(m2)
A(m1, m2)
min(s(m1)) s(m)))
with s(.) E [0, 1] defined in Equation
Substituting s(m) by the ORACLE value, and s(m,) by
s(m) leads to the relative regret r(m). We discuss the impact of alternative notions in Section 5.4
4 EXPERIMENTAL DESIGN
Our goal is to assess which model search strategies achieve low regret when presented with a diverse set of models. As discussed, there are three key variables: (i) The set of downstream tasks, which serve as a proxy for computing the expected regret of any given strategy, (ii) the model pool, namely the set explore to find low-regret models, and (iii) the transfer-lcaminn algorithms.
4.1 DATASETS AND MODELS
Datasets. We use IK, a few-shot learning benchmark composed of 19 tasks partitioned into groups natural, specialized, and structured (Zhai et al., 2019). The natural image tasks include images of the natural world captured through standard cameras, representing generic objects, fine-grat ine-grained classes, or abstract concepts. Specialized tasks contain images captured using specialist equipment, such as medical images or remote sensing. The structured tasks are often derive from artificial environments that target understanding of specific changes between images, such as predicting the distance to an object in 3D scene (e.g. DeepMind Lab), counting objects (e.g. CLEVR), or detecting orientation (e.g. dSprites for disentangled representations). Each task has 800 training examples, 200 validation examples, and the full test set. This allows to evaluate model search strategies on a variety of tasks and in a setting where transfer learning offers clear benefits with respect to training from scratch (Zhai et al. 2019).
4

---

Under review as a conference paper at ICLR 2021
Models. The motivation behind the model pools is to simulate several use-cases that are ubiquitous in practice. We first collect 46 classification models (detailed overview in Appendix A
15 models trained the ILSVRC 2012 (ImageNet) classification task (Russakovsky et al, 2013) including Inception VI-V3 models (Szegedy et al. 2016), ResNet V1 and V2 (depth 50, 101, and 152) (He et al. 2016), MobileNet V1 and V2 (Howard et al. 2017). NasNet (Zoph)
et al., 2018) and PNasNet (Liu et al. 2018) networks.
16 ResNet-50-V V2 models trained on (subsets of) JFT (Puigcerver et al. 2020). These models
are trained on different subsets of a larger dataset and perform significantly better on a small subset of downstream tasks we consider (i.e. they can be considered as experts).
15 models from the VTAB benchmark with a diverse coverage of losses (e.g. generative, self- supervised, self-supervised combined with supervised, etc.) and architectures. In all cases the upstream dataset was ImageNet, but the evaluation was performed across the VTAB benchmark which does not include ImageNet.
4.2 MODEL POOLS
(A) Identifying good resource-constrained models (RESNET-50, DIM2048). Here we con- sider two cases: (i) RESNET-50: All models with the number of parameters smaller or equal to ResNet50-V2 V2. While the number of parameters is clearly not the ideal predictor, this set roughly captures the models with limited memory footprint and inference time typically used in practice. Most notably, this pool excludes the NasNet and PNasNet architectures, and includes the expert models. (ii) DIM2048: The transfer strategies discussed in Section 2 might be sensitive to the size of the representation. In addition, restricting the representation size common constraint in prac- tical settings. Hence, we consider a model pool where the representation dimension is limited to a maximum of 2048.
(B) Identifying expert models in presence of non-experts (EXPERT). We consider a pool of 16 ResNet-50-V2 models from Puigcerver et al. 2020). These models which considered as experts are trained on different subsets of a larger dataset. As the number of available models and the upstream training regimes increase, the number of such experts is likely to increase. As such, this presents a realistic scenario in which an expert for the target task may be present, but it is hard to identify due to the presence of other models, some of which might perform well on average.
(C) Do better agno-agiostic models transfer better CONNETACCURACIER the ability choose an upstream representation-leaming technique that is best suited for a specific downstream task. This pool is mainly used to validate the idea that (a) ImageNet models transfer well across different tasks (Huh et al. 2016 He al. 2019) and that (b) better ImageNet models transfer better (Kornblith et al. 2019).
This pool offers
(D) All models (ALI.). Finally, we consider the hardest setting, namely when the model pool contains all 46 models and no conceptual nor computational restrictions are in place. We note that: EXPERT RESNET-50 C DIM2048 C ALL and IMNETACCURACIES C ALL..
4.3 EVALUATION PROCEDURES
Fine tuning. To assign a downstream test accuracy to each pair of model and task, we use the median test performance of 5 models obtained as follows: (i) Add a linear layer followed by softmax layer and train model on all examples of the training set. (ii) Fine-tune the obtained model twice, considering two learning rates, and 2500 steps of SGD and a batch size of 512 (Zhai) jet al. 2019). (iii) Return the model with the highest validation accuracy. Note that this case, the entire model, and not only the linear layer, is retrained. As result, there are 10 runs for each model and we obtain 8740 trained models (46 x 19 x x 2).
Linear evaluation. We train logistic regression classifier added to the model representations (fixed) using SGD. We consider two learning rates (0.1 and 0.01) for 2500 steps and select the model with the best validation accuracy. For robustness run this procedure 5 times and take the median validation accuracy out of those resulting values. As a result, we obtain again 8740 models.
ANN evaluation. We compute the validation accuracy by assigning each of the 200 validation samples the label of the nearest training example (i.e. l = 1) using standard Fuclidean distance.
htttps:/tt dew.tbb..dee//aa.
5

---

Under review as a conference paper at ICLR 2021
ALL. (8 1,8=1.8= =18
RESNET-50 (8= 1.8
EXPERT (8
Relative Regret
1.00
1.00
100
0.75
0.75
0.50
0.25
0.00
0.00
Carmesi
CIFAR 100
Flow FLowmsst mnst e2
PRx
Suavir
SVIN
ON
Bunn8
Clavi Cocort
Cumme DMLL
KETTI Det
SNOR As0u
SNORE Eaay
Caunsont
CIPAR-10 100
DTD
SVIN
Cammoro
DUROSAT
Rasnels
Cureaa Coo
uimm Oniiet
KITTI-DO
SNORU- Ann
SNORB- ELEv
Cacmonio
DTD
FLow Euowsott
PETs
SVRN
Curee Daaee--rrew Conner
Cunnn Det
DMLLA
ANOREL
SNORD
Figure 3: Task-agnostic strategies. Relative regret (r(m), ef. Section 3 with B (transparent)
and B 2 (solid) the ALL. RESNET-50 and EXPERT pools, bearing in mind that there is only one aagossii model in EXPERT. By defimition, task-agnostic strategies exclude experts yielding high regret on the RESNET-50 and EXPERT pools, particularly on natural or structured datasets.
ALL
RESNET-50
EXPERT
Relative Regret
1.00
0.75
0.50
0.50
0.25
0.25
0.00
CIEAR 100
FLOWERSUD
Para
Susalit
Clavos Couor
Cuevp
DMLss
aSyn-Loo
Kitr Der
NNDR AMM SVORE Elav
Comeenner CIFAR-100
D FLowresid
ErsosAT
Ramicin
Bh Recronottt poneatt
CLavs Cocer
DMLxs
uSm Ohuperr
RITTIE TETTTLI DiDer
SNORI- Jas
SNORB- ELe
0.00
Camreaii DTD
BLowzeei Booweeeti od
Sustin
SVBS
Cusce-C Baacs-Co Des
Couser
Cherne
oSe Semiaa
SNORB ARF
ORD
8
Figure 4: Task-aware strategies (linear). Relative regret for = 1 (transparent) and B (solid)
on the ALL, RESNET-50, and EXPERT pools. Compared to task-agnostic strategies, observe improvement on natural datasets (except SVHN) and on restricted pools (except DSPR-LOC), due to its ability to properly choose experts.
KEY EXPERIMENTAL RESULTS
In this section we challenge common assumptions and highlight the most important findings of this study, whilst the extended analysis containing all the plots and tables can be found in the supplemen- tary material. We remark that in the main body we only consider three main pools of models ALL., RESNET-50 and EXPERT, very similarly to RESNET-50, whereas INNETACCURACHO Kornblith et al. (2019), the results of ablation studies involving these two pools can be found in Appendix D Finally, in this section we mainly investigate linear evaluation as the aaske choice; all the corresponding plots for &NN can be found in Appendix E
see them the most representative ones. Since DIM2048 behaves is used only to confirm the findings
5.1 HIGH REGRET OF TASK- AGNOSTICI STRATEGIES
Figure shows the results for task aask-agnostic methods with budget B 1 and B on the ALI. RESNET-50, and EXPERT pools. We observe significant regret, in particularly for RESNET-50 and EXPERT pools (30% of the datasets have a relative regret larger than 25% on those two pools). This highlights the fact that task-agnostic methods are not able to pick expert models, in particular on natural and structured datasets. As more experts become available, this gap is likely to grow, making it clear that task-agnostic strategies are inadequate on its
5.2 ARE TASK-AWARE ALWAYS A GOOD PREDICTOR?
Intuitively, having access to the downstream dataset should be beneficial. We evaluate both the linear and the &NN predictor as detailed in Section 4 Figure 4 provides overall results for the linear model, whereas analogous results for ANN are presented Appendix The method struggles on some structured datasets (in particular LOOR-
Compared to the task-agnostic strategy, as presented in Figure 5 for = 1, we observe significant improvements restricted model pools. The EXPERT pool benefits the most: linear evaluation

---

Under review as a conference paper at ICLR 2021
ALL
RESNEr-50
EXPERT
Relative Delta
1.0
1.0
TLomaes 102
Cheen- Cunvn-Cour CoUNT
Pitts
CAurOO
DYTD
CLavs Den---ee
ANDRUL-RUF
Rnstoond
CIFAR-180
KITTE DIST
SNORE- RRE-AAD AnD
DXLL
BERORAT
SVRN
USre- Loc
TLowse
Purs
Cuarn Conn-Co
KITTE Der
DTD
ANORB--RU RuFy
Clack, Diev
DMLA30
Camneni
CIFAI 100
SNORE Ann
18in-Ours
BEROAAT
aSPs-1oc
SVIN
FLowsr 102
DSPn Loc
Puts
CuRvR- Count
SXTTR
BrmoSAT
UNDRE-EBUE
Resport
508397
REmOPPTTI
08P-- ORISKT
DMLLas
CIEAR 100
DTD
SXORI- Autu
Clacke Dues--De Dirt
KITTE-
Figure 5: Task- -ask-tgnottic (positive if better) vs Task-aware (linear) (negative if better) for B = 1. On the ALL pool, the methods perform in similar fashion, with respect to the number of datasets and the amount in which one outperforms the other. When one restricts the pool to RESNET-50 or EXPERT task-aware methods outperform the task-agnostic method most datasets. The relative delta is defined in Equation in Section
ALL
RESNET-50
EXPERT
Relative Delta
1.0
0.4
0.15
0.2
0.10
0.5
0.0
0.05
0.2
0.00
DTD
CIFAR
Prowers
ANORB-FUU
Curva Covers
ANDRD And
KITT RITTLE
fuseeces 5
DMLLAn
E ROSOAAT SVHS
-04
DTD
EisosAT
Clarne Dart
Sindlr
Ber RETTOOPATTS Prssoett
Part
KITT
FLOWEEd
DMLL
Cleve Conser
Clsisenn venn in
SNORB Eus Eus
A Aupa
ENOREE
-00
DSP
Claven Dss
RET RESIORRAATE
C- D1
SUNAFF EVHN
SNORB- Eurv
Flowars 100
DMLLxa DTD
Ouave- Cnui-r
ANORI- KITTI L Day
Figure 6: Hybrid linear (positive if better) vs Linear evaluation (negative if better) for B = 2. We observe that hybrid linear significantly outperforms linear with the same budget on the ALL pool. Even though for RESNET-50 and EXPERT pools there are datasets on which linear performs better than hybrid, the amounts in which it does are usually small. We note that most significant gains of hybrid come on certain structured datasets, the hardest task for every strategy.
outperforms task agnostic methods on almost every dataset (task-aware is only outperformed on three datasets by more than 1%, and by 10% in the worst case on the KITTI-DIST dataset). On the other hand, task-agnostic and task-aware strategies seem to outperform each other on a similar number of datasets and by comparable magnitude in the ALL pool. This suggests that no single strategy uniformly dominates all other strategies across pools.
In order to understand this further, we perform an ablation study where we plot the linear and ANN regret on the IMNETACCUR ACCURACIES pool Appendix [ In Figures [5] and T6 we observe that task-aware search methods perform rather poorly when having access only to different architectures trained on the same upstream data. The IMNETACCURACIES models are included in the ALL pool, and in some datasets some of those models the best-performing ones.
Performance of the ANN predictor is on par on half of the datasets across the pools, and slightly worse than linear evaluation the other half. We present these findings in Figure 20 Appendix
5.3 HYBRID APPROACH YIELDS THE BEST OF BOTH WORLDS
A hybrid approach that selects both the top-1 task-agnostic model and the top- - 1) task-aware models leads to strong overall results. Figure(dJshows how the hybrid approach with linear evaluation as the task-aware method significantly outperforms its linear counterpart with B 2. This is most noticeable in the ALL. pool where the task-agnostic model provides a large boost on some datasets.
As we saw in Figure 5 when looking at the ALL pool, the task-agnostic candidate tends to beat the linear one on datasets such as DSPR-LOC, SVHN or EUROSAT. Similarly, the linear candidate model clearly outperforms its task-agnostic counterpart on many natural datasets such as FLOWERS or PETS. A comparison of Figures 5 and 6 reflects how the dominance of linear-only strategy vanishes on most datasets when confronted with the hybrid approach. For the RESNET-50 and EXPERT pools, expected, the hybrid algorithm preserves the good picks of the linear proxy. That said, we observe an increase of 36% on OSPR-LoG in the RESNET-50, and 11% on KITTI-DIST. Both structured datasets which the linear proxy task performs poorly, as shown in Figure 4
7

---

Under review as a conference paper at ICLR 2021
Linear
Hybrid Linear
Task- Taak-Agnsticc Oracle
ALL
RESNET-50
EXPERT
Optimal Picks
0.50
0.25
Budget
Budget
Budget
Figure 7: Optimal picks as a function of the computational budget. The number of picked models (relative) with zero regret across three representative pools. We note that hybrid linear outperforms all other methods on ALL, whilst being comparable with the linear strategy on restricted pools where linear alone already performs well. Here, the task-agnostic oracle refers to a method which ranks models based on their average accuracy across all datasets (more details Section 5.4).
The hybrid strategy requires to fine-tune at least two models. Given that it performs well across all model pools and datasets, this reasonable price to pay in practice, and suggest its use as the off-the-shelf approach. Figures 21 and 22 in Appendix E depict the results for kNN. In EXPERT models, the second &NN pick tends beat the task-agnostic one hurting the KNN hybrid outcomes. Overall, the hybrid linear approach consistently outperforms the one based on kNN.
5.4 FURTHER ABLATION STUDIES
How does the computational budget impact the findings? We have seen that for a limited bud- get of B = 2 the proposed hybrid method outperforms the other strategies. A natural question that follows is: how do these methods perform as a function of the computational budget B? In particu- lar, for each budget B, we compute how frequently does strategy pick the best model. The results are shown in Figure 7 We observe that the hybrid linear strategy outperforms all individual strate- gies the ALL pool. Furthermore, it also outperforms strong impractical task-agnostic oracle which is allowed to rank the models by the average fine-tune accuracy over all datasets. Our hybrid strategy achieves on par performance with the linear approach on pools on which linear performs well. When task-aware strategies perform badly (e.g. pools without expert models), hybrid linear is significantly stronger (cf. Figure 17 in Appendix D). These empirical results demonstrate that the hybrid strategy is simple yet effective practical choice.
Alternative evaluation scores. Both Meiseles Rokach (0220) and Kornblith al. (0119) com- pute the correlation between the ImageNet test accuracy and the average fine-tune accuracy across datasets. Although this provides good task-agnostic evaluation method for the average perfor- mance (cf. Figure/ 1, right, in Kornblith et al. (2019)), it can be significantly impacted by outliers that have poor correlations on a specific dataset (cf. Figure 2, middle row, in Kornblith et al. (2019). In Appendix B we highlight another limitation of using correlation scores in the context of model- search strategies across heterogeneous pools. Nevertheless, we empirically validate that ranking the models based their ImageNet test accuracy on the IMNETACCURACIES, pool transfers well to our evaluation setting (ef. Figure [4] in the Appendix [D.. Furthermore, we show that reporting the differences of logit-transformed accuracies (log-odds) leads similar conclusions as ours (ef. Appendix C). We opt for the relative regret r(m), defined in Section 3 as it is more intuitive and
contained In -1, 1].
Impact of the ANN hyperparameters. The kNN classifier suffers from the curse of dimensional- ity (Snapp et al. 1991), which why we study the impact of the dimension (i.e. the representation size) on the ANN evaluation. We fix dataset, and plot a model's kNN score versus its representation dimension. In order to have a single point per dimension and avoid an over-repressntation of the expert models that are all of the same architecture, we choose the model with the best &NN accuracy. By calculating the Pearson correlation coefficient between the dimension and the respective ANN scores, we observe a moderate anti-correlation (Rt -0.5) for 3 datasets, moderate correlation (R > 0.5) for other datasets, and either small or no correlation for the remaining 13 datasets. Based on this empirical evidence conclude that there is significant correlation between the ANN classifier accuracy and the dimension. We provide more details in Figure 23 of Appendix F Regarding k; our preliminary experiments with k = offered no significant advantages over h = T.
8

---

Under review as a conference paper at ICLR 2021
6 OTHER RELATED WORK
Given access to the meta-data such as the source datasets, one could compare the upstream and downstream datasets (Bhattacharjee et al. 2020); blend the source and target data by reweighting the upstream data to reflect its similarity to the downstream task (Ngiam et al., 2018); or construct a joint dataset by identifying subsets of the upstream data that are well-aligned with the downstream data (Ge & Yul2017) These approaches are restricted to one model per upstream dataset, while also being less practical they necessitate training a new model as well as access to upstream datasets which might be unavailable due to proprietary or privacy concerns.
Another branch of related work given by Zamir et al. (018). Taskonomy provides an approach that finds the best source dataset(s) by exhaustively exploring the space of all possibilities. However, it is not-comparable to our setting for two reasons: (i) the input domain and data is assumed to remain constant and task are only different in their labels, (ii) the chosen architecture keeps the weights of the encoder frozen, and the decoder trained on top usually consists of multiple fully connected layers, opposed to our fine- toning regime with single linear layer top. Improvements by [Song et al. (2020) and Dwivedi et al. (2020) make taskonomy faster, but they still do not distinguish multiple models trained on the same dataset, nor bypass the constraints described previously.
Finally, best-arm identification bandits algorithms suggest the successive elimination of sub-optimal choices (Even-Dar et al. 2006) derived by fine-tuning models for shorter time. This combines model selection and downstream training, noting that its computation and theoretical properties heavily depend on the architectures of pre-trained models well as hyper-parameters used therein (e.g.. batch norms), which is still far from being understood by the community. Therefore, this line of work including partial fine-tuning as a proxy task is deliberately omitted in this work.
7 CONCLUSIONS. LIMITATIONS, AND FUTURE WORK
Transfer learning offers data-efficient solution train models for range of downstream tasks. As we witness an increasing number of models becoming available in repositories such as TensorFlow Hub, though, finding the right pre-trained models for target tasks is getting harder. Fine-tuning all of them is not an option. In practice, the computational budget is limited and efficient model search strategies become paramount. We motivate and formalize the problem of efficient model search through a notion of regret, and argue that regret is better suited to evaluate search algorithms than correlation- based metrics. Empirical evaluation results for the predominant strategies, namely task- agnostic and task-aware search strategies, are presented across several scenarios, showing that both can sometimes yield high regret. For any individual method we study, there exists pool of models on which the method fails. Finally, we propose a simple and computationally efficient hybrid search strategy which consistently outperforms the existing approaches over 19 diverse vision tasks and across all the defined model pools.
Limitations and future work. To further stress-test the generalization of analysed strategies, the number of relevant model pools could be increased by incorporating more diverse upstream tasks, in particular neural architectures, losses, and datasets. This would potentially yield more expert models making the task even more challenging, and it could further highlight the advantages of an effective search strategy. Secondly, we observe that taske methods consistently perform poorly in specific cases, such as when we consider diverse architectures trained only on ImageNet. There is no obvious reason for such failures. Similarly, there seems to be clear pattern where task-aware methods perform significantly worse on structured datasets than on natural ones. We hypothesise that this is due to the lack of adequate expert models for these domains. However, an in-depth analysis of these specific cases might be beneficial and insightful. Thirdly, despite the fact that we did not include meta-learned task-aware search strategies on purpose, we believe that the hybrid strategy would significantly improve the performance of Vode12Vec, and hence Task2Vec (Achille) ct al. 2019), or other meta- leamnd task-aware search strategies in general. For example, in Figure 3 in Achille et al. (2019) the expert selection procedure is outperformed by the simple generalist approach in 15 out of 50 tasks, sometimes by large margin.
Finally, transfer learning is a successful strategy in various natural language processing tasks, which did not explore here due to the lack of diverse sources of text representation and heterogeneous pools in online repositories. Going beyond vision tasks presents a very clear research direction.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Char- less C Fowlkes, Stefano Soatto, and Pietro Perona. Task2vec: Task embedding for meta-learning. Proceedings of the IEEE International Conference Computer Vision, pp. 6430-6439, 2019.
Yajie Bao, Yang Li, Shao-Lun Huang, Lin Zhang, Lizhong Zheng, Amir Zamir, and Leonidas Guibas. An cotrmattion-theoretee approach transferability in task transfer learning. 2019 IEEE International Conference on Image Processing (ICIP), pp. 2309-23- 2309-2313, 2019.
Bishwaranjan Bhattacharjee, John R Kender, Matthew Hill, Parijat Dube, Siyu Huo, Michael R Glass, Brian Belgodere, Sharath Pankanti, Noel Codella, and Patrick Watson. P21: Predicting transfer learning for images and semantic relations. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 7600761, 2020.
Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Dar- rell. Decaf: A deep convolutional activation feature for generic visual recognition. International Conference on Machine Learning, 2014.
Kshitij Dwivedi, Jiahui Huang, Radoslaw Martin Cichy, and Gemma Roig. Duality diagram sim- ilarity: a generic framework for initialization selection in task transfer learning. Proceedings of the European Conference on Computer Vision (ECCV), 2020.
Eyal Even-Dar, Shie Mannor, and Yishay Mansour. Action elimination and stopping conditions for the multi-armed bandit and reinforcement learning problems. Journal of Machine Learning Research, 2006.
Weifeng Ge and Yizhou Yu. Borrowing treasures from the wealthy: Deep transfer learning through selective joint fine-tuning. IEEE Conference on Computer Vision and Pattern Recognition, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. IEEE Conference on Computer Vision and Pattern Recognition, 2016.
Kaiming He, Ross Girshick, and Piotr Dollár. Rethinking imagenet pre-training. IEEE Conference on Computer Vision and Pattern Recognition, 2019.
Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andrectto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint (iXxx:170.004860 2017.
Minyoung Huh, Pulkit Agrawal, and Alexei A Efros. What makes ImageNet good for transfer 08.14, 2016.
learning? arXiv preprint
Alexander Kolesnikov, Xiaohua Zhai, and Lucas Beyer. Revisiting self-supervised visual represen- tation learning. IEEE Conference on Computer Vision and Pattern Recognition, 2019.
Simon Kornblith, Jonathon Shlens, and Quoc V Le. Do better Imagenet models transfer better? IEEE Conference on Computer Vision and Pattern Recognition, 2019.
Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. European Conference on Computer Vision, 2018.
Amiel Meiseles and Lior Rokach. Source model selection for deep learning in the time series domain. IEEE Access, 2020.
Jiquan Ngiam, Daiyi Peng, Vijay Vasudevan, Simon Kornblith, Quoc V Le, and Ruoming Pang. Do- main adaptive transfer learning with specialist models. arXiv preprint arXiv:1811.07056 2018.
Cuong V Nguyen, Tal Hassner, Cedric Archambeau, and Matthias Seeger. Leep: A new measure to evaluate transferability learned representations. International Conference on Machine Leam- ing, 2020.
10

---

Under review as a conference paper at ICLR 2021
Maxime Oquab, Leon Bottou, Ivan Laptev, and Josef Sivic. Learning and transferring mid-level image representations using convolutional neural networks. IEEE Conference on Computer Vision and Pattern Recognition, 2014.
Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering, 2009.
Joan Puigcerver, Carlos Riquelme, Basil Mustafa, Cedric Renggli, André Susano Pinto, Sylvain Gelly, Daniel Keysers, and Neil Houlsby. Scalable transfer learning with expert models. arXiv preprint arXiv: 09407000000 2020.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 2015.
Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson. CNN features off-the-shelf: An astounding baseline for recognition. IEEE Conference on Computer Vision and Pattern Recognition, 2014.
Robert Snapp, Demetri Psaltis, and Santosh Venkatesh. Asymptotic slowing down of the nearest-neighhor classifier. Advances in Neural Information Processing Systems, 1991.
Jie Song. Yixin Chen, Jingwen Ye, Xinchao Wang. Chengchao Shen, Feng Mao, and Mingli Song. Depara: Deep attribution graph for deep knowledge transferability. Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 3922-3930, 2020.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the Inception architecture for computer vision. IEEE Conference on Computer Vision and Pattern Recognition, 2016.
Chuanqi Tan, Fuchun Sun, Tao Kong, Wenchang Zhang, Chao Yang, and Chunfang Liu. A survey deep transfer learning. International Conference Artificial Neural Networks, 2018.
Anh T Tran, Cuong V Nguyen, and Tal Hassner. Transferability and hardness of supervised classi- fication tasks. Proceedings of the IEEE International Conference on Computer Vision, pp. 1395- 1405, 2019.
Zirui Wang. Theoretical guarantees of transfer learning. arXiv preprint arXiv:1810.0550 2018
Karl Weiss, Taghi M Khoshgoftaar, and DingDing Wang. A survey of transfer learning. Journal of Big data, 2016.
Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. IEEE Conference on Computer Vision and Paltern Recognition, 2018.
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. The Visual Task Adaptation Benchmark. arXiv preprint arXiv: 1910.04877. 2019.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. IEEE Conference Computer Vision and Pattern Recognition, 2018.
11


Reviews:
[{"review_id": "sw1CohP7e4o", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "[Summary] This paper presents a large-scale study on model-selection strategies for transfer learning, by performing task-agnostic and task-aware strategies on a large number of models evaluated on a diverse range of tasks. \n\n[Strength] The problem setting is novel and interesting. The proposed quantitative measurement of the quality of selected models, named \"regret\" is well designed.\n\n[Weakness] The major weakness of this paper is that it seems there is no consistent strategy to out-perform all other methods in every task. Intuitively, task-aware strategies should be better than task-agnostic strategies, but they perform similarly (almost equally) in all model pools, which is quite surprising.\n\nEven if for the advanced strategy hybrid proposed in the latter part of this paper, the optimal pick for this hybrid method is almost identical to the linear evaluation in task-aware strategy in ResNet-50 and expert model pools.\n\nSo my biggest concern for this paper is that we don't have a take-home message, other than showing the \"No-Free Lunch Theorem\" in model selection. So, I hope the authors could re-emphasize what we really learn from this large-scale study.\n\nAn interesting direction might be, how we can design a really fast approximation of fine-tuning, so that we can evaluate a model's fitness only by a few iterations (within a very short amount of training time), instead of performing full fine-tuning on the target task.\n\nConsidering this limited effective information from this paper, I think it's not suitable for publishing.\n\n  ", "extra_scores": {}, "extra_text": {}}, {"review_id": "quU7eawAk_", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a large scale empirical study on pretrained model selection for transfer learning and show that a hybrid approach that combines task-agnostic and task-aware methods outperforms the existing approaches on VTAB benchmark. The paper is well written and easy to follow. Experiments using 46 pretrained models and 19 downstream tasks show the effectiveness of the hybrid strategy in selecting the right model for transfer learning with low computational complexity. \n\nOverall, I vote for rejecting the paper as the paper has very limited novelty and experiments are not convincing. In particular, I fail to find the major contributions of the paper except the empirical study on VTAB dataset. While papers related to empirical study are interesting and worth of acceptance, this paper does not provide any major insight that could be useful for the future research on transfer learning. Furthermore, many experiments and comparisons are missing which should be included in the paper for a better understanding of the empirical study.\n\nHow is the proposed hybrid ranking strategy comparable to the model selection approaches presented in Duality Diagram Similarity: a generic framework for initialization selection in task transfer learning, ECCV 2020; DEPARA: Deep Attribution Graph for Deep Knowledge Transferability, CVPR 2020. These papers should be clearly discussed with possible comparisons in the experiments to show the advantage of the hybrid approach.\n\nComparison with many simple baselines are missing in the paper. E.g., How does the hybrid strategy comparable to fine-tuning with early stopping. Can we select pre-trained models by finetuning for only few epochs? How does the number of epoch affect the final performance while comparing to the hybrid strategy? \n\nHow is the current method comparable to Leep: A new measure to evaluate transferability of learned representations? Experiments and analysis should be included in the experiments to verify the effectiveness of the hybrid strategy.\n\nHow does the size of representation/feature affect the final performance? Does the conclusion still hold with different size of features? How does amount of data in the target task affects the performance of ranking? What is the effect of number of pretrained models on the ranking?\n\nMutual information between the features and discrete labels of the downstream task can be used to rank different models for transfer learning. How does the proposed hybrid strategy related to mutual information based ranking strategy? Experimental comparison should be included in the paper to verify this.\n\nDoes the ranking strategy and analysis presented in the paper limited to only classification models? In particular, can models trained using self supervised learning where there is no classification head, be used as pre-trained models in the current approach? How does the analysis change while considering self-supervised models which are now-a-days quite popular in representation learning? What about considering discriminators of generative models, e.g., VAE or BigGAN in the transfer learning analysis? More experiments and analysis should be performed in the experiments.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "prEXPl82hel", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Paper summary: This paper looks at the problem of efficiently choosing pre-trained models as initialization for downstream target tasks. It compares 3 strategies, a task-agnostic one which uses imagenet accuracies, a task-aware one which uses the acccuracy of linear classifiers on fixed representations, and a hybrid one which combines the two.\n\nPros:\n+ The evaluation is fairly thorough. I especially like the fact that the authors consider the different axes along which pre-trained models differ (model capacity, generalist/experts etc.)\n+ The pool of downstream datasets is large.\n+ The suggested strategy is simple and easy to implement. \n+ The problem is significant in practice since almost all practical applications of neural networks have this prroblem, and the gains seem large. I wish there was more work on this problem.\n\nCons:\n- The biggest issue is that this paper ignores several important papers publiished before on this problem. Especially of note is the Task2Vec approach, which computes model and task embeddings. I would like comparisons both in terms of accuracy/regret as well as computational cost:\nAlessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Charless C. Fowlkes, Stefano Soatto, Pietro Perona; Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019, pp. 6430-6439\n\nOther papers that are also relevant and should be cited and comparisons discussed:\nBishwaranjan Bhattacharjee, John R. Kender, Matthew Hill, Parijat Dube, Siyu Huo, Michael R. Glass, Brian Belgodere, Sharath Pankanti, Noel Codella, Patrick Watson; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2020, pp. 760-761\n\nAmir R. Zamir, Alexander Sax, William Shen, Leonidas J. Guibas, Jitendra Malik, Silvio Savarese; Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 3712-3722\n\n- The approach is not particularly novel. There is also no novel technical insight that explains the results.\n\n- The use of the JFT dataset hampers reproducibility since the dataset is not public. I'd like to see results with JFT excluded.\n\nFor acceptance, I would definitely want to see the first of these convincingly addressed.\n[Updated rating]", "extra_scores": {}, "extra_text": {}}, {"review_id": "9E8ZWMJsska", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\n\nThe paper evaluates three procedures for selecting models for transfer learning. The choices are task-agnostic selection, linear training, and the hybrid approach. They empirically show that the hybrid algorithm works the best on few-shot learning on images.\n\n### Feedback\n\n* The paper is a straightforward paper and easily understandable. The message is practical, but not very surprising. Unfortunately, it is only on image data; it would have been great if the authors had used an example from NLP too.\n* The hybrid approach is super-simple, which is nice. The results in Figure 6 confirm that how its ensemble nature helps. Although it does not necessarily outperform the linear algorithm in Figure 6. The ensembling approach also does not seems to be the optimal solution. The authors could study the generalization performance of the hybrid algorithm to provide further insights.\n* An empirical run-time analysis is missing.\n* While the authors indicate that all models perform comparatively poorly on the structured tasks, they do not provide specific insights about the root cause of this.\n* Overall, the idea is simple and practical, but the methodological contributions of this paper is rather limited.\n\n--------\n### Post-Response Update\nUnfortunately, the authors' response is not satisfactory on multiple issues. Thus, I reduce my rating by one point.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies efficient strategies for selection of pre-trained models for a downstream task. The main concerns consistently raised by the reviewers were limited methodological novelty, insufficient experimental analysis, unclear findings, and positioning of the paper with respect to related work that was ignored in the initial version. After the author response, R4 raised the score to borderline accept (still indicating the paper is weak without proper comparisons with other methods), whereas all other reviewers remained negative. The paper does have merits, as the methods are simple, and the problem is very practical (and somewhat understudied). However, the AC agrees with the majority that the paper is not ready for ICLR. The novelty is limited and the paper would benefit from more experiments, such as comparisons with simple baselines like early stopping as indicated by R1 and R3, and other methods such as Task2vec which address the same problem. The authors are encouraged to revise the paper according to the reviewers comments and submit it to another top conference.
Decision: Reject