Scalable Diverse Model Selection for Accessible Transfer Learning
Authors: Daniel Bolya
Rohit Mittapalli
Judy Hoffman
Venue: NeurIPS 2021 Poster (2021)
Keywords: Computer Vision
Transfer Learning
Model Selection
Transferability Estimation
Benchmark
Accessibility

Abstract: With the preponderance of pretrained deep learning models available off-the-shelf from model banks today, finding the best weights to fine-tune to your use-case can be a daunting task. Several methods have recently been proposed to find good models for transfer learning, but they either don't scale well to large model banks or don't perform well on the diversity of off-the-shelf models. Ideally the question we want to answer is, "given some data and a source model, can you quickly predict the model's accuracy after fine-tuning?" In this paper, we formalize this setting as "Scalable Diverse Model Selection" and propose several benchmarks for evaluating on this task. We find that existing model selection and transferability estimation methods perform poorly here and analyze why this is the case. We then introduce simple techniques to improve the performance and speed of these algorithms. Finally, we iterate on existing methods to create PARC, which outperforms all other methods on diverse model selection. We have released the benchmarks and method code in hope to inspire future work in model selection for accessible transfer learning.

Scalable Diverse Model Selection for Accessible
Transfer Learning
Daniel Bolya"
Rohit Mittapalli
Judy Hoffman Georgia Tech judy@gatech. edu
Georgia Tech
Georgia Tech smittapall130gatech. edu
dbolya@gatech. edu
Abstract
With the preponderance of pretrained deep learning models available off-the-shelff from model banks today, finding the best weights to fine-tune to your use-case can be a daunting task. Several methods have recently been proposed find good models for transfer learning, but they either don't It scale well to large model banks don't 't perform well on the diversity of off-the-shelf models. Ideally the question we want to answer is, "given some data and source model, can you quickly predict the model's ss accuracy after fine-tuning?" In this paper, we formalize this setting as "Scalable Diverse Model Selection" and propose several benchmarks for evaluating on this task. We find that cxisting model selection and transferability estimation methods perform poorly here and analyze why this is the case. We then introduce simple techniques to improve the performance and speed of these algorithms. Finally, we iterate existing methods to create PARC, which outperforms all other methods on diverse model selection. We have released the benchmarks and method code in hope to inspire future work in model selection for accessible transfer learning.
Introduction
Deep Neural Networks (DNNs) have shown to be very capable of solving wide variety of visual tasks. However, these networks often require large amounts of data and training time to perform well, limiting the accessibility of deep learning for computer vision. One approach alleviate this problem to employ transfer learning, commonly by fine-tuning an off-the-shelf model on the desired task.
With the increasing number of off-the-shelf models available spanning different tasks, datasets, training methods, and architectures, choosing the best model from which to transfer a challenging endeavor. A common heuristic in computer vision has been to use models pretrained on ImageNet [7] (specifically the ILSVRC challenge data), but more recent work is starting to expose weaknesses in the generalization performance of ImageNet features [39, 28, 40], and many of these works find that other pretraining datasets may perform better for some target tasks. Aside from source dataset, which architecture select isn't clear (when comparing similarly fast models). One could theoretically train several transfers from a diverse set of pretrained weights and keep the best performing model, but this isn't feasible in most practical applications where compute is limited.
This motivates the need for a model selection method that could query a large bank of existing pretrained models (of which several already exist, e.g., [36, 56, 18]) with a small subset of the target data and return a set weights which performs well when fine-tuned on all target data. Because of the sheer quantity of pretrained weights available to download off the internet today, such model bank has the potential to be massive and cover a wide variety of tasks, datasets, and architectures. Any
Equaa Contribution
httpps://bolla. werooolllgggttuttt wht...oooopaat..... to/paar////
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

selection method that intends to operate on such a massive library of weights would thus need two important properties: it would need to be sealable in order to accommodate 100s to 1000s of source models, and it would need to perform well on diverse set of input weights due to the wide variety of models available. In this paper, we formalize this setting as Scalable Diverse Model Selection.
Currently there are two main lines of work that could address this scenario. The first is model selection [11, 46, 10], which attempts to select viable transfer from a suite of arbitrary pretrained models. However, existing approaches require an initial model trained on the target data to suggest the transfer (which limits accessibility) and are not very successful when comparing across architectures (see Tab. 1). The second line of work is in transferability estimation [4, 52, 34], which attempts to predict how well a source model will transfer a target task. While this might seem similar to model selection, the two tasks are subtly different in evaluation. Transfcrability estimation methods usually fix one source model and vary the target task (i.e., "For my source model, which task would it transfer to the best?"), while model selection methods do the opposite: fix target task and vary the source models (i.e., "For my target task, which source model will transfer the best?"). While it might seem the same methods could work for either case, this turns out to not be the case (see Tab. 2).
Contributions. We formalize the task of Scalable Diverse Model Selection, which intends to make deep learning for computer vision more accessible. While other papers might have explored aspects of this space already, we standardize it by introducing several tools and benchmarks for evaluating model selection methods in this setting. First, we provide a controlled environment that includes exhaustively trained transfers from 8 source datasets to 6 target datasets across 4 different commonly used architectures for a total of 168 ground truth transfers for analysis (Sec. 3). We show that current state-of of-the-art transferability and model selection methods fail to beat simple baselines in this new setting (Tab. 1). We then analyze why this is the case and provide techniques to improve performance (Sec. 4). Using insights from this analysis, we develop PARC, a method that outperforms other methods on this benchmark (Sec. 5). Finally, we show that these results generalize to a larger experiment with an extra dataset and 33 additional off-the-shelf pretrained models downloaded from the internet (for a total of 65 source models and 423 transfers) and by extending PARC to object detection (Sec. 6). We have released all benchmarks and evaluation code at https: : //bolyaa github. so/parc/ in hopes to further the development of this promising area of research.
2 Related Work
In this paper, we introduce a new task for Scalable Diverse Model Selection, which attempts to make transfer learning more accessible by selecting the best pretrained model for downstream task from a massive bank of off-the-shelf models. This is adjacent to several fields such as accessible transfer learning, transferability estimation, and Taskonomy model selection.
Transfer Learning. Transfer learning is the act of using a model trained in one setting to boost the performance or speed up the training of a model in another setting, and is an extensively studied field in deep learning [49]. Several approaches exist for transfer learning from a source model to target dataset, the most popular of which include fine-tuning a new head to the target domain [9, 43, 60, 39], and lnn-ttnnig the entire network [2, 17]. While there are more sophisticated methods for fine-tuning (e.g.. [19, 37J), we study fine-tuning the entire network, as it's simple and widely adopted.
Accessible Transfer Leaning. A few works have tried make transfer learning more accessible. Neural Data Server [57] allows users to augment their own data with similar data indexed from several massive image datasets. While helpful in limited data environment, this requires the user to have additional compute in order to train with the extra images. Iterating on this idca, Scalable Transfer Learning with Expert Models [38] suggests pretrained models to the user instead, relaxing the compute requirement. However, the focus of the paper is primarily on creating these pretrained "expert" models and not in selecting between them. We argue that there are several "expert" models already widely available and trained with large amount of data (e.g., [47, 33, 39]), and thus focus on the model selection aspect of this problem. Both of these works use simple baselines to guide their model selection: Neural Data Server [57] uses the accuracy of logistic classifier on the source features fit to the target task, while Scalable Transfer Learning [38] uses nearest neighbors with hold-one-out cross coss-validation We benchmark both baseline techniques in our setting.

---

Transferability Estimation. There are several works that attempt to predict how well a model will transfer to new tasks, ranging from methods that attempt to assess a models capacity for transfer [23, 28, 40] to those that attempt to predict transfer accuracy [12, 3, 45]. Others attempt to predict the gap in generalization between training and test time [24, 53, 25J. There has also been a recent line of work that directly attempts to estimate transfer learning accuracy given source model and target dataset [4, 52, 34j. This line of work is the most applicable to our setting because the only assumption they make the source model is that it was trained on classification. While not ideal, this allows us to benchmark these methods (H-Score [4], NCE [52], and LEEP [34]) in our setting. Otherwise, Deshpande et al. [8] propose perhaps what is most directly applicable to work. Their setting is very similar (and fairly concurrent), though not as diverse and with no restrictions on evaluation speed. In addition, LogMe [58] is concurrent work that also focuses practical transferability estimation, but they don't I release their benchmark and their setting is more narrow.
Taskonomy Model Selection. The Taskonomy [59] models and dataset has been used to benchmark a previous line of model selection algorithms [11, 46, 10]. Taskonomy attempts to model similarities between tasks by how well models transferring from one task to another perform after fine-tuning. This makes a natural test-bed for model selection evaluation, as it contains a large number of pretrained transfers to use as a benchmark. However, using Taskonomy as a benchmark is incomplete. All Taskonomy models are trained on the same data (with different labels) and follow roughly the same architecture, only varying the source and target task. This means that the benchmark doesn't 't test robustness to source model diversiry, which is a core tenant of this work. Furthermore, typical model selection works on Taskonomy require network trained on the target data to suggest transfers, which should be avoided to make transfer learning more accessible. We benchmark the performance of RSA [11] and DDS [10], as they are the best performing methods on Taskonomy model selection. There are other works in this space such as [6] and [1], but the former scales poorly with the number of sources, and the latter cannot compare across architectures by design, so we don't 't include either in our experiments.
3 Scalable Diverse Model Selection
In this work, we address the problem of model selection for transfer learning but through an expressly practical lens. The goal of model selection from a practitioner's point of view is to find a pretrained off-the-shelf model that will perform well after fine-tuning on their data. In order for model selection method perform well in this setting. the models selects from need be diverse (i.e., cover a wide variety of source datasets, architectures, and pretraining tasks), and the selection method needs to be scalable (since the number of off-the-shelf models available today is massive and growing).
In order to provide a realistic transfer learning model selection scenario, we propose to select a model from large source model bank (over 100 models spanning several datasets and architectures) that transfers well to target training set after full model fine-tuning. The naive approach to this problem would be simply fully fine-tune transfer from each source model to the target training set, but this would of course be computationally infeasible. We'd like to work in the practical setting where we don't 't train any extra models, so we'd d like a computationally efficient transferability estimation method to predict which source models would transfer well. For the same reason, it's also infeasible to use the entire target training set for this estimate, since extracting all the features for 100 different source models is akin to training new model for 100 epochs (though without the backward pass). Thus, we restrict the method to only use small subset of the target training data for its estimate.
In this scenario, the target dataset is fixed while the source models can vary in dataset, architecture, and task. Previous work in model selection and transferability estimation typically only vary one of these aspects when evaluating their method (i.e., just task [59, 4, 11, 10]. just dataset [34, 52], or just architecture [34], but not all three at once). It's unclear how well, if at all, these methods benchmarked by varying only one factor will perform when all of these factors can change. Thus, we'v set out to test these methods in this much more challenging setting.
3.1 Creating a Diverse Benchmark
Because no model selection benchmark currently exists that varies more than one source factor at time, we create our own from 8 source datasets and 6 target datasets across different architectures for classification, varying both source dataset and architecture. We will also vary task in Sec. 6.
3

---

Datasets and Architectures. An ideal selection of source datasets would contain related datasets, so that transfer learning makes sense. Thus, for this benchmark, we choose well-known classifi- cation datasets of various difficulties that contain related subthemes: Pets: Stanford Dogs [26] and Oxford Pets [35], Birds: CUB200 [55] and NA Birds [54], and Miscellaneous: CIFARTO [29] and Caltech101 [14]. We also include VOC2007 [13] and ImageNet 1k [7] as the 7th and 8th source datasets, but not as targets. The other 6 datasets are also included as targets.
When selecting a model, a practitioner is often interested in the aceuracy-speed trade-off, meaning that the benchmark should include architectures at several tiers of evaluation speed. To facilitate this, we include three tiers of architectures: ResNet-50 [22] as the slowest, R8 [22] and GoogLeNet [48] in the middle, and AlexNet [30] as the fastest.
The goal in model selection is find a source model that will transfer well to target task. Ideally, this would be the best performing model, but that might be unreasonably difficult when there are 100s of models to choose from. Furthermore, practitioners are typically interested in the trade-off between performance and inference speed, which requires us to consider more than just the highest scoring models. What we really need in this case is a score for each model that correlates well with final fine-tuned accuracy on some target data. To benchmark such score, we use Pearson Correlation [15], widely adopted correlation metric (with 0 implying no correlation and 100 implying perfect correlation), between the model selection algorithm's 's transferability scores and the final fine-tuned accuracy.
Evaluation.
Thus we employ the following procedure to test a model selection method A on our benchmark: for each target dataset D' (indexed by t 7), we sample an n image "probe set" Pt D Then, for each source model parameterized by 0, (indexed by E S), we obtain
(1) as the predicted score for how well the source model 0, will transfer to the target dataset D'_ Then, we train each transfer from 0, to D' and evaluate on the test set of to obtain the final transfer accuracies wh. Finally, we compute Pearson correlation (denoted by pearsonr) between the predicted and final transfer accuracy and average over all target datasets:
a A(0,, P())
Mean PC (Varying Source)
1
Σ LET
pearsonr(fa : E twg S SE))
Because there can be large amount of variance in the probe set sampled, we further report mean and standard deviation over 5 different randomly sampled probe sets.
Note that we explicitly about is selecting the most accurate model (without limits on speed or other model parameters), other metrics such as top-k accuracy may be more suitable. Thus, we include some extra metrics in the Appendix.
Pearson Correlation because it incorporates all data points, if all you care
Implementation Details. For simplicity, images from all datasets are resized to 224 x 224. When constructing the probe sets, ensure that there are at least examples of each class (necessary for several methods) and randomly subsample classes this results in more than n = 500 images. We train all source models and transfers using SGD with weights frozen (i.e., full fine-tuning) and employ grid search to find optimal hyperparameters for each target dataset, architecture pair. Previous work in this space assume that each target model is trained with exactly the same hyperparameters. However, in a practical setting we expect some tuning to be done when training on the target dataset, so we have done the same. This makes the selection task more challenging but also more aligned with best case transfer outcomes. All models are trained on Titan Xp GPUs and all transferability methods are evaluated on the CPU. Note that times should be taken as lower bounds, since they' 're evaluated with expensive hardware. Many would-be practitioners don't 't have such hardware at their disposal.
3.2 Benchmarking Existing Work
We use this benchmark to evaluate several recent model selection and transferability estimation methods. We also include some typical baselines to contextualize the performance of these methods.
Probability-Bane Based Methods. We test two very recent transferability estimation methods, NCE [52] and LEEP [34]. Both operate similarly: given source model parameterized by 0 that predicts

---

Training
Source Task Agnostic
Target Task Agnostic
Mean PC (% 1)
Method
Input
Time (ms 4)
Time
NCE [52]
(a(= u).9 y
2.1 1 0.7
3.3 4.6
LEEP [34]
(a x). y
10.8 1 0.1
3.4 1 4.3
H-Score [4]
fo(x)) y
-5.4 = 4.9
5049.3 + 7200.2
RSA AlexNet [1]]
fo(x)
I Hour
1.4 = 0.6
93.7 : 19.1
DDS AlexNet [10]
Sa(z))
I Hour
1.7 = 0.4
65.8 = 17.9
RSA ResNet-50 [11]
fa(x)
Hours
57.3 = 0.4
102.7 = 15.6
DDS ResNet-50 [10]
Sa(m)
Hours
56.1 = 0.3
37.8 = 10.7
Heuristic
N/A
$1.0 =
A/A 42.1 + 8.3
I-NN CV
fo(r). y
60.8 + 1.2
Lugistic
fo(x). y
61.9 1.4 716.2 + 633.2
Table 1: Existing Work Underperforms on Diverse Source Models. Average Pearson correlation and time taken per transfer averaged over the 6 target datasets of our benchmark for several existing model selection and transferability estimation methods. Some methods use source model probabilities po(= I), some use latent features fo (zz), and some additionally require target data labels y. RSA and DDS need an extra model trained on the target data, for which we report average training time. All non-baseline methods either have almost correlation or take exorbitant amounts of time
probabilities po(= x) over the source classes z, input target images æ and estimate a joint probability p(z, v) between the source classes and the target classes y. Then (o zz) can be estimated as
poly x)
Σ ply 24) (o x))
Transferability is then computed by aggregating this probability distribution for all target images x. NCE and LEEP differ mainly in how they produce p(z, y): NCE (as extended in [34]) produces this by counting when the source model predicts for a target image with label y, while LEEP incorporates the entire distribution po(= x). The former naturally requires more data.
Feature-Based Methods. We also include several recent works from the Taskonomy [59] model selection literature: H-Score [4], RSA [11], and DDS [10]. All of these methods compare the penultimate layer's 's features fe(z) of a source model parameterized by 0 evaluated on target images a different set of features. H-score compares the covariance of the features with the covariance of their mean over the target classes y. Note that H-Score scales poorly with latent feature size, since they invert the covariance of the features (which can be as big as 4096 4096 for AlexNet [30]).
RSA [11] and DDS [10], on the other hand, compare the source model's extracted features to that of a "probe" model already trained on the target data. They assume that if two images are far apart in the prohe model's feature space, then they should also be far apart in the feature space of a good source model. RSA and DDS vary in how they construct these distances and how they aggregate them to produce final scores: RSA uses one minus the correlation coefficient between each pair of images for distance and Spearman correlation for the final score, while DDS tests large number of combinations, of which we choose the best performing in [10] (cosine distance and z-score). Note that both of these methods require training an additional model before they can begin recommending transfers, and the architecture used for this probe model highly impacts performance (see Tab. 1). Because these methods requires extra training time and expert knowledge of what architecture to use, we include them only for reference. While 3 extra hours of training time on Titan Xp might not seem like much, it drastically limits accessibility on cheaper hardware, where training can take days.
Baseline Methods. Finally, we include three fairly standard baselines. The first two are simple classifiers learned on top of the penultimate layer's features (z) and tested the probe set to get an estimate of final fine-tuning accuracy (outputting accuracy as the score). For these methods, we include i 1 nearest neighbors with leave one out cross- vesssvvaiittion (denoted as k-NN CV, used in [38]) and a logistic classifier trained on one half of the probe set and evaluated on the other half (used in [57]). We also include simple heuristic that rates performance the number of layers in the source network l, plus the log of the total number of images in both the source and target sets:
heuristic la log (ID*) D
(4)
5

---

2'
Method
20
Ralabaa Time Takan (Rail))
LCEP
Reatvv Morn (0N)
Reonel-d0 do
Resnal-50
DOG Resiet 1480 CV
UDS Kesnet-oU
Logistic
2
Logistic
5
24
100
200
500
1000
2000
5000
100
200
500
1030
2000
5000
Probe Budget
(a) Relative correlation to 7 500.
(b) Ratio of time taken to 72 500.
Figure I: Varying Probe Set Size. We allow each model selection method to see 500 image "probe set" of the target training data. To test whether this is a limiting factor for these methods, we vary probe set size and plot the resulting relative Pearson Correlation (a) and ratio of time taken (b) when compared to using 500 images. NCE [52], H-Score [4], and LEEP [34] all benefit significantly from more data (a), implying that their poor performance could be partially attributed to lack of data.
This captures the intuition that more layers and more data better (with log data scaling).
Results. In Tab. we list the Mean Pearson Correlation (computed as described in Sec. 3.1) and average time taken per transfer (not including feature extraction) for each method. We also include important properties each method that can impact its scalability (training time, time taken) and whether it supports diverse set of models (input requirements, being agnostic source / target task).
From this experiment, we can see that current works fare poorly when applied to this new benchmark. In fact, none of these methods perform better than the simple k = nearest neighbors baseline. Some even have close to no correlation (LEEP, NCE, H-Score). Furthermore, many of these methods extremely expensive to compute (e.g.. H-Score inverts a very large matrix when evaluating AlexNet). RSA and DDS also require training of target model with a manually selected architecture and high variability if poorly chosen (see Table 1). In the next section, explore why these methods, which were not designed for our challenging transfer setting, may fail to generalize.
A Note on Scalability. This benchmark includes 168 transfers, which should be enough to test the scalability of model selection approaches. However, when evaluating current methods, run the method on each source-target transfer and thus the "scalability" is linear per transfer. No current work exists to make this process sub-linear (e.g., through some kind of a weight embedding that you can binary search through), so instead we focus on the runtime speed of these methods. However, the end-goal of Scalable Diverse Model Selection really to have sub-linear scaling, so we hope that future work can become even more scalable in that sense.
4 Analyzing Failure Modes of Existing Selection Methods
In this section, we explore what causes existing algorithms to fail in Sec. and provide simple techniques to improve these methods the challenging task of diverse model selection.
Small Probe Set Size. One potential reason why these methods fail to generalize this setting is that, for efficiency, we use a probe set of n 500 target train samples to compute transferability. while the final transfer model fine-tunes correlation (a) and the ratio of time taken (b) for each method. NCE, LEEP, and H-Score all gain significant boost from the extra data, implying that perhaps the restriction of probe set major limiting factor for their performance. On the other hand, RSA, DDS, and the baseline methods seem be able to perform well with even fewer samples, which boon scalability.
all the data. In Fig. 1, vary and show relative
Robustness to Evaluation Mode. There's subtle, but important, point about evaluation that's often overlooked in transfcrability estimation. In model selection, you assume that you have fixed

---

Method
Source Dataset Arch
Architecture
Source Datasel
Target Datasel
NCE [52]
2.1 0.7
21.1 L 2.8
5.8 0.4
78.5 0.1
LEEP [34]
10.8 1 0.1
8.0 L 0.6
20.7 1 0.2
67.6 1 0.3
H-Score [4]
5.4 4.9
11.0 1
3.1
51.7 1
RSA Resnet-50 [11]
57.3 1 0.4
67.5 1 0.6
61.5 0.3
30.6 1 2.7
DDS Resnet-50 [10)
56.1 1 0.3
67.1 + 0.3
62.3 0.3
28.5 - 3.2
Heuristic
51.05 + 0.0
61.27 = 0.0
7.13 0.0
13.63 = 0.0
1-NN CV
60.8 + 1.2
67.4 2.0
67.0 :
79.0 + 1.9
Logistic
61.9 + 1.4
68.7 + 2.9
68.3 +: 1.5
81.0
Table 2: Varying the Evaluation Mode. In Eq. 2, we compute correlation over all source datasets and architectures. Here, we try varying each factor individually more closely match each method's original setup (marked as o). Most methods are poorly calibrated for other evaluation modes.
Method
Mean PC (%)
Time (ms)
RSA Foeret
Mean Condatton (S)
84
UES Mamme-30
1.NN DV
H-Score [4]
+
5049.3 7200.2
Log Will
HScore f = 32
55.9 = 0.6
43.1 + 21.4
62
RSA R-50 [11]
57.3 +
102.7 15.6
80
RSA R-50 = 64
66.2 + 0.5
183.4 + 42.4
DDS R-50 [10]
56.1 + 0.3
37.8 10.7
DDS R-50 f = 64
63.9 = 0.4
107.6 + 32.1
58
I-NN CV
60.8 1.2
42.1 1 8.3
I-NN CV f 256
60.6 + 1.3
192.8 + 40.9
Logistic
61.9 1.4
716.2 1 633.2
128
258
Logistic f f 256
61.7 1.6
339 4 1: 155.9
Number of Dimensions ?
(a) Varying the number PCA components.
(b) PCA can improve performance and speed.
Figure Calibrating Features with PCA. Feature-based methods with PCA applied the features beforehand for different numbers of outputs dimensions f (a). Applying feature reduction significantly improves all non- con-baseline methods (b), indicating that feature calibration was an issue. H-Score and Logistic, two methods that depend heavily on the number of features, also become much faster.
target dataset and would like to select the best source model for transfer. Transferability estimation often answers the opposite question: given fixed source model, which target dataset will it best transfer to? The former evaluates following Eq. 2, while the latter considers the following correlation:
Mean PC (Varying Target) = S
Σ pearsonr( (a : t EE (w::te)) aES
(5)
One might assume that the same method would work well in both situations, but this is not the case.
In Tab. 2, we test different evaluation settings by swapping what set correlation is being computed over (i.e., S for source and architecture, T for target, and subsets of for the rest). For each non- baseline method, we' we've marked their native evaluation mode (using source dataset for Taskonomy). Most methods work well using their original evaluation protocol, but are not robust to other settings. Since each evaluation mode compares outputs across different factors, hypothesize that the poor performance may be due to miscalibration and propose ways to mitigate this the following sections.
Dimensionality Reduction. For the feature based methods, there's a lot of potential variation in the number of features between architectures. For instance, the number of features [x)() for AlexNet is 4096, while for ResNet-18, 18, it's 1024. Not only does this cause some methods to vary wildly in evaluation time (e.g., H-Score), but it also can be a source of miscalibration. To address this, we apply PCA dimensionality reduction [16] to the features. fo(x), down to fixed dimension before computing each selection method. The results for this experiment are displayed Fig. 2. All non- baseline feature- based methods receive significant boost in performance from this dimensionality
7

---

Mean PC (%)
Mean PC (%) With Heuristic
Method
Mean PC (%)
Time (ms)
Method
Original
1-NN CV +
68.0 0.7
42.1 1 8.3
LEEP [34]
10.8 1 0.1
26.8 1 0.1
Logistic +-E
69.2 1 1.1
716.2 1 633.2
NCE [52]
2.1 1 0.7
28.3 0.5
RSA R-50 [11]
57.3 1 0.4
102.7 15.6
H-Score [4]
-5.4 + 4.9
25.6 + 9.2
RSA R-50 f 32
64.9 14.9.00.. 0.2
183.4 1 42.4
RSA R-50 [11]
56.1 +: 0.3
65.7 0.2
RSA R-50 12, 32
68.2
183.4 + 42.4
DDS R-50 [10]
57.3 0.4
64.9 0.2
PARC
53.0 - 0.9
49.4 18.3
1-NN CV
60.8 + 1.2
68.0 0.7
PARC f 32
59.3 - 0.7
107.0 31.1
Logistic
61.9 1.4
69.2 + 1.1
PARC +6,f = 32
70.3 + 0.5
107.0
Table 3: Modeling Capacity to Change. In this experiment, we add the depth E, of the source model to each transferability predic- tion. This results in significant performance boost for all methods on our benchmark.
Table 4: PARC Results. PARC compared with all the beneficial tweaks Sec. 4 (f for feature reduc- tion and +- for the layer heuristic, where f 32 is best when combined). PARC outperforms other methods, but requires these tweaks to work well.
reduction step. RSA and DDS even outperform the baselines after dimensionality reduction. For subsequent experiments, we will use the best value for f found in Fig. 2a for each method.
Capacity to Change. The goal of a scalable diverse model selection algorithm is predict a score that correlates well with full fine-tuning on the target set. However, all methods so far have used the source model as fixed features or probabilities, without considering the potential for those features to change after fine-tuning. Thus, we observe poor performance on more difficult target datasets (such as NA Birds, see the Appendix), where models with high capacity of learning are required. While modeling the capacity for a network to learn is out of the scope of this work, we can substitute that with simple heuristic. A strong indicator of how much information CNN can learn that applies to most off- thel architectures (e.g.. [30, 44, 48, 22, 50) is simply the depth of the model (i.e., number of layers). In Tab. 3, compare the performance and after adding this heuristic. To integrate this intuition, we first normalize the predicted scores of for each method by their mean j² and standard deviation of over all transfers and then add the relative source model depth l, over the maximum depth possible, fmxx (i.c., fax 50 for our experiments):
our benchmark for each method before
(a
0
16
L.
(6)
r²
&max
This change results in a significant boost in performance for all methods and will be denoted as +6 for future experiments. We explore other ways incorporate this model capacity in the Appendix.
5 Pairwise Annotation Representation Comparison (PARC)
Following the insights gained in Sec. 4, we devise a new method for the task of scalable diverse model selection. RSA [11] with dimensionality reduction performs extremely well (see Fig. 2). however it requires a "probe model" trained on the target data. To alleviate this restriction, we perform an intuitive modification: we replace the probe model with the ground truth labels. Features that work well for a target task should consider two images dissimilar if they were annotated differently.
More formally, given a probe set P,, of target images and labels y and model parameterized by 0, PARC produces two distance matrices Do, D,, of shape n X n as
Dg = - corrcoef(.f(z))
Dy = corrcoef(g(v))
where corrcoef computes pairwise Pearson product-moment correlation between the features of each pair of images (as used in RSA) and g maps the labels y to some vector representation. For classification, maps y to the corresponding one-hot vector, but in general, can be any function that maps the annotations to a vector. For instance, with semantic segmentation the target task, g could produce a pixel-wise average of the annotations, and similar extensions exist for other computer vision tasks. We explore this further in Sec. 6.2.
Then, to compute the final PARC score, like RSA we simply compute the Spearman correlation between the two distance matrices for all pairs of images:
PA P.)) spearmanr (Doli i}, (D, :i i))
(8)
8

---

Method
Mean PC (%)
Time (ms)
Method
Training Time
PC (%)
RSA R-50 f 64
50.64 0.21
180.6
RSA F-RONN
12 Hours
96.33 0.31
DDS R-50 He, = 64
50.72 0.19
120.4 26.2
DDS F-RCNN
12 Hours
95.97 1 0.68
1-NN CV f = 256
51.35 0.67
209.6 61.5
1-NN CV
None
89.67 6.
PARC f = 32 52.04 0.52
122.9 29.1
PARC
None
92.20 5.63
Table 5: Crowd-Soureed Models. A more general benchmark obtained by training transfers from arbi- trary crowd-soureed models. This includes 65 mod- els from a variety of domains and 423 total transfers.
Table 6: Object Detection. Predicting trans- fer performance for object detection. We em- ploy a simple scheme to extend classification- based methods to object detection.
Results. We apply all the same improvements discussed in Sec. 4 and report the performance of PARC on our benchmark in Tab. With both the dimensionality reduction and heuristic ensemble, PARC outperforms every other method (even with the same improvements). PARC observes a much larger boost from the heuristic ensemble than RSA, likely because RSA is allowed to use ResNet-50 model fine-tuned on the target set. Thus, it already has some information about how features can change over ane-tuning built in (which what the heuristic is trying to accomplish). PARC, on the other hand, isn't allowed this extra information, and thus adding the heuristic helps with calibration tremendously. We include more results for PARC in the Appendix.
6 Extended Benchmarks
Our benchmark in Sec. 3 describes a more practically useful setting than previous works, but it doesn't 't fully encapsulate scalable diverse model selection. Ideally, we could predict transferability between any source model (for any dataset, architecture, or task) to any target dataset or task. In this section, we explore arbitrary crowd-sourced source models and object detection as the target task.
6.1 A Crowd-Sourced Benchmark
In this experiment, we test how well PARC and other model selection methods perform on an even more diverse source model bank. To facilitate this, we collect 33 models from online model banks (i.e., [56, 18) that span several source datasets (17, 13, 32, 20, 51, 61, 33, 5J) and model architectures ([22, 42, 21, 27, 31]) covering many different pretext tasks (see the Appendix for full details). We combine this with the original 32 models we trained for a total 65 source models and we add VOC2007 [13] multi-class classification
additional target task, resulting in 423 transfers total. We place no restrictions on how the original models were trained, but we do normalize their features.
Results for this benchmark are available in Tab. 5. We only test on the subset of methods that support multi-class classification out of the box and apply all improvements from Sec. 4. While PARC outperforms the other methods, we note that none of the methods perform very well here (all having around 50% correlation with transfer accuracy). This indicates that selecting from an extremely diverse set of source models is still a very challenging task and warrants further study.
6.2 Transferability Detection
we've ve only considered classification the target task. However, as mentioned in Sec. 5, we can apply PARC to other tasks by summarizing the annotations y with a vector g c.g., by averaging the annotations pixel-wise. To average "pixel-wise" for object detection, we count all pixels belonging to boxes for each class, and then normalize by the total area of all boxes in the image. To extend I-NN, we measure the L, distance between pairs of these aggregate label vectors.
So far, we'
In Tab. 6, we display the performance of each method for predicting transfers from source detectors (Faster and Mask R- R-CNN [42, 21] on Cityscapes [5] and COCO [32], and Retinanet [31] and YOLOv3 [41] on just COCO) transferring to VOC2007 detection [13]. We compute Pearson correlation between the predicted scores and the mAP of each fine-tuned model, and we provide RSA and DDS a Faster R-CNN model trained on VOC2007. Because there are only 6 transfers, this is a much easier
experiment than main benchmark (and thus we found the techniques discussed in Sec. 4 to not

---

be important). While it doesn't 't perform as well as RSA or DDS here, PARC requires no additional model trained on the target data. Yet, it still highly correlated with final fine-tuned mAP.
7 Conclusion and Limitations
In this work, we introduce Scalable Diverse Model Selection, create several benchmarks to test this task, analyze existing methods in this setting, address multiple techniques to improve performance, and finally iterate on existing methods to create a new approach that works well. While the techniques we found combined with PARC improve performance on this setting, there are a few limitations of our work. First, we assume that the model selection method is applied to every source model, which could get extremely expensive. Subsequent work could try relaxing this assumption. Second, lowering the barrier to entry to transfer learning makes deep learning more accessible, but the model returned by these algorithms isn't guaranteed be the best, so more work would likely need to be done to temper the expectations of non-experts. Finally, selecting from any arbitrary set of models in a scalable way still remains challenging (especially on sourred models) and thus is still an open problem. Many factors can affect fine-tuned performance, but we only consider source feature quality and architecture capacity in this paper. We address this latter point with a simple heuristic, which is slightly unsatisfying. For instance, an ideal system could have some comprehensive learnability score for each architecture instead. Several papers look at factors that affect fine-tuning performance in isolation, but none combine everything into one recommendation system. We hope that this paper can be the first step in creating such a diverse model selection algorithm. We believe that a robust system to recommend pretrained models for transfer learning would be an incredible boon for the accessibility of deep learning and hope that future work can study this task in further detail.
References
[1] Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji. Charless C Fowlkes, Stefano Soatto, and Pietro Perona. Task2vec: Task embedding for meta-learning. In ICCV, 2019.
[2] Pulkit Agrawal, Ross Girshick, and Jitendra Malik. Analyzing the performance of multilayer neural networks for object recognition. In ECCV, 2014.
[3] Haitham Bou Ammar, Eric Eaton, Matthew E. Taylor, Decibal C. Mocanu, Kurt Driessens, Gerhard Weiss, and Karl Tuyls. An automated measure mdp similarity for transfer in reinforcement learning. In AAAI Machine Learning for Interactive Systems Workshop. 2014. 3
[4] Yajie Bao, Yang Li, Shao-Lun Huang, Lin Zhang, Lizhong Zheng, Amir Zamir, and Leonidas Guibas. An information -ecoorttccccr approach to transfcrability in task transfer learning. In ICIP, 2019. 3, 6, 7, 8 [5] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bert Schiole. The cityscapes dataset for semantic urban seene understanding.
In CVPR, 2016. 9
[6] Yin Cui, Yang Song, Chen Sun, Andrew Howard, and Serge Belongie. Large scale fine-grained categoriza- tion and domain-specific transfer learning. In CVPR, 2018. 3
[7] Deng. Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: large-scale hierarchical image database. In CVPR, 2009. 1, 9
[8] Aditya Deshpande, Alessandro Achille, Avinash Ravichandran, Hao Li, Luca Zancato, Charless Fowlkes, Rahul Bhotika, Stefano Soatto, and Pietro Perona, linearized framework and new henchmark for model selection for fine-tuning. Arxiv, 2021. 3
[9] Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In ICML, 2014. [10] Kshitij Dwivedi, Jiahui Huang, Radoslaw Martin Cichy, and Gemma Roig. Duality diagram similarity: a generic framework for initialization selection in task transfer learning. In ECCV, 2020. 3. 7,8
[11] Kshitij Dwivedi and Gemma Roig. Representation similarity analysis for efficient task taxonomy & transfer
learning. In CVPR, 2019, 2, 3, 5, 7,8 8
[12] Eric Eaton, Terran Lane, et al. Modeling transfer relationships between learning tasks for improved inductive transfer. In ECML PKDD, 2008. 3
[13] Mark Everingham, Luc Van Gool, Christopher Williams, John Winn, and Andrew Zisserman. The pascal
visual object classes (voc) challenge. IJCV, 2010. 4. 9
[14] Li Fei-Fei. Rob Fergus, and Pietro Perona. One-shot learning of object categories. TPAMI, 2006. 4
[15] David Freedman, Robert Pisani, and Roger Purves. Statistics (international student edition). Pisani, R.
Purves, 4th edn. WW Norton & Company, New York, 2007. 4
10

---

[16] Karl Pearson F.R.S. Liii. on lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 2(11):559-572. 1901. 7 [17] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hicrarchies for accurate object detection and semantic segmentation. In CVPR. 2014. 2
[18] Priya Goyal, Quentin Duval, Jeremy Reizenstein, Matthew Leavitt, Min Xu, Benjamin Lefaudeux, Mannat Singh, Vinicius Reis, Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Ishan Misra. Vissl. https:
// //github. cattsuubcoo///e
2021. 1,
[19] Yunhui Guo, Honghui Shi, Abhishck Kumar, Kristen Grauman, Tajana Rosing, and Rogerio Feris. Spottune:
transfer learning through adaptive fine-tuning. In CVPR, 2019. 2
[20] Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation.
In CVPR, 2019. 9
[21] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Cirshick. Mask r-cnn. In ICCV, 2017. 9
[22] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition.
In CVPR. 2016. 4, 8,9 9
[23] Minyoung Huh, Pulkit Agrawal, and Alexei A Efros. What makes imagenet good for transfer learning?
Arxiv, 2016. 3
[24] Yiding Jiang. Dilip Krishnan, Hossein Mobahi, and Samy Bengio. Predicting the generalization gap in
deep networks with margin distributions. ICIR, 2019.
[25] Yiding Jiang, Behnam Neyshabur, Hossein Mohahi, Dilip Krishnan, and Samy Bengio. Fantastic general-
ization measures and where to find them. ICLR, 2020.
[26] Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, Fei-Fii Li. Novel dataset for fine-graing fine-grained image categorization: Stanford dogs. In CVPR Workshop on Fine-Grained Visual Categorization (FGVC), 2011. 4
[27] Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Dollár. Panoptic feature pyramid networks. In CVPR, 2019. 9
[28] Simon Kornblith, Jonathon Shlens, and Quoe V Le. Do better imagenet models transfer better? In CVPR,
2019. 3.
[29] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers features from tiny images. 2009, 4 [30] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional
neural networks. Advances in neural information processing systems, 25, 2012. 4, 5, 8
[31] Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming Hc, and Piotr Dollár. Focal loss for dense object
detection. In ICCV, 2017. 9
[32] Tsung-Yi Yi Lin, Michael Maire, Serge Belongic, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014. 9
[33] Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten. Exploring the limits of weakly supervised pretraining. In ECCV, 2018. 2.9
(34] Cuong Nguyen, Tal Hassner, Matthias Seeger, and Cedric Archambeau. Leep: new measure to evaluate transferability of learned representations. In ICML, 2020. 2, 3, 4, 5, 6, 7,8
[35] Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. Jawahar. Cats and dogs. In CVPR, 2012.
[36] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan. Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf. Edward Yang. Zachary Vitioo Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjic Bai, and Soumith Chintala. Pytorch: An imperative style, high- teggh-eeffrmnncc deep learning library. In NewrIPS. 2019.
[37] Matthew E Peters, Sebastian Ruder, and Noah A Smith. To tune or not to tune? adapting pretrained representations to diverse tasks. Arxiv, 2019. 2
[38] Joan Puigcerver. Carlos Riquelme. Basil Mustafa, Cedric Renggli, André Susano Pinto. Sylvain Gelly, Daniel Keysers, and Neil Houlsby. Scalable transfer learning with expert models. Arxiv, 2020. 2,
[39] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh. Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from
natural language supervision. Arxiv, 2020. 1,2
[40] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize imagenet? In ICML, 2019. 1,3
[41] Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement. Arxiv, 2018. 9
[42] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. NewrlPS, 2015.
[43] Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson. Cnn features off-the-shelf:
an astounding baseline for recognition. In CVPR, 2014. 2
[44] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni-
tion. Arxiv, 2014. 8
11

---

[45] Jivko Sinapov, Sanmit Narvekar, Matteo Leonetti, and Peter Stone. Learning inter-task transferability in the absence of target task samples. In AAMAS, 2015. 3
[46] Jie Song. Yixin Chen, Xinchao Wang. Chengchao Shen, and Mingli Song. Deep model transferability from attribution maps. Arziv, 2019. 2.3
[47]
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In Proceedings of the IEEE international conference on computer vision,
pages 843-852, 2017. 2
[48] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet. Scott Reed, Dragomir Anguelov, Dumitru Erhan. Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, 4, 8
[49]
Chuanqi Tan, Fuchun Sun, Tao Kong. Wenchang Zhang. Chao Yang. and Chunfang Liu. survey on deep
transfer learning. In ICANN, 2018. 2
[50]
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In ICML, 2019.
[51] Bart Thomee. David A Shamma. Gerald Friedland. Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfec100m: The new data in multimedia research. Communications the ACM,
59(2):4473, 2016. 9
[52] Anh T Tran, Cuong V Nguyen, and Tal Hassner. Transferability and hardness of supervised classification tasks. In ICCV, 2019. 3, 4, 6, 7,8
[53] Thomas Unterthiner, Daniel Keysers, Sylvain Gelly, Olivier Bousquet, and llya Tolstikhin. Predicting neural network accuracy from weights. Arxiv, 2020. 3
[54] Grant Van Horn, Steve Branson, Ryan Farrell. Scott Haber, Jessie Barry. Panos Ipeirotis. Pietro Perona, and Serge Belongie Building bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection. In CVPR, 2015. 4
[55] P. Welinder, S. Branson, T. Mita. C. Wah, Schroff, Belongie, and P. Perona. Caltech-UCSD Birds 200. Technical Report CNS- TR-2010-001, California Institute of Technology, 2010. 4
[56] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan- Wan-Yen Lo, and Ross Girshick. Detectron2. https:
//ittub. http//aaebbookeeearcc/// www.rrc/dddteeroo..... 2019. 1,
[57] Xi Yan, David Acuna, and Sanja Fidler. Neural data server: A large-scale search engine for transfer
learning data. In CVPR, 2020. 5.
[58] Kaichao You, Yong Liu, Jianmin Wang, and Mingsheng Long. Logme: Practical assessment of pre-traiee pre-trained models for transfer learning. In JCML, 2021. 3
[59] Amir R Zamir, Alexander Sax. William Shen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In CVPR, 2018. 3.5
[60] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In ECCV, 2014.
[61] Bolei Zhou, Agata Lapedriza, Jianxiong Xiao, Antonio Torralba, and Aude Oliva. Learning deep features for scene recognition using places database. 2014. 9
12


Reviews:
[{"review_id": "wECjzqE5D5F", "reviewer": "Reviewer_LikY", "summary": "This paper introduced a unified benchmark to test the performance of existing transferability estimation algorithms. The main contribution is firstly its unified benchmark, and then, its experiment models and datasets are large, which show a fair comparison for all methods. Finally, they modified an existing algorithm to propose a new one, which achieves better Person correlation score over others.", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. My main concern is the applicability of the transferability estimation algorithms in real applications. From the recent results of LEEP, NCE, and the recent ones (LogME [1] and OTCE [2]), we see that although their evaluation scores (e.g., Person correlation) vary, but the results seem quite unified: no matter what algorithm you use, the most suitable source model will always be the biggest one (say, all methods will select ResNet-50 over ResNet-18 since we know that ResNet-50 has more capacity). So I question this line of research by the following questions:\n- Is it suitable to use a simple Person correlation score as the evaluation metric? Selecting the best model is a 0/1 task, if all methods give the correct model, then it is not necessary to measure their scores.\n- If all methods select the model with biggest capacity  / biggest pre-trained dataset, then what's the role of these selection algorithms? We can simply use the big models as the source model, why bother selecting?\n- Last, the Person correlation score is not that convincing: e.g. it is hard to understand what an improvement of 3% in Person correlation stands for.\n\n2. It is hard to understand the word \"scalable\". I think all other methods can also work in the situation of increasing source models / target models. Plus, from the results, the time complexity of PARC is just comparable to others, making me really confusing about this claim.\n\n3. There is no theoretical justification why this selection method can work.\n\n[1] You K, Liu Y, Wang J, et al. Logme: Practical assessment of pre-trained models for transfer learning[C]//International Conference on Machine Learning. PMLR, 2021: 12133-12143.\n\n[2] Tan Y, Li Y, Huang S L. OTCE: A Transferability Metric for Cross-Domain Cross-Task Representations[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021: 15779-15788.\n\n--- POST REBUTTAL --- \nAuthor's rebuttal did address some of my concerns. But I still think this paper has some issues that need to be addressed. Plus, the writing  should also be improved. Thus, I keep my score.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No negative societal impact", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "iRt-O-Yv1UR", "reviewer": "Reviewer_5SWG", "summary": "The paper studies methods to select source models from a diverse set, considering both the quality of the proxy for downstream performance, and the speed of computing the proxy. The paper introduces a new benchmark for evaluating the ability to choose from diverse source models, studies a number of existing methods, and highlights some shortcomings. It then proposes some tweaks to improve these. Finally, the paper proposes a new proxy PARC, which performs well on their benchmark (also using the aforementioned tricks), and also on an extended evaluation on crowd-sourced models.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Summary*\n\nOverall I think that the paper is well written and organized. It makes a substantial contribution to a relevant practical problem. The experiments look very thorough. Overall, I recommend acceptance, with just a few small comments, that if addressed I might increase my score further.\n\n*Originality*\n\nThe task of selecting from different source models has been studied before. However, this work introduces a substantially more systematic study than previous work. The paper also considers practical aspects carefully. I believe this is a significant original contribution. The literature is well cited, one extra relevant paper that studies different proxies that should probably be mentioned is [Which Model to Transfer? Finding the Needle in the Growing Haystack](https://arxiv.org/abs/2010.06402).\n\n\n*Quality*\n\nOverall, the benchmark and evaluation are thorough and informative. I have no major concerns, a few comments:\n\n- I like the detailed practical considerations: in particular reporting real-world wall clock times. To compliment these timings, it would be useful to also have an analysis of the various methods that is not tied so strongly to the particular hardware used: such as reporting total FLOPs, or even an asymptotic analysis.\n- I like the clear ablation of the tricks to improve existing methods, and the core PARC algorithm.\n- The additional experiments on detection are a strength, even though PARC is not the best performing algorithm.\n- How much does the analysis depend on the correlation measure, would the results be significantly different if Spearman's correlation was reported in Table 1?\n- It would be nice to see a summary of which models are actually selected for the different methods, and identify qualitatively their failure modes.\n- The paper mentions \"Furthermore, practitioners are typically interested in the trade-off between performance and inference speed\" -- however, as far as I can tell, the gold standard downstream performance is just accuracy (after finetuning) and does not take into account the speed of the resulting model? (I think this is OK, but needs to be clarified that this aspect is not included).\n\n*Clarity*\n\nThe paper is well organized and clear. A couple of minor things\n- PARC acronym is used before it is defined\n- The notation in Eqns (2) and (5) seems confusing. It seems that correlation is being computed between two unpaired sets, whereas isn't it computed over a set of pairs?\n-  typo: \"We train all source models and transfers using SGD\" -> transfer\n\n*Significance*\n\nI think the PARC method could be useful, although it is not a major extension over existing approaches, I think it is of moderate significance. The benchmark and thorough analysis of existing approaches I think is of high significance. Having a meaningful dataset to analyze and compare methods would be of great value.\n\n\nPOST REBUTTAL\n\nThe additional results are appreciated. It is interesting that under Spearman the rank of the methods is quite different. I think this should be acknowledged as a limitation, but I will keep my positive score.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations are well acknowledged. There is not much discussion of societal impact, however, for this generic ML/CV paper there is no need for this.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "RsoWlj8_S5A", "reviewer": "Reviewer_YN5P", "summary": "The paper studies model selection for transfer learning: given a number of source or upstream models, and a new downstream or target data, how can we choose the best model to finetune? The paper offers two main contributions. First, it proposes a specific benchmark with source models, target tasks, and an evaluation metric to compare different approaches. Second, after testing on a number of methods from the recent literature, it identifies some algorithmic aspects that seem to hurt these methods and proposes an alternative approach called Pairwise Annotation Representation Comparison (PARC) which addresses those. Experiments suggest PARC outperforms the original algorithms.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I'll list strong (+) and weak (-) points, together with questions (?).\n\n(+) I think model selection for transfer learning is indeed of very high practical importance.\n\n(+) The code and benchmark (pre-trained models?) will be open-sourced, thus helping people to use and report the benchmark.\n\n(-) The source datasets (and models) seem fairly small. It would make sense to add models trained on Imagenet (or parts of it).\n\n(-) My main concern is related to the evaluation metric (presented in equation 2). The goal of model selection for transfer is to identify one model to transfer (or, say, 2 or 3 if one has more budget). We only care about the finetuned \"quality\" of the selected model. However, the correlation approach presented in eq. (2) measures agreement for all the models --including the long tail of \"irrelevant\" ones. I don't think we need to account (at least so heavily) for those sub-optimal models. If the benchmark has access to the \\omega_s^t for all (s, t) --that is, the fine-tuned performance for source model s on target data t--, I think a more reasonable performance metric would be something like: for some small k >= 1, the top-k hindsight performance is the average \\omega_s^t over the top-k models s with highest \\alpha_s^t scores. In other words, we use the \\alpha's to select the models, and then we look at the finetuned performance \\omega we would get (we can normalize it somehow, to give all target datasets a similar importance). I'd add two columns (k=1 and k=3 for instance) to the tables in the paper, reporting a metric like this one (other work reports a notion of regret, [1]). In this case, I'm open to revisit my score (no need to remove the current mean PC, but discuss this).\n\n(+) The proposed heuristic where some sort of model complexity (like the second term in equation 6) seems to work well. I wonder if this is indeed equally useful when looking at a metric that focuses on choosing the right model. In any case, it would be nice if this aspect is studied in follow-up work.\n\n(+) PARC is relatively fast to compute, and it doesn't require access to a pre-trained model in the task data (as RSA or DDS do).\n\n(+) I think previous work hasn't been able to properly deal with the fact that different source models may offer representations with very different dimensionality. Using PCA seems like a good idea (and results suggest so too).\n\n(+) PARC can naturally deal with tasks other than classification (as g(.) unlocks the use of any relevant mapping).\n\nTypos:\n\n- line 71: methods.\n- line 85: transferability.\n\n\n[1] - Which Model to Transfer? Finding the Needle in the Growing Haystack (https://arxiv.org/abs/2010.06402)", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "NgvnEDs4vH", "reviewer": "Reviewer_rwfu", "summary": "The authors introduce the “scalable diverse model selection” task and introduce several tools and benchmarks for evaluating model selection methods in this setting. They show that current transferability and model selection methods fail to beat simple baselines in this new setting. They analyze the reason for this case and provide techniques to improve performance. They develop PARC, a method that outperforms other methods on diverse model selection.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros: \n- The paper is well written and it is a timely paper to discuss the limitations of existing model selection methods, such as they don’t consider different model architectures. The comparison of different methods regarding probe size, architectures, and feature dimension might be interesting to readers in this area.\n\n\nCons:\n\n- The authors compare several model selection algorithms, i.e., probabilistic methods and feature-based methods. However, those algorithms are not introduced anywhere and the authors may assume that readers are familiar with those algorithms.\n- My major concern is that the proposed benchmark can not fully evaluate “scalable” model selection and it is not clear how to scale the benchmark. The authors collected 8 source datasets to 6 target datasets across 4 different commonly used architectures. However, the size of the source datasets and target datasets in total is 32, which is not quite big. The architectures used (ResNet-50/18, GoogleNet, AlexNet) are not quite representative. Architectures with different input resolutions would be great to add. The target 6 datasets are also quite small, among which their performance is almost saturated, such as the accuracy of Pets/CIFAR10 is above 95%. There are Google VTAB datasets (19) which could be much larger as the target sets.\n\n- It is not clear why the new proposed PARC has an advantage over existing methods such as RSA. The PARC seems pretty similar with the Label-Feature Correlation proposed in [1], which would be great to include for discussion. \n\n- It would be clearer if the authors can make clear what exactly are the models and datasets described in section 6.1. It is referenced but not stated anywhere.\n\n- It is not clear how the benchmarks can be released with protocols and facilitate future comparisons.\n\n[1] A linearized framework and a new benchmark for model selection for fine-tuning, arXiv 2021", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "ENTP8vhmyom", "reviewer": "Reviewer_cgQC", "summary": "The paper proposes a model selection method wherein best experts are selected from a model zoo to fine-tune on the target task. The proposed method \"Pairwise Annotation Representation Comparison\" is an improvement over RSA [8], instead of using a small dnn trained on target task (or probe network) the authors use spearman correlation between RDM of features and label as model selection score (called PARC score). The proposed method is compared to many baselines NCE, LEEP, RSA, DDS etc. and shows better correlation to fine-tuning accuracy.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+ The proposed method is compared to many different baselines. In Tab. 2, 3 and 4 the proposed method shows better correlation to fine-tuning accuracy compared to RSA, LEEP, NCE, Nearest Neighbors model selection baselines. Tab. 2 also performs an ablation of the correlation score when architecture, source dataset and target dataset are changed.\n+ The approach is extended to show that selection with the same scoring function works for object detection tasks. Model selection for object detection is not studied in any of the prior arts.\n\nWeakness:\n- The results on transfer between different models to the target task are missing. E.g. does imagenet trained expert or some of the candidate architecture always work better than the rest?\n- Some other relevant baselines to the proposed method which are not discussed are below:\n1. Cui et al \"Large Scale Fine-Grained Categorization and Domain-Specific Transfer Learning\" -- Uses EMD distance between average class vectors of source, target task to estimate transferability\n2. Achille et al \"Task2Vec: Task Embedding for Meta-Learning\" -- Estimates an embedding space for tasks. If source and target task are close in the embedding space, the transfer is better\n3. Deshpande et al \"A linearized framework and a new benchmark for model selection for fine-tuning\" -- Proposes a Label Feature Correlation method for model selection that uses correlation between label/feature similarity matrix. The score function is similar to proposed approach of Annotation (i.e. label), Representation (i.e. feature) Comparison.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations are addressed in detail in Sec. 7.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: In summary, the reviewers rate this paper as follows:
* 3 of 5 reviewers view it as being close to the acceptance threshold (2 of 5 slightly below, 1 of 5 slightly above, not updated from "5" to "6" in the official rating, but in a discussion comment).
* 2 of 5 reviewers recommend the paper for acceptance (with scores "7" and "8").
So the reviewers overall are leaning towards acceptance with no reviewer opposing acceptance.

The paper discusses several methods for model selection and ranking and proposes a new method (PARC) for this task. Experiments compare the methods on a new benchmark created for that purpose.

Highlights of the paper mentioned by the reviewers include:
* The presented study is enlightening and thorough.
* The subject is of high practical importance. 
* Open-sourcing the benchmark will be valuable.
* The paper is well written.
* The authors' responses addressed several of the reviewers' initial concerns.

Some concerns expressed by the reviewers include the following:
* Limited improvements of the introduced method; similar performance of other approaches.
* Some over-claiming regarding novelty of the approach.
* A lack of the definition of "scalability".
* The value of ranking models vs selecting best models is debated.
* Some related work is missing or could be discussed more appropriately.

In case the paper is published, it would be appreciated if the authors could address the concerns as far as possible. Specifically, the results presented in the authors' responses would be valuable to include.

Overall, I would recommend the paper for acceptance as a poster.
Decision: Accept (Poster)