Scalable Transfer Learning with Expert Models
Authors: Joan Puigcerver
Carlos Riquelme Ruiz
Basil Mustafa
Cedric Renggli
André Susano Pinto
Sylvain Gelly
Daniel Keysers
Neil Houlsby
Venue: ICLR 2021 Poster (2021)
Keywords: Transfer Learning
Expert Models
Few Shot

Abstract: Transfer of pre-trained representations can improve sample efficiency and reduce computational requirements for new tasks. However, representations used for transfer are usually generic, and are not tailored to a particular distribution of downstream tasks. We explore the use of expert representations for transfer with a simple, yet effective, strategy. We train a diverse set of experts by exploiting existing label structures, and use cheap-to-compute performance proxies to select the relevant expert for each target task. This strategy scales the process of transferring to new tasks, since it does not revisit the pre-training data during transfer. Accordingly, it requires little extra compute per target task, and results in a speed-up of 2-3 orders of magnitude compared to competing approaches. Further, we provide an adapter-based architecture able to compress many experts into a single model. We evaluate our approach on two different data sources and demonstrate that it outperforms baselines on over 20 diverse vision tasks in both cases.

Published as a conference paper at ICLR 2021
SCALABLE TRANSFER LEARNING WITH EXPERT MODELS
Joan Puigcerver' Google Research
Carlos Riquelme" Google Research
Basil Mustafa
Cedric Renggli
Google Research
ETH Zurich
André Susano Pinto Google Research
Sylvain Gelly Google Research
Daniel Keysers
Neil Houlsby Google Research
Google Research
ABSTRACT
Transfer of pre-trained representations can improve sample efficiency and reduce computational requirements for new tasks. However, representations used for transfer are usually generic, and are not tailored to particular distribution of down- stream tasks. We explore the use of expert representations for transfer with simple, yet effective, strategy. We train a diverse set of experts by exploiting existing label structures, and use theap-to-compune expert for each target task. This strategy scales the process of transferring new tasks, since it does not revisit the pre-training data during transfer. Accordingly, it requires little extra compute per target task, and results in speed-up of 2-3 orders of magnitude compared to competing approaches. Further, we provide adapter-based architecture able compress many experts into single model. We evaluate our approach on two different data sources and demonstrate that it outperforms baselines on over 20 diverse vision tasks in both cases.
performance proxies to select the relevant
INTRODUCTION
Deep learning has been successful on many computer vision tasks. Unfortunately, this success often requires large amount of per-task data and compute. To scale deep learning to new vision tasks, practitioners often turn to transfer learning. Transfer learning involves re-using models trained on large source task, and tuning them the target task. This
improve both convergence rates
(Ben-David et al. 2007 2010 Blitzer et al. 2008: Du et al.) 2017 Kuzborskij & Orabona 2013
Mansour et al. (2009) and empirical performance (Dai al.) 2007 Donahue et al. [2014] Oquab et 2014 Tan et al. 2018). Transfer learning reduces per-task data or compute requirements, given large one-off pre-training cost. In practice, this one-off down payment may not be made by the practitioner, since pre-trained networks are made available through platforms like Py ToTch and TensorFlow Hub" For instance, ImageNet pre-training is popular since freely available and works well for many tasks (Donahue et al. 2014 Oquab et al. 2014 Sharif Razavian et al. 2014).
In contrast to generic homogeneous models (e.g. most pre-trained ImageNet networks), Mixture of Experts (MoE) include multiple heterogeneous sub-models ("experts") that specialize to sub- nub-problems of the full task. MoEs have been studied for decades (Eigen et al. 20133] Jacobs & Jordan 1993), and have also been successful in deep learning (Shazeer et al. (017). Yet, the application of experts for deep transfer learning has been less explored. We study visual transfer with experts, and present a simple, scalable, yet effective strategy.
Transfer of specialist models has been studied before. However, they either require expensive re- training the source dataset for every target task (Ngiam et al. 2018) Yan et al., 22000) or operate at a small scale where all experts can be applied simultaneously (Dvorik et al. (020). Further, most of them are tested only a limited suite of natural single-object classification tasks. We lift these
" 'Equal contribution. Order decided by coin toss. 1 Work done while interming at Google Research.
httttp:://pyttrc..orr./.// and httpstttttttttttttee pettps://tfhub.... dev/ respectively.

---

Published as a conference paper at ICLR 2021
1
uncondilional pre-training
2
experts training
3
expert selection
4
downstream fine-ture tan-tunnnn
upstream clssess
KNN
upstream data
downstream data
downstream data
upstream data
Figure I: Transfer Learning with Per-Task Routing of Experts. Step 1. A single baseline model trained on the entire upstream dataset. Step 2. The upstream data is divided in semantic subsets (possibly overlapping). One expert is trained on each subset using the weights from B as initialization. Step 3. Given new downstream task Dr (Xr, Yr), we compute the image representations (xT) from each expert e. We use kNN to compute the accuracy on the supervised problem Dr. (Mr(Xr)) Yr). and select the expert e* with highest accuracy. Step 4. We add new head to e* and fine-tune its whole network with the downstream data, leading to the final model.
constraints, and present a practical approach that scales to hundreds of large experts, while requiring relatively little compute per target task.
Our strategy consists of four stages (fig. TD. (1) Unconditional pre-training. A single baseline model is trained on the entire upstream data. (2) Experts training. Multiple experts are pre-trained by exploiting the label hierarchy present in many large-scale image datasets, such as ImageNet and JFT. In addition to entire expert networks, we explore residual adapters that allow all of the expertise to be packed into a single model that can be loaded into memory. These two stages may be expensive, but are done only once. (3) Expert selection. Applying all experts to each task does not scale well; some sort of sparsification is required. We focus on inexpensive model selection that can be applied to hundreds or thousands of experts.
(4) Downstream finiig We take the output of the model selection phase and tune it on the target task. Importantly, this phase does not require revisiting the source dataset, which may be unavailable or expensive to train
We show that this approach yields remarkably strong performance on many diverse tasks. We evaluate not only on classic vision tasks, but also on the diverse VTABB benchmark of 19 tasks (Zhai al.) 2019) Our contributions can be summarized as follows.
We propose a transfer learning algorithm with a large number of experts based on per-task routing via nearest neighbors selection. Once we have amortized the pre-training cost, this algorithm requires little compute per target task, achieving an speed-up of 500x-1000x compared to competing strategies. Also, it can be easily replicated with any large upstream multilabel dataset.
We achieve mean accuracy improvement of 3.6% over the state-of-the- ofttthaartt performance on 19 VTAB datasets using ResNet50 networks. Our algorithm offers improvements on every group of tasks: natural, specialized, and structured. Figure Iummmarizes these results.
We explore using sub-networks as experts via residual adapters, allowing all experts to be packed into a single model. Surprisingly these perform almost as well as their full-network counterparts.
RELATED WORK
Transfer Learning. Tasks with little training data can benefit from other larger datasets, often from similar domain. Transfer learning concerns the link between the source and target dataset (Pan
& Yang. 2009 Weiss et al.] 2016 Tan et al., 2018 Wang 2018) One family of methods creates single training dataset, where source instances are we-ghihhted according to their relevance (Dai et al. 2007; Pardoe & Stone 2010: Wan et al. 2011 Xu et al. (017). popular method consists of fine-tuning a model that was tre-trained on the source data (Donahue et al., 2014 Oquab et al.)
2014; Sharif Razavian et al. 2014). Some transfer learning algorithmss condition the mitiil source
model on the target dataset itself (Ngiam al. 2018: Xie et al. 2019: Yalniz al. 2019). while
2

---

Published as a conference paper at ICLR 2021
VTAB
Natural
Specialized
Structured
Average eccurary
7L.
72.3
50.0 0
78.3
80 2
B5.7
83.6
59
57.2
58.6
58.8
58.5
77.4
05.0
50
20
88
73
77
50
66
70 Baseline (FT)
74
Experts (FT))
All Experts (IN214)
All Experts GFT + (N21K)
Figure 2: Summary of results on the VTAB-IK benchmark, combining experts with different architec- tures trained on two different data sources (JFT. ImageNet21k). In each of the 19 datasets, we use the median accuracy over 30 runs. The average of the accuracies in each group is shown, as well as (percentile) bootstrap confidence intervals at the 95% level.
others (like ours) are agnostic about the downstream task when the initial model is trained on the source data (Kolesmikov et al., 2019). We offer an in-depth comparison with 4Ngiam et al.) 2018) in section 6.6 In the context of few-shot learning, where out-of-the-box fine-tuning may not work, generic representations are sometimes frozen, and simple feature selection (Dvornik et al. 2020) or model training (Chen et al. (209)) techniques are applied on top. Instead of relying on fixed universal representations, (Rebuffi et al. 2017 2018) use small additional modules, or adapters, that incorporate knowledge from several visual domains. Our work also explores this idea.
Multi-task Learning. MTL tries to leverage the common aspects of several learning tasks (Caruana 1997). A prominent approach uses explicit parameter sharing; for instance, by means of common low-level layers leading to different heads. Among others, this has been successfully applied to vision (Zhang et al.] 2014), language (Liu et al.) (013)), and reinforcement learning (Fedus et al. 2019) tasks. In addition, a variety of ways to combine task-specific representations have arisen, such as cross-stitch networks (Misra et al. 2016), or lateral connections (Rusu et al., (2166). A different family of methods impose joint constraints on the -possibly different- models corresponding each task. We can combine the learning problems via regularization and shared sparsity patterns (Argyriou al. 2007; Lounici et al., 2009), or by imposing some prior knowledge regarding the task structure (Evgeniou et al.) 2005; Jacob et al. 22099] Kim et al. 2012).
THE TRANSFER LEARNING FRAMEWORK
In this section, we describe our transfer learning setup of interest. The high-level goal is to train strong models for arbitrary downstream tasks, possibly under severe data and compute limitations. To do so efficiently, one can offload computation to a previous upstream phase which is executed a priori, without knowing the downstream tasks in advance. Accordingly, the upstream model should not depend on any specific target data. We are mostly interested in the low data regime where downstream tasks contain few datapoints. These restrictions have a practical motivation: we would like to build and deploy universal representations that are easily transferred to a wide range of downstream settings. Any transfer algorithm must implement the following three stages.
Upstream Training. Given the upstream data Du, the algorithm first outputs source model M. The goal is to provide useful initial representations for various new tasks. This stage could actually produce family of models {M,} rather than single one. These models might not be disjoint, and could share parameters. The upstream learning problems are auxiliary; accordingly, Dy could include diverse set of classification, regression, even synthetic learning instances.
Model Selection. When a new downstream task is given, a selection algorithm is applied to choose the upstream model(s) to transfer, possibly depending the downstream data. This phase should be computationally cheap; thus, the upstream data is no longer available. Sometimes, there is no choice to make (say, with single ImageNet representation). Alternatively, in models with complex structure, one may choose which parts, routes, or modules to keep in a data-dependent fashion.
Downstream Training. The final stage fine-tunes the selected model using the downstream data, either fully or partially. For neural nets, a new head is added as the output classes are task-specific.
Our overall algorithm is depicted in fig. We give details about each step in the following sections.
3

---

Published as a conference paper at ICLR 2021
inpu
Root
Bock
Block
Block
Block
Head
di
de
(a)
(b)
Figure 3: (a) ResNet with expert adapters before blocks. A layer of experts is placed before every block. (b) Each individual adapter including the overall skip connection. N, A, C stand for (Group) Normalization, (ReLU) Activation, and Convolution layers, respectively.
4 UPSTREAM TRAINING
4.1 EXPERT ARCHITECTURES
Our experts should provide feature extractions that are good starting point to learn future tasks related to the expert's 's upstream training data. We explore two different model architectures to train such experts. As an obvious choice, we first look ResNets (He et al. 2016b). These are powerful models; however, storing and deploying many of them can be challenging. As alternative, we also develop more compact adapter modules that can all be assembled in a single architecture. Also, their individual size be easily customized to meet memory and computational constraints, which makes them an ideal candidate for combining multiple experts in a single model, when needed. We informally refer to these as full and adapter modules (or experts), respectively.
Full ResNet Modules. As a base architecture for full experts we use ResNets. In particular, all of our experiments focus on the ResNe150-v2 architecture (R50) (He et al. 2016a), which sequentially stacks a root block and 4 blocks with (3, 4, 6, 3) residual units. The initial step in every experiment consists of training baseline model B the whole upstream data (see stage in fig. [ This baseline is subsequently fine-tuned by both full and adapter experts, but in different ways. A full expert trained on slice of data simply the baseline fine-tume Line-tuned on that data. The head will later be discarded for transfer. This approach requires as many R50s as there are experts.
Adapter Modules. Residual adapters were proposed adapt a neural network to a particular downstream task without needing to fine-tune the entire network (Rebuffi et al.] 2017). Instead, we use them to adapt the baseline architecture to slices the upstream data. Originally, they were x 1 convolutions placed after each 3 3 convolution, with residual connection. Instead we place them before each of the R50's blocks. Finally, our adapters have a bottleneck and are non-linear, as in (Houlsby al. 2019). We insert several in parallel into the backbone B. When creating expert, only the adapters are tuned and the backbone weights are frozen.
Figure 3a depicts the ResNet architecture with multiple expert adapters (a ....,, (3) Let F, be the function implemented by the i-th block the backbone network. We adapt imput by computing
the output as == F + ae (x;-1)), where = R(r) is the identity of the selected expert, given by some routing function R. and is the original input. During upstream training, the function R may also use the labels in addition to the image, as we discuss in section 4.3
Figure 3h shows the adapter's bottleneck architecture. An adapter sequentially applies components A and A2. Each component performs a group normalization (N) (Wu & He 2018). ReLU activation (A) Glorot et al., 2011), and a convolution (C) (LeCun et al. 1989). in that order. Due to the skip connection, the output dimension of A₂ A must match that of its input, c. However, we can change the output channels l of A1, in order to limit the amount of parameters. Thus, we set l: = á so that the number of parameters equals that of linear adapter. Each adapter only increases the parameter count of the R50 backbone by 6%. We briefly explored placing these adapters in other locations, or using other variations (Rebuffi al. 2018), but we did not observe any significant improvement.
4

---

Published as a conference paper at ICLR 2021
It is important to emphasize that the names Full and Adapters refer to the parameters that specialized to given subset of the upstream data before any downstream fine-tuning In both cases, we fine-tune the entire network, after the model selection phase, to the downstream task.
4.2 UPSTREAM DATA AND EXPERT DEFINITION
We train our upstream models on large vision datasets with thousands of classes. Morcover, the datasets include an expressive hierarchy, linking classes and ancestor concepts via "is-a" relationships. Our experts' domains are nodes in this hierarchy, which are selected automatically based on the number of images. Due the multi-label nature of the datasets, several experts could simultaneously apply to an image. For example, for an image of a lion, all of organism, animal, carnivore, ferivooeefelldaee, and lion could be relevant expert domains. In particular, we use two different upstream image datasets, and independently train a set of experts on each. We further describe them in section 6.
4.3 EXPERT TRAINING
Recall we denote by B the baseline R50 model trained on the whole upstream dataset Dy. As shown in fig. [. the second step of upstream training consists of training each expert individually on different subsets of the upstream dataset. Let D. (X,, Y.) Dy be the data corresponding expert e. The subsets corresponding to different experts may overlap (c.g. for the animal and dog experts).
As mentioned before, the full experts completely fine-tune B De. For the adapter experts the weights corresponding to the adapter (modules in red in fig. are trained on De, but the shared blocks and head parameters are frozen. Note that, due to the sharing scheme, we can train all experts independently in parallel. We train all experts for the same number of steps, regardless of the size of De. Instead of learning a routing function, we exploit the structure of the upstream labels and use a hard-coded routing. We found this makes learning easier, and leads to powerful specialized models.
EXPERT SELECTION
Given a new downstream dataset Dy = (XT, ). we must choose an expert to use. We consider three approaches: domain prediction, label mafching, and performance proxy.
Domain Prediction. This strategy selects the expert solely based on the images XT. It effectively selects the expert whose domain best matches the target images. We implement this by training an auxiliary network (the "Expert Prediction Network" or EPN) to classify the expert from the image (i.e. learn the hard-coded routing mentioned previously). The EPN is trained upstream using the pre-training data and expert assignments. During transfer, an expert is selected using the highest geometric mean EPN predictive probability across the dataset. Details are the Appendix [
Label Matching. Matching the task with an expert can be done in the label space opposed to the input space, by computing the affinity of each expert to a new downstream task in the label space of the upstream dataset. We first use a generic network trained on all upstream labels to predict upstream labels on the downstream images. We compute the KL-divergence between the distribution of labels on the downstream task images, and the prior distribution of labels for each expert. This per-expert prior is computed as the empirical distribution labels on the images used to train that expert. We select the expert with the smallest KL-divergence. Details are in the Appendix B
Performance Proxy. The aforementioned two strategies are simple, but do not use the training labels Yr available for downstream tasks, which may contain key information. It would be too expensive to fine-tune every expert to every new task and select the best with hindsight, so we propose a proxy for the final performance. For this, we use k-nearest neighbors classifier (Altman 1992) with the image embeddings produced by each expert. In the case of full experts, we simply apply the corresponding full network compute these embeddings. For adapter-based experts, we apply the specific expert and ignore the remaining ones. Concretely. let M (x) be the embedding corresponding to expert on input x, and let Dr ( Nr be our downstream task. In order to score each expert, we apply kNN classifier the embedded dataset Dr. (I:) Mr J. with A 1 and Euclidean distance. The accuracy acc(Dr,e) is computed via leave-one-out cross-validation Finally, we select the expert with highest accuracy: e" arg max, acc(Dr,e). There are other alternative proxies that
5

---

Published as a conference paper at ICLR 2021
cheaper than full fine-tuning for example fitting logistic regression, SVM, decision trees to the features. Although these proxies may better match the fine-tuning accuracy, we use kNN since it is computationally cheap it only requires forward pass through the data, and leave-one-out cross-validation requires no additional inference per-fold - and it performs well (section 6).
5.1 DOWNSTREAM TRANSFER
The expert selection algorithm could choose several experts to be combined to solve any target task. However, limit the scope of our work to transferring single expert per task, since this approach is simple and turns out to be effective. Thus, the downstream transfer procedure is straightforward: it simply involves fine-tuning the selected expert model. We fine-tune the entire expert network to the downstream dataset, including the adapters when applicable. While it was valuable to restrict the scope of upstream training focus on specializing the expert adapter parameters, we found fine-tuning the whole network downstream to be greatly beneficial. This differs from the original residual adapters work (Rebufli et al. 2017), where only the adapters were fine-tuned.
EXPERIMENTAL RESULTS
6.1 UPSTREAM TRAINING
ImageNet21k (Deng et al. 2009) is public dataset containing 13 million images, and 14 million labels of 21 843 classes, which are WordNet synsets (Fellbaum) 2012). In addition to the 21k classes, consider the 741 synsets that are their ancestors. We the 50 synsets of ImageNet2 with the largest number of images to train the expert models. These include e.g. animal, artifact, organism. food, structure, person, vehicle, plan, instrument. We released 48 of these ImageNet211 model
JFT (Sun et al. 2017) is an even larger dataset containing 300 million images and 18 291 classes. Each image can belong to multiple classes, and for ImageNet21k, lk the classes are organized in hierarchy. We select as expert domains the classes with a sufficiently large number of examples: the 240 classes with more than 850 000 images. Some of the automatically chosen experts are animal, arts, bird, food, material, person, phenomenon, plant, or product.
We pre-train generic models on a Cloud TPUv3-512, as done in (Kolesnikov et al. 2019). Then fine- tune them briefly on each slice to create the expert models. Check additional details in appendix D
6.2 DOWNSTREAM TASKS
We evaluate on two suites of tasks, each consisting of several datasets. The first is the Visual Task Adaptation Benchmark (VTAB) (Zhai al., (019)), which consists of 19 datasets. We evaluate on VTAB-IK, where each task contains only 1k training examples. The tasks are diverse, and divided into three groups: natural images (single object classification), structured tasks (count, estimate distance, etc.), and specialized ones (medical, satellite images). Appendix E..1 contains further details. The second suite is a collection of popular natural datasets commonly used in transfer learning literature and, particularly in (Ngiam et al., 2018), to which we compare our work.
6.3 TRANSFER EVALUATION PROTOCOL
When transferring to new tasks we need to perform expert selection and choose other hyperparameters (c.g. learning rate for fine-tuning). For each downstream task, we use the following three step protocol.
Expert Transfer. We select the expert to transfer using one of the methods presented in section 5 In both sets of tasks, we use 1k training examples per dataset. Details are provided in appendix
Hyperparameter Selection. In VTAB-1k we use the recommended hyperparameter sweep and 100-traiinn/200-vaaidaattt split. We independently repeat the hyperparameter selection procedure 10 times for confidence intervals. For the other datasets we perform a single random search over 36 hyperparameter sets and select the best set based on the validation performance. This is a similar computational budget to that of (Ngiam et al. 2018). See appendices E.2 and F.T for sweep details.

---

Published as a conference paper at ICLR 2021
Table I: VTAB-IK results of different selection algorithms, using full experts trained on JFT. The average accuracy across each group of tasks and across all VTAB is reported. In each dataset, the median accuracy over 30 runs used. Bootstrapped confidence intervals 95% level are included.
NATURAL
SPECIALIZED
STRUCTURED
ALL
Baseline (No Experts)
77.4 (77.3-776]
81.6 (815-42.0)
57.2 [52.-5882]
69.8
[68007002]]
Random Expert
60.6 (59.1-6399]
81.2 (809 81.8]
56.8 (519-57.8]
63.3
Domain Prediction
75.9 (74.4-7744
81.5 [81.3-82.2]
57.0 [86.1-57.4]
69.1 [68.4-998)
Label Matching
77.6 1778878881
80.3 (7911-82.51
56.9 (55.6-57.2]
69.6 (68.9-70.0]
Performance Proxy
79.7 179.5-8001
83.6 18333-338]
55.3 [52.1-56.3]
70.2 [68.9-70.6]
Table 2: VTAB-Ik results of the baseline models and different expert architectures using kNN selection, pre-trained on ImageNet21k (IN21K) and JFT. The average accuracy across each group of tasks and across all 19 tasks shown. In each dataset, the median accuracy over 30 runs is used.
NATURAL
SPECIALIZED
STRUCTURED
ALL
Baseline
77.7 (77.-7788]
82.0 178.4
56.8
155.955221
69.8 [68.8 70.3)
IN21K
Adapters
78.1 1 (78.0-78.3)
83.5 [3.1 R3,6]
57.5
58.2
70.6 (70.3 70.9
Full
78.3 [78.1-78.6]
83.4 [33.2-366]
59.4
[587-59.8]
71.4
(711-71.6)
All Experts
78.3 [78.1-78.6]
83.6
[834-83.7]
58.8
[580-59.4]
71.2
(708-71.5)
Baseline
77.4 (77.3-7766]
81.6 (81.5-2.0]]
57.2
(528-582]
69.8
(680-20.2)
JFT
Adapters
79.0 (18.6-7911
81.3 17.2--2251
59.1
[833-6011
71.1
(705-216
Full
79.7 [29.5-8000]
83.6
[83.3-833]]
55.3
(822-5622
70.2
(68.9-0066
All Experts
80.0 (79.2-50.4]
83.7
(83.6-83.8]
58.6
(58.0-9944
71.8
(71.3-72.2]
IN21k + JFT
All Experts
80.2 (29.8-80) 80.31
84.0 (83.7-84.2]
59.5
(587-6011
72.3 (7199-22.)
Final Re-training. Using the hyperparameters from the previous step, we re-train the selected expert on the entire task (training plus validation set), In VTAB-IK, we repeat this step 3 times for each of the 10 trials of hyperparameter selection and compute the test accuracy, yielding 30 outcomes per method per task. We compute the median of these 30 outcomes as the final accuracy in the dataset.
6.4 PERFORMANCE OF DIFFERENT EXPERT SELECTION STRATEGIES
We first establish which of the expert selection strategies presented in section 5 performs best. As baseline we also try selecting a random, uniformly drawn, expert per task. Table I shows the results on VTAB-11K using full experts trained JFT. 3 the results with adapters.
Overall, all methods perform better than random selection, particularly on the NATURAL group. This confirms that selecting good experts is essential. Overall, the performance proxy (kNN) selection performs better than the other alternatives. kNN's average accuracy is 1 1% (relative) and 5.5% higher than that of the domain prediction and label matching, respectively. Thus, making use of the downstream labels offers a significant advantage in expert prediction. Therefore, in all subsequent experiments we use the kNN- selection. We did not see a strong difference for the STRUCTURED datasets. We provide an extensive analysis of the kNN accuracy distribution per expert in appendix C Appendix G shows how training experts on random subsets of the upstream data does not work well.
6.5 RESULTS ON VTAB
Table shows the average accuracy across all the 19 VTAB-1k datasets broken down by type (natural, specialized, and structured). We summarize findings as follows:
Improvement over Non-expert Baseline. All the algorithms, trained on either JFT or ImageNet2 improve their corresponding baseline. Differences most pronounced on the NATURAL datasets. While we also see improvements in SPECIALIZED and STRUCTURED datasets, some of the confidence intervals overlap. The performance of both JFT and ImageNet21k models is fairly similar in general. This is not unexpected: has been observed before that, with restricted model capacity, they perform very similarly (Kolesnikov et al. 2019). Appendix Chols the selected experts.

---

Published as a conference paper at ICLR 2021
Table 3: Accuracy on the datasets used by DAT Ngiia al. 2018 and their average accuracy.
Bootstrapped confidence intervals at 95% level are shown next to the accuracy where available. DAT uses Inception-và v3 (In-v3) and a larger network, AmoebaNet-B (Am-B). Models pre-trained on JFT.
AIRCRAFT BIRDS
CARS
CIFARIO
FOOD
PETS*
AVG.
Baseline
91.4 (10-0177) 78.8 (760-79.4) 95.6
(95.-577) 7) 97.8
(97.7-97.)) 91.3 (9:2-015) 94.5
(44-0666 91.6 (9.4-0127)
Adapters
92.5 (22--028)
79.4 (887-30 |
95.9
(95 0)
97.9
(97.8-0..)
91.6 (--0-177
94.6
(4.4--4.8
92.0 9-19--211
Full
94.8 (9.5-95.11 83.6 18.1943.9)
96.1
196. 96.31
97.8 190.7-97.9)
93.1 1928 90.21
97.0 1969 97.31
93.7 192.6 9338
DAT (In-v3)
94.1
81.7
95.7
98.3
94.1
97.1
93.5
DAT (Am-B) 92.8
85.1
95.8
98.6
95.3
96.8
94.1
*Pets results mean per class accuracy, not accuracy.
Quality of Natural Representations. The upstream datasets used to train the experts mostly contain natural images. Consequently, the spectrum of representations offered by our models seem very effective in downstream natural datasets. More concretely, all models lead to improvements over the baseline performance, with average gains ranging from 1 %% to over 3.3% on the 7 natural datasets.
Full vs. Adapters. JFT Experts. Full models outperform adapters convincingly in NATURAL and SPECIALIZED datasets. However, they do a poor job on STRUCTURED datasets -mainly due to the failure on one specific dataset. ImageNet21k Experts. In this case. the advantage of full experts comes precisely from STRUCTURED datasets. Appendix E provides results broken down by each dataset.
Combining All Experts. The previous results suggest adding all types of expert models to the selection pool (full or adapter, trained on JFT or ImageNet), since the optimal expert architecture and upstream data vary per task. Results are remarkable: kNN is able select good candidates to fine-tune from a pool of almost 600 models, and the relative improvement over the Baseline accuracy across all VTAB datasets 3.6%, showing gains on all dataset types (see last row table 2
6.6 OUR APPROACH VS. DOMAIN ADAPTIVE TRANSFER
Domain Adaptive Transfer (Ngiam et al. 2018) (DAT) also relies on specialist models pre-trained on JFT. First it trains a generalist model on the upstream data, similar to our B. For any new task, then re-weights the upstream images based on a forward pass on the downstream data, and fine-tunes a new specialist model using the re- re-weighted upstream data. Finally, the model is further tuned on the target task. DAT falls outside of our transfer setup presented in section 5 as the downstream data directly influences the upstream training. This incurs a significant cost learning every new target task.
Remarkably, our algorithm works in setups where access to upstream data is not available (e.g. for privacy proprietary reasons). We also use downstream labels, which proved to carry key information about the task (see section And most importantly, our method is more practical by amortizing the cost of expert pre-training as more downstream tasks are served. Under same models and hardware, running KNN (with 240 models) is between 500 100x-1000> faster than fine-tuning the baseline model with the re- re-weighted upstream data. Appendix F includes additional details.
Table shows the mean accuracy over 30 trials per dataset, on the same datasets and under a similar hyperparameter budget as DAT, These tasks close to VTAB's NATURAL group and yield similar results: full experts outperform adapters. Our results are not directly comparable to those of DAT since they use Inception-v3 (Szegedy et al. (0166). and AmoebaNet-B - (Real al. 2019) architectures. Inception-v3 and R50 are similar in performance and size: the former has 24M parameters, attaining 78.8% top-1 on ILSVRC2012 ftrom-scracch,, whereas the latter has 26M parameters and attains 76.0%. The AmoebaNet-B (N=18, F=512) is 22 times larger, with more than 550M parameters. Despite the differences, our method is competitive and matches or beats DAT in half the datasets.
7 DISCUSSION
Algorithm. Our results suggest that there are strong potential benefits to using smartly routed pre- trained experts when the domain of the experts broadly matches that of the downstream tasks. We have clearly seen this with natural images. Instead, as expected, when there is a skill mismatch (c.g. trying to solve counting task with diverse single-object recognition experts) have not observed
8

---

Published as a conference paper at ICLR 2021
any significant gain or loss. Still, in these cases, the expert selector can fall back the generic model or representation. When there is an extremely relevant expert for a task -say, our flower or plans models for the Oxford Flowers 102 task-, using full network experts proved beneficial. In contrast, many datasets did not have a perfect match, and adapters seemed easier to fine-tune in these cases.
Impact. In the near future, we foresee large computer vision systems composed by wide range of pre-trained specialist modules. These modules may be based on huge amounts of data, small but high-quality curated repositories, or even private and proprietary content, and they would cover diverse spectrum of canonical tasks (object recognition, some way of narrow reasoning, counting, sorting, etc.). Some of them may not even need be end-to-end learned from data.
Future Directions. There are number of exciting follow-up research directions. Selecting and combining multiple experts for any downstream task a natural extension of our work. This could be especially useful for tasks that require understanding several concepts, not necessarily captured by a single expert. Per-example routing (i.e. applying routes tailored each individual data-point) could also lead to improvements based targeted processing, for example, in the context of tasks with instances of various difficulties. Finally, moving beyond our experts based on label hierarchies, and towards automatic discovering and training of experts could unlock even further gains.
ACKNOWLEDGMENTS
We would like to thank Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai and Jessica Yung, who worked on the original BiT, that we use baseline, and shared their insights with us. Josip Djolonga and Maxim Neumann provided very useful feedback on an earlier version of this work. Finally, we thank whole Google Brain team Zürich for many useful discussions and support.

---

Published as a conference paper at ICLR 2021
REFERENCES
Naomi S Altman. An introduction to kernel and nearest-neighhor nonparametric regression. The American Stafistician, 46(3): 175-188, 1992.
Andreas Argyriou, Theodoros Evgeniou, and Massimiliano Pontil. Multi-task feature learning. In Advances in neural information processing systems, pp. 41-48, 2007.
Shai nen-David, John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of representations for domain adaptation. In Advances in neural information processing systems, pp- 137-144, 2007.
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine learning, 19(1-2):151-175, 2010.
John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman. Learning bounds for domain adaptation. In Advances in neural information processing systems, pp. 129-136, 2008.
Rich Caruana. Multitask learning. Machine learning, 28(1):41-755 1997.
Wei- Yu Chen, YennCheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. closer look at few-shot classification. arXiv preprint arXiv:1904.0422 2019.
Wenyuan Dai, Qiang Yang, Gui-Rong Xue, and Yong Yu. Boosting for transfer learning. In Proceedings of the 24th international conference on Machine learning, pp. 193-200, 2007.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255, Icee, 2009.
Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In International conference machine learning, pp. 647-655, 2014.
Simon S Du, Jayanth Koushik, Aarti Singh, and Barnabás Póczos. Hypothesis transfer learning via transformation functions. In Advances in neural information processing systems, pp. 574-584, 2017.
Nikita Dvomik, Cordelia Schmid, and Julien Mairal. Selecting relevant features from universal representation for few-shot classification. arXiv preprint arXiv:2003.09333 2020.
David Eigen, Marc' Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of experts. arXiv preprint 1rXiv:1312.433 2013.
Theodoros Evgeniou, Charles A Micchelli, and Massimiliano Pontil. Learning multiple tasks with kernel methods. Journal of machine learning research, 6(Apr):615-6)) 5-77,, 2005.
William Fedus, Carles Gelada, Yoshua Bengio, Mare G Bellemare, and Hugo Larochelle, Hyperbolic discounting and learning over multiple horizons. arXiv preprint arXiv: 190.:0662..0.... 2019.
Christiane Fellbaum. Wordnet. The encyclopedia of applied linguistics, 2012.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference on artificial intelligence and statisties, pp. 315-323, 2011.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European Conference on Computer Vision (ECCV), pp. 630-645, 2016a.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016b.
Neil Houlsby, Andrei Giurgiu. Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. efffmeetr-efficcent transfer learning for NLP. arXiv preprint arXiv: 1902.00077 2019.
10

---

Published as a conference paper at ICLR 2021
Laurent Jacob, Jean- PhanPPhllippe Vert, and Francis R Bach. Clustered multi-task learning: A convex formulation. In Advances in neural information processing systems, pp. 745-752, 2009.
Robert A Jacobs and Michael I Jordan. Learning piecewise control strategies in modular neural network architecture. IEEE Transactions on Systems, Man, and Cybernetics, 23(2):33--355, 1993.
Seyoung Kim, Eric P Xing, et al. Tree-guided group lasso for multi-response regression with structured sparsity, with an application to equ mapping. The Annals of Applied Statistics, 6(3): 1095-1117, 2012.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (BiT): General visual representation learning. arXiv preprint arXiv: 2019.
Ilja Kuzborskij and Francesco Orabona. Stability and hypothesis transfer learning. In International Conference on Machine Learning, pp. 942-950, 2013.
Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard Howard, Wayne Hubbard, and Lawrence D Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, 144):541 551,, 1989.
Xiaodong Liu, Jianfeng Gao, Xiaodong He, Li Deng, Kevin Duh, and Ye- Yi Wang. Representation learning using multi-task deep neural networks for semantic classification and information retrieval. In Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguisties, 2015.
Karim Lounici, Massimiliano Pontil, Alexandre B Tsybakov, and Sara Van De Geer. Taking advantage of sparsity in multi-task learning. arXiv preprint arXiv:0903. 1468, 2009.
Yishay Mansour, Mehryar Mohri, and Afshin Rostamizadeh. Domain adaptation with multiple sources. In Advances in neural information processing systems, pp. 1041-1048, 2009.
Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. Cross-stitch networks for multi-task learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3994 4003, 2016.
Jiquan Ngiam, Daiyi Peng, Vijay Vasudevan, Simon Kornblith, Quoc V Le, and Ruoming Pang. Domain adaptive transfer learning with specialist models. arXiv preprint /8/ 2018.
Maxime Oquab, Leon Bottou, Ivan Laptev, and Josef Sivie. Learning and transferring mid-level image representations using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1717-1724, 2014.
Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on knowledge and data engineering, 22(10):1345 1359, 2009.
David Pardoe and Peter Stone. Boosting for regression transfer. In Proceedings of the 27th Interna- fional Conference on International Conference Machine Learning, pp. 863-870, 2010.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe Le. Regularized evolution for image classifier architecture search. In Proceedings of the aaai conference artificial intelligence. volume 33, pp. 4780-4789, 2019.
Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. Advances in Neural Information Processing Systems, pp. 506-516, 2017.
Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Efficient parametrization of multi- domain deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 81 1119771 2018.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick. Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv: arXiv:1606.0440 04671, 2016.
11

---

Published as a conference paper at ICLR 2021
Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson. CNN features off-the-shelf: an astounding baseline for recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 806-813 2014.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.066 06658, 2017.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In Proceedings of the IEEE international conference computer vision, pp. 843-852, 2017.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818 2826.,,62226 2016.
Chuanqi Tan, Fuchun Sun, Tao Kong, Wenchang Zhang, Chao Yang, and Chunfang Liu. A survey on deep transfer learning. In International conference on artificial neural networks, pp. 270-279. Springer, 2018.
Chang Wan, Rong Pan, and Jiefei Li. Bi-weighting domain adaptation for cross-language text classification. In Twenty-Second International Joint Conference on Artificial Intelligence, 2011.
Zirui Wang. Theoretical guarantees of transfer learning. arXiv preprint arXiv:1810.05996. 2018
Karl Weiss, Taghi M Khoshgoftaar, and DingDing Wang. A survey of transfer learning. Journal of Big data, 3(1):9, 2016.
Yuxin Wu and Kaiming He. Group normalization. In European Conference on Computer Vision (ECCV), September 2018
Qizhe Xie, Eduard Hovy, Minh ThangTThng Luong, and Quoc V Le. Self-training with noisy student improves imagenet classification. arXiv preprint arXiv:1911.04252, 2019.
Yonghui Xu, Sinno Jialin Pan, Hui Xiong, Qingyao Wu, Ronghua Luo, Huaqing Min, and Hengjie Song. unified framework for metric transfer learning. IEEE Transactions on Knowledge and Data Engineering. 29(6):158-1177 2017.
I Zeki Yalniz, Hervé Jégou, Kan Chen, Manohar Paluri, and Dhruv Mahajan. Billion-scale semi- supervised learning for image classification. arXiv preprint arXiv: 00000000046.... 2019.
Xi Yan, David Acuna, and Sanja Fidler. Neural data server: A large-scale search engine for transfer learning data, 2020.
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. The visual task adaptation benchmark. arXiv preprint arXiv:1910.04867, 2019.
Zhanpeng Zhang, Ping Luo, Chen Change Loy, and Xiaoou Tang. Facial landmark detection by deep tulti-task learning. In European conference computer vision, pp. 94- Springer, 2014.
12


Reviews:
[{"review_id": "youy-IT_3y", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The authors propose to choose the pre-trained model from several experts for better transfer learning performance.\n\nQuality: The method is a heuristic one, but generally simple and reasonable. The authors perform plenty of experiments to show the influence of different design and how the method compares with other methods. Overall, the paper is sound.\n\nClarity: The paper is well written and easy to follow. I have two questions:\n1. Does the number $n$ of adapters equals the number of experts? Or (because there are four blocks) does the number of experts equal the combination of adapters, $n^4$?\n2. I have one important question in experiments: Did you compare your method with direct transferring B (the initialization model)? This is the actual baseline to justify the effect of experts and your selection scheme. I think because of residual connection in adapters, B is included in the experts. But still, B could be the best choice but not selected due to the imperfection of selection scheme. I would like to see the results, I may change my score based on that.\n \nOriginality: The paper is a novel method for transfer learning, though using multi-expert system/adapters in network are not new.\n\nSignificance: This paper could be an inspiring work for transfer learning community. Using multi-expert system for transfer learning seems under explored in deep learning era.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Jk3GsONx6zR", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe authors propose a four-stage transfer learning strategy:\n1. pre-train a baseline model on the entire source data\n2. fine tune the baseline model on different parts of the source data (determined by the label hierarchy) to get multiple experts.\n3. for the target task of interest, select the best expert based on the nearest-neighbor performance \n4. fine-tune the selected expert on the target task\nThe authors tested their method on two transfer setting: 1) from ImageNet to VTAB 2) from JFT to VTAB.  Empirical results show that their method outperform the baseline.\n\nStrength:\n1. The paper is clearly written and easy to understand.\n2. Exploiting the label hierarchy to train multiple experts and select the best one makes sense.\n\nWeakness:\n1. Maybe I am missing something, but I don't see much new insights from the paper. When doing multi-source transfer, selecting the best performing expert for the target task seems like a straightforward baseline.\n2. The method does show empirical improvements, but perhaps not strong enough. From IN21k to VTAB-1k, the full model achieves 71.4 while the baseline achieves 69.8 (2.29% relative improvement). From JFT to VTAB-1k, the full model achieves 70.2 while the baseline achieves 69.8 (0.57% relative improvement). \n\nOther questions:\n1. I didn't quite understand what do you mean by \"combining all experts\" in section 6.5. Is that the same as doing ensemble? If so I think the comparison between \"All Experts\" and \"Baseline\" is not very fair.", "extra_scores": {}, "extra_text": {}}, {"review_id": "D2pTcZ0iZo", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n**Summary:**\n \nThis paper presents a novel method for obtaining better representations for transfer learning. Specifically, instead of using a generic representation for various down-stream tasks, this paper proposed to create a family of expert models in pre-training, and selectively choose one expert to generate representation depending on the target transfer learning task. A simple yet effective k-nearest neighbor strategy is used for picking the best-fitting expert. This paper has extensive experiments including pre-training models on two large-scale image datasets, and evaluated on two transfer learning benchmarks (VTAB and datasets from DAT).\n\n\n**Reasons for score:**\n\nThe general idea of this paper (i.e. replacing generic representation with one from target-dependent expert model) is very intuitive, and the experimental validations are very solid. However, the novelty and technical contribution of this paper is only moderate. Overall, I think it's a good paper and may inspire future work on more efficient and effective transfer learning.\n\n\n**Pros:**\n\n1. The proposed idea is intuitive, and empirically very effective. Different from focusing on architectures for transfer learning, this paper focused on using different representations to improve transfer learning quality. This is complementary with many existing techniques on transfer learning.\n2. The experimental validations are extensive and solid, e.g. all reported accuracies have confidence interval so the comparison is more informative.\n3. The paper is well-written and easy to follow.\n \n**Cons:**\n\n1. In Section 4.1 two variations of \"MoE family\" are proposed, i.e. Full ResNet Modules and Adapter Modules. For Adapter Modules, it seems all experts share blocks and only differ in adapter module (Fig.3 b). However, the effectiveness of constructing \"expert\" model in this way lacks supporting evidence, i.e. how well each \"expert\" performs on the corresponding/non-corresponding domains?\n2. I am wondering if Table 1 should add performance comparison with the baseline model B, so it would be more straightforward whether the expert branch selected by the proposed strategy is more effective on the target datasets.\n3. For \"All Experts\" in Table 2, I find it unclear as in Section 6.5 \"Combining All Experts\" it doesn't explain how this model works. Does it mean all experts are simultaneously selected, i.e. $x_i := F_i(x_{i-1} + \\sum_e a_e^{(i)}(x_{i-1}))$? In that case, if each adapter module introduced 6% parameters (Section 4.1) the extra parameters will be non-negligible.\n4. Please clarify: as shown in Figure 3 (a) ResNet-50 has four blocks and adapter module is added before each block; does the proposed system support choosing different adapter module at different block? E.g. for the first two blocks of a specific dataset $a^{(1)}_i$ and $a^{(2)}_j$, is it possible that $i != j$? In other words, the \"performance proxy\" strategy is applied to each block, or at the end of the entire network?\n5. Section 6.1 mentioned that ImageNet21k use 50 experts and JFT use 240 experts, but in supplementary JFT seems to have 244 experts. Also from Table 4 in supplementary C. 6, some dataset choose \"baseline\" as the selected expert. Does the \"baseline\" serve as a standalone expert, or a base of all experts (e.g. in adapter modules setting do the adapter serve as residual to the baseline)? There seems to be some inconsistencies here.\n\n\n**Questions during rebuttal period:**\n\nPlease address my questions in the cons section.\n  \n\n**Some typos and minor issues:**\n\n--  Supplementary D.1, \"Unconditional pre-training\" the last sentence is incomplete.", "extra_scores": {}, "extra_text": {}}, {"review_id": "BosXVuWQqZl", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors address transfer learning scenarios. In particular, the authors resort to training to a diverse set of experts and \"cheap\" performance proxies to select, for a given task, the relevant expert. This \"per-task routing\" is conducted via a nearest neighbor classifier based on a reduced representation for each expert. Two variants are considered: (1) full ResNet50 models are used (one for each expert) and (2) \"compact adapter modules\", which depict expert layers between ResNet block (all experts are learnt simultaneously with the same model backbone).\n\nComment: The procedure how the kNN proxy works remains a bit unclear in the main text (maybe move some material from the supplemental material to the main text).\n\nPositive:\n- On-pair performance with other approaches,  but faster and less parameters\n- The authors provide test statistics (i.e., not only single runs)\n- The work is well written and well structured)\n- Comprehensive supplemental materia\n\nNegative:\n- The authors argue that their approach is 500-1000 times faster (main contribution), but this does not become clear in the main paper (also not in the supplemetal material?)\n- Minor: Training the full model version (i.e., full ResNet model for each expert) is very expensive from a computational perspective.\n\nOverall, I think this work might be worth being accepted at ICLR. I am not an expert for (recent) transfer learning approaches, so I might be missing something.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The presented idea is aligned with past work using multiple experts or multiple sources for transfer. However, it is positioned uniquely and cleverly in that the approach is developed with scalability in mind. Within this setting, the paper is convincing. Although the approach does not come with strong backing theory, it is intuitive and seems to work well. During the discussions phase, the authors have clarified some questions that made the paper convincing, even if it is a relatively heuristic approach. The results are strong if one is concerned with both quantitative performance and efficiency, a combination of objectives very often encountered in practice. Overall, it is expected that this idea can stimulate further research along those lines, especially since this paper is very nice and easy to read.
Decision: Accept (Poster)