Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: The authors propose to choose the pre-trained model from several experts for better transfer learning performance.

Quality: The method is a heuristic one, but generally simple and reasonable. The authors perform plenty of experiments to show the influence of different design and how the method compares with other methods. Overall, the paper is sound.

Clarity: The paper is well written and easy to follow. I have two questions:
1. Does the number $n$ of adapters equals the number of experts? Or (because there are four blocks) does the number of experts equal the combination of adapters, $n^4$?
2. I have one important question in experiments: Did you compare your method with direct transferring B (the initialization model)? This is the actual baseline to justify the effect of experts and your selection scheme. I think because of residual connection in adapters, B is included in the experts. But still, B could be the best choice but not selected due to the imperfection of selection scheme. I would like to see the results, I may change my score based on that.
 
Originality: The paper is a novel method for transfer learning, though using multi-expert system/adapters in network are not new.

Significance: This paper could be an inspiring work for transfer learning community. Using multi-expert system for transfer learning seems under explored in deep learning era.
review_title: Reasonable Heuristic Method, Thorough Experimental Evaluation

final_score: rating: 7: Good paper, accept
review_content: The authors address transfer learning scenarios. In particular, the authors resort to training to a diverse set of experts and "cheap" performance proxies to select, for a given task, the relevant expert. This "per-task routing" is conducted via a nearest neighbor classifier based on a reduced representation for each expert. Two variants are considered: (1) full ResNet50 models are used (one for each expert) and (2) "compact adapter modules", which depict expert layers between ResNet block (all experts are learnt simultaneously with the same model backbone).

Comment: The procedure how the kNN proxy works remains a bit unclear in the main text (maybe move some material from the supplemental material to the main text).

Positive:
- On-pair performance with other approaches,  but faster and less parameters
- The authors provide test statistics (i.e., not only single runs)
- The work is well written and well structured)
- Comprehensive supplemental materia

Negative:
- The authors argue that their approach is 500-1000 times faster (main contribution), but this does not become clear in the main paper (also not in the supplemetal material?)
- Minor: Training the full model version (i.e., full ResNet model for each expert) is very expensive from a computational perspective.

Overall, I think this work might be worth being accepted at ICLR. I am not an expert for (recent) transfer learning approaches, so I might be missing something.

review_title: Good paper

final_score: rating: 7: Good paper, accept
review_content: 
**Summary:**
 
This paper presents a novel method for obtaining better representations for transfer learning. Specifically, instead of using a generic representation for various down-stream tasks, this paper proposed to create a family of expert models in pre-training, and selectively choose one expert to generate representation depending on the target transfer learning task. A simple yet effective k-nearest neighbor strategy is used for picking the best-fitting expert. This paper has extensive experiments including pre-training models on two large-scale image datasets, and evaluated on two transfer learning benchmarks (VTAB and datasets from DAT).


**Reasons for score:**

The general idea of this paper (i.e. replacing generic representation with one from target-dependent expert model) is very intuitive, and the experimental validations are very solid. However, the novelty and technical contribution of this paper is only moderate. Overall, I think it's a good paper and may inspire future work on more efficient and effective transfer learning.


**Pros:**

1. The proposed idea is intuitive, and empirically very effective. Different from focusing on architectures for transfer learning, this paper focused on using different representations to improve transfer learning quality. This is complementary with many existing techniques on transfer learning.
2. The experimental validations are extensive and solid, e.g. all reported accuracies have confidence interval so the comparison is more informative.
3. The paper is well-written and easy to follow.
 
**Cons:**

1. In Section 4.1 two variations of "MoE family" are proposed, i.e. Full ResNet Modules and Adapter Modules. For Adapter Modules, it seems all experts share blocks and only differ in adapter module (Fig.3 b). However, the effectiveness of constructing "expert" model in this way lacks supporting evidence, i.e. how well each "expert" performs on the corresponding/non-corresponding domains?
2. I am wondering if Table 1 should add performance comparison with the baseline model B, so it would be more straightforward whether the expert branch selected by the proposed strategy is more effective on the target datasets.
3. For "All Experts" in Table 2, I find it unclear as in Section 6.5 "Combining All Experts" it doesn't explain how this model works. Does it mean all experts are simultaneously selected, i.e. $x_i := F_i(x_{i-1} + \sum_e a_e^{(i)}(x_{i-1}))$? In that case, if each adapter module introduced 6% parameters (Section 4.1) the extra parameters will be non-negligible.
4. Please clarify: as shown in Figure 3 (a) ResNet-50 has four blocks and adapter module is added before each block; does the proposed system support choosing different adapter module at different block? E.g. for the first two blocks of a specific dataset $a^{(1)}_i$ and $a^{(2)}_j$, is it possible that $i != j$? In other words, the "performance proxy" strategy is applied to each block, or at the end of the entire network?
5. Section 6.1 mentioned that ImageNet21k use 50 experts and JFT use 240 experts, but in supplementary JFT seems to have 244 experts. Also from Table 4 in supplementary C. 6, some dataset choose "baseline" as the selected expert. Does the "baseline" serve as a standalone expert, or a base of all experts (e.g. in adapter modules setting do the adapter serve as residual to the baseline)? There seems to be some inconsistencies here.


**Questions during rebuttal period:**

Please address my questions in the cons section.
  

**Some typos and minor issues:**

--  Supplementary D.1, "Unconditional pre-training" the last sentence is incomplete.
review_title: Expert model provides better representations for transfer learning

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:
The authors propose a four-stage transfer learning strategy:
1. pre-train a baseline model on the entire source data
2. fine tune the baseline model on different parts of the source data (determined by the label hierarchy) to get multiple experts.
3. for the target task of interest, select the best expert based on the nearest-neighbor performance 
4. fine-tune the selected expert on the target task
The authors tested their method on two transfer setting: 1) from ImageNet to VTAB 2) from JFT to VTAB.  Empirical results show that their method outperform the baseline.

Strength:
1. The paper is clearly written and easy to understand.
2. Exploiting the label hierarchy to train multiple experts and select the best one makes sense.

Weakness:
1. Maybe I am missing something, but I don't see much new insights from the paper. When doing multi-source transfer, selecting the best performing expert for the target task seems like a straightforward baseline.
2. The method does show empirical improvements, but perhaps not strong enough. From IN21k to VTAB-1k, the full model achieves 71.4 while the baseline achieves 69.8 (2.29% relative improvement). From JFT to VTAB-1k, the full model achieves 70.2 while the baseline achieves 69.8 (0.57% relative improvement). 

Other questions:
1. I didn't quite understand what do you mean by "combining all experts" in section 6.5. Is that the same as doing ensemble? If so I think the comparison between "All Experts" and "Baseline" is not very fair.
review_title: not enough technical innovation

Meta-review: The presented idea is aligned with past work using multiple experts or multiple sources for transfer. However, it is positioned uniquely and cleverly in that the approach is developed with scalability in mind. Within this setting, the paper is convincing. Although the approach does not come with strong backing theory, it is intuitive and seems to work well. During the discussions phase, the authors have clarified some questions that made the paper convincing, even if it is a relatively heuristic approach. The results are strong if one is concerned with both quantitative performance and efficiency, a combination of objectives very often encountered in practice. Overall, it is expected that this idea can stimulate further research along those lines, especially since this paper is very nice and easy to read.
Decision: Accept (Poster)