Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: [Summary] This paper presents different ways of creating ensembles from pre-trained models. Specifically, authors first utilize nearest-neighbor accuracy to to rank pre-trained models, then fine-tune the best ones with a small hyperparameter sweep, and finally greedily construct an ensemble to minimize validation cross-entropy. Experiments on the Visual Task Adaptation Benchmark show the efficacy of the approach in selecting few models within a computational budget.

[Score] Overall, I found the paper is well-written with experiments using large-scale benchmarks such as JFT, ImageNet21K and VTAB datasets. I like the problem of model selection for transfer learning. However, my major concern is about the novelty of the paper including concerns regarding prior works. Given the lack of novelty and convincing experiments, I vote for rejecting the paper. Hopefully the authors can address my concerns in the rebuttal period. 

[Weaknesses] The technical novelty of the paper is very limited. Besides combining few prior methods (e.g., Puigcerver et al. (2020); Caruana et al. (2004)) and then performing large scale experiments on JFT/ImageNet21K datasets, what are the main contributions of the paper are not clear. Although I admit that papers on analysis or study of different methods are quite interesting, I failed to find any major insights from the study of different diverse ensemble techniques. Is the upstream pre-training achieves better accuracy than that from the downstream fine-tuning stage the major take away message of the paper? Authors should clearly explain the major contributions of the paper.

There are few recent papers which discuss model selection for transfer learning. E.g., Duality Diagram Similarity: a generic framework for initialization selection in task transfer learning, ECCV 2020; DEPARA: Deep Attribution Graph for Deep Knowledge Transferability, CVPR 2020. How is the proposed approach related to these prior works? These paper should be clearly discussed with proper comparison in the experiments. 

Comparison with prior methods is not satisfactory. Authors should clearly discuss what are the different ways of selecting models and creating ensembles out of that in the experiments. Specifically, what are the different alternatives to KNN and greedy approach used to construct ensembles? What about the performance of those methods? How is the proposed simple approach comparable to them in terms of performance vs complexity. E.g., how is the proposed approach comparable to the pretrained model selection strategy based on Task2Vec: see TASK2VEC: Task Embedding for Meta-Learning?

How is the proposed method related to Leep: A new measure to evaluate transferability of learned representations? Furthermore, how is the current approach comparable to a simple baseline on fine-tuning with early stopping?

Figure 1 is not clear and it is not described clearly anywhere in the paper. I would like the authors to clearly explain this figure either in the caption or text in the introduction section.
review_title: Recommendation to Reject based on limited novelty and lack of convincing experiments

final_score: rating: 3: Clear rejection
review_content: Summary:
Paper proposed an ensemble learning approach for the low-data regime. Paper uses various sources of diversity - pre-training, fine-tuning and combined to create ensembles. It then uses nearest-neighbor accuracy to rank pre-trained models, fine-tune the best ones with a small hyper-parameter sweep, and greedily construct an ensemble to minimize validation cross-entropy. Paper claims to achieve state-of-the art performance with much lower inference budget. 

Recommendation: Based on my understanding of the paper I recommend a clear rejection. Please look at the details below: 

Strength: 
1) Authors have tried to lot of experiments and give summary of conclusion/results in section 4. 

2) Experimental setup is clear and the motivation is valid. 

Weakness/Questions: 
1) Paper was very hard to read. I had to go back and forth between pages to make sense of what’s defined and make my own definitions in many cases. In some cases, terms are defined but never used and in other cases terns are never defined. For example, 
a) AugEnsembles: Where is this used?
b) ExpertEnsembles: Where is this defined? 
c) HyperExperts: Where is this defined?
d) AugExperts: Where is this defined? 

2) In figure 2, Single-model SOTA has only one model. Do you have a graph for total cost (training + inference) vs VTAB_{1K} performance for all the models that are shown in figure 2? Only showing an inference budget may not tell the entire picture here. 

3) In Table 2, how is computational cost different for different sources of diversity (D, U and C)? If C needs more computational cost than U and D then is the comparison fair? 

4) Appendix A.2 mentions the hyper parameters used when using “hyper ensembles” and then there is a default hyper parameter sweep - “Default Hyper Parameter Sweep” in appendix A.1. 
Did you find any pattern in the hyperparameters with the best model?  How were the hyperparameters chosen for baselines in table 1? 


minor: 
1) VTAB should have been defined just before listing contributions - 
“new form of diversity improves on the Visual Task Adaptation Benchmark (VTAB) SOTA by 1.8% (Zhai et al., 2019).

2) Paper repeatedly cites Puigcerver et al 2020 [1] to justify experimental framework or as a follow up paper which is also very similar to the current paper in terms of motivation. 

[1] Puigcerver, Joan, Carlos Riquelme, Basil Mustafa, Cedric Renggli, André Susano Pinto, Sylvain Gelly, Daniel Keysers, and Neil Houlsby. "Scalable transfer learning with expert models." arXiv preprint arXiv:2009.13239 (2020).
review_title: Papers needs more clarity, better writing and total computational cost justification. 

final_score: rating: 3: Clear rejection
review_content: This paper does achieve good performance but its method is quite about engineering using very intuitive training tricks that everybody could be able to use given a lot of GPU machines. I would not like to encourage such work to be published as a research paper.

Pros:

1. The proposed framework achieves a good performance compared to its related works.

2. It is a good organization of a lot of training techniques, and a good reference for engineering.

Cons:

1. No technique contribution. The main framework of this submission is very similar to the existing work [Scalable Transfer Learning with Expert Models] which has not been officially published but only on arXiv. Besides the common methods of pre-training and ensembling, it involves three "new" methods in its main framework: the first one is kNN selection on pre-trained models (referred to the same technique in the work [Scalable Transfer Learning with Expert Models]); the second is the hyperensembles by fine-tuning multiple diverse copies of the models (referred to the hyperparameter sets used in another related work [Big transfer (BiT): General visual representation learning]); and the last is greedy ensemble (referred to the third related word [Ensemble selection from libraries of models]). Not sure what is the contribution of this submission.

2. The paper is quite about engineering tricks or combinations of tricks. In addition, in terms of engineering, it is not fair to compare to related methods under the condition of using the same numbers of pre-trained models. A better way may be based on the total computational COSTS such as the max running epochs, the network architectures, the total training time under the same usage of GPU machines.

review_title: Three main "contributions" in its framework are all from exisiting (related) works!

final_score: rating: 5: Marginally below acceptance threshold
review_content: Update after author response: I appreciate the authors' efforts to address my concerns and to raise some interesting points I missed. I still find the paper's insights are lacking some novelty to be published, but I think that this line of research is worth it! 


------------------------


In the low data regime, the use of transfer learning techniques collides with a widely used strategy: training multiple models for different purposes, being the main obstacle the lack of a clear diversity source.

This paper proposes a simple method to circumvent this problem: identifying pre-training itself as an easily accessible and valuable form of diversity and proposing the greedy combination of several pre-trained models.  Experiments show that the proposed strategy can achieve state-of-the-art performance on 19 different tasks. One necessary assumption of the method is the availability of a large pool of related models and the ability to look at the target data to make a decision on which models to fine-tune.

One of the critical points of the method is the use of cheap proxy metrics which assess the suitability of a pre-trained model before training it.  To this end, the paper proposes the use of leave-one-out nearest-neighbour accuracy.

Pros:
+ The paper takes one of the most important issues of deep learning: training high-performance models in the low data regime. 
+ The results section is well structured and experiments are convincing. The proposed method is evaluated from several points of view.

Cons:
- The paper refers to a previous publication (Puigcerver et al., 2020) and from this point of view, the proposal represents only an incremental step, with a low level of novelty. 
- The description of the method is not very specific and it refers to other existing methods as the main steps (Puigcerver et al. (2020) and  Caruana et al. (2004))
- The proposed method is based on heuristics and there are no hints about why it does work. Diversity is a generic concept and there is a large number of papers that have explored several measures of diversity in order to understand "when" and "why" it is helpful. I miss some references to this previous knowledge. See, for example: Bian, Yijun, and Huanhuan Chen. "When does Diversity Help Generalization in Classification Ensembles?." arXiv (2019): arXiv-1910.

My main concern is not about the results, which I think are good, but about the level of novelty with respect to some existing publications (mainly Puigcerver et al. (2020)) and the lack of experiments devoted to understanding the role of diversity. It is a well-known fact that diversity per se is not sufficient to build strong multiple classifiers, and different kinds of diversity measures are helpful to diagnose it.

review_title: Deep Ensembles for Low-Data Transfer Learning

Meta-review: All reviewers recommend that the paper be rejected.  The reviewers appreciate the line of research and is worthwhile, but find that the paper lacks in technical novelty and insight.  The AC is in consensus with their reviews due to the concerns raised regarding novelty and insight and recommends rejection.
Decision: Reject