Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper presents a large scale empirical study on pretrained model selection for transfer learning and show that a hybrid approach that combines task-agnostic and task-aware methods outperforms the existing approaches on VTAB benchmark. The paper is well written and easy to follow. Experiments using 46 pretrained models and 19 downstream tasks show the effectiveness of the hybrid strategy in selecting the right model for transfer learning with low computational complexity. 

Overall, I vote for rejecting the paper as the paper has very limited novelty and experiments are not convincing. In particular, I fail to find the major contributions of the paper except the empirical study on VTAB dataset. While papers related to empirical study are interesting and worth of acceptance, this paper does not provide any major insight that could be useful for the future research on transfer learning. Furthermore, many experiments and comparisons are missing which should be included in the paper for a better understanding of the empirical study.

How is the proposed hybrid ranking strategy comparable to the model selection approaches presented in Duality Diagram Similarity: a generic framework for initialization selection in task transfer learning, ECCV 2020; DEPARA: Deep Attribution Graph for Deep Knowledge Transferability, CVPR 2020. These papers should be clearly discussed with possible comparisons in the experiments to show the advantage of the hybrid approach.

Comparison with many simple baselines are missing in the paper. E.g., How does the hybrid strategy comparable to fine-tuning with early stopping. Can we select pre-trained models by finetuning for only few epochs? How does the number of epoch affect the final performance while comparing to the hybrid strategy? 

How is the current method comparable to Leep: A new measure to evaluate transferability of learned representations? Experiments and analysis should be included in the experiments to verify the effectiveness of the hybrid strategy.

How does the size of representation/feature affect the final performance? Does the conclusion still hold with different size of features? How does amount of data in the target task affects the performance of ranking? What is the effect of number of pretrained models on the ranking?

Mutual information between the features and discrete labels of the downstream task can be used to rank different models for transfer learning. How does the proposed hybrid strategy related to mutual information based ranking strategy? Experimental comparison should be included in the paper to verify this.

Does the ranking strategy and analysis presented in the paper limited to only classification models? In particular, can models trained using self supervised learning where there is no classification head, be used as pre-trained models in the current approach? How does the analysis change while considering self-supervised models which are now-a-days quite popular in representation learning? What about considering discriminators of generative models, e.g., VAE or BigGAN in the transfer learning analysis? More experiments and analysis should be performed in the experiments.

review_title: Reject due to limited novelty and lack of convincing experiments

final_score: rating: 4: Ok but not good enough - rejection
review_content: [Summary] This paper presents a large-scale study on model-selection strategies for transfer learning, by performing task-agnostic and task-aware strategies on a large number of models evaluated on a diverse range of tasks. 

[Strength] The problem setting is novel and interesting. The proposed quantitative measurement of the quality of selected models, named "regret" is well designed.

[Weakness] The major weakness of this paper is that it seems there is no consistent strategy to out-perform all other methods in every task. Intuitively, task-aware strategies should be better than task-agnostic strategies, but they perform similarly (almost equally) in all model pools, which is quite surprising.

Even if for the advanced strategy hybrid proposed in the latter part of this paper, the optimal pick for this hybrid method is almost identical to the linear evaluation in task-aware strategy in ResNet-50 and expert model pools.

So my biggest concern for this paper is that we don't have a take-home message, other than showing the "No-Free Lunch Theorem" in model selection. So, I hope the authors could re-emphasize what we really learn from this large-scale study.

An interesting direction might be, how we can design a really fast approximation of fine-tuning, so that we can evaluate a model's fitness only by a few iterations (within a very short amount of training time), instead of performing full fine-tuning on the target task.

Considering this limited effective information from this paper, I think it's not suitable for publishing.

  
review_title: Unclear findings

final_score: rating: 6: Marginally above acceptance threshold
review_content: Paper summary: This paper looks at the problem of efficiently choosing pre-trained models as initialization for downstream target tasks. It compares 3 strategies, a task-agnostic one which uses imagenet accuracies, a task-aware one which uses the acccuracy of linear classifiers on fixed representations, and a hybrid one which combines the two.

Pros:
+ The evaluation is fairly thorough. I especially like the fact that the authors consider the different axes along which pre-trained models differ (model capacity, generalist/experts etc.)
+ The pool of downstream datasets is large.
+ The suggested strategy is simple and easy to implement. 
+ The problem is significant in practice since almost all practical applications of neural networks have this prroblem, and the gains seem large. I wish there was more work on this problem.

Cons:
- The biggest issue is that this paper ignores several important papers publiished before on this problem. Especially of note is the Task2Vec approach, which computes model and task embeddings. I would like comparisons both in terms of accuracy/regret as well as computational cost:
Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Charless C. Fowlkes, Stefano Soatto, Pietro Perona; Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019, pp. 6430-6439

Other papers that are also relevant and should be cited and comparisons discussed:
Bishwaranjan Bhattacharjee, John R. Kender, Matthew Hill, Parijat Dube, Siyu Huo, Michael R. Glass, Brian Belgodere, Sharath Pankanti, Noel Codella, Patrick Watson; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2020, pp. 760-761

Amir R. Zamir, Alexander Sax, William Shen, Leonidas J. Guibas, Jitendra Malik, Silvio Savarese; Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 3712-3722

- The approach is not particularly novel. There is also no novel technical insight that explains the results.

- The use of the JFT dataset hampers reproducibility since the dataset is not public. I'd like to see results with JFT excluded.

For acceptance, I would definitely want to see the first of these convincingly addressed.
[Updated rating]
review_title: Good baselines for model selection, but paper ignores many prior papers on this problem

final_score: rating: 4: Ok but not good enough - rejection
review_content: ### Summary

The paper evaluates three procedures for selecting models for transfer learning. The choices are task-agnostic selection, linear training, and the hybrid approach. They empirically show that the hybrid algorithm works the best on few-shot learning on images.

### Feedback

* The paper is a straightforward paper and easily understandable. The message is practical, but not very surprising. Unfortunately, it is only on image data; it would have been great if the authors had used an example from NLP too.
* The hybrid approach is super-simple, which is nice. The results in Figure 6 confirm that how its ensemble nature helps. Although it does not necessarily outperform the linear algorithm in Figure 6. The ensembling approach also does not seems to be the optimal solution. The authors could study the generalization performance of the hybrid algorithm to provide further insights.
* An empirical run-time analysis is missing.
* While the authors indicate that all models perform comparatively poorly on the structured tasks, they do not provide specific insights about the root cause of this.
* Overall, the idea is simple and practical, but the methodological contributions of this paper is rather limited.

--------
### Post-Response Update
Unfortunately, the authors' response is not satisfactory on multiple issues. Thus, I reduce my rating by one point.
review_title: A Hybrid (Ensemble) Approach for Pretrained Model Search

Meta-review: The paper studies efficient strategies for selection of pre-trained models for a downstream task. The main concerns consistently raised by the reviewers were limited methodological novelty, insufficient experimental analysis, unclear findings, and positioning of the paper with respect to related work that was ignored in the initial version. After the author response, R4 raised the score to borderline accept (still indicating the paper is weak without proper comparisons with other methods), whereas all other reviewers remained negative. The paper does have merits, as the methods are simple, and the problem is very practical (and somewhat understudied). However, the AC agrees with the majority that the paper is not ready for ICLR. The novelty is limited and the paper would benefit from more experiments, such as comparisons with simple baselines like early stopping as indicated by R1 and R3, and other methods such as Task2vec which address the same problem. The authors are encouraged to revise the paper according to the reviewers comments and submit it to another top conference.
Decision: Reject