Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The authors introduce the “scalable diverse model selection” task and introduce several tools and benchmarks for evaluating model selection methods in this setting. They show that current transferability and model selection methods fail to beat simple baselines in this new setting. They analyze the reason for this case and provide techniques to improve performance. They develop PARC, a method that outperforms other methods on diverse model selection.
main_review: Pros: 
- The paper is well written and it is a timely paper to discuss the limitations of existing model selection methods, such as they don’t consider different model architectures. The comparison of different methods regarding probe size, architectures, and feature dimension might be interesting to readers in this area.


Cons:

- The authors compare several model selection algorithms, i.e., probabilistic methods and feature-based methods. However, those algorithms are not introduced anywhere and the authors may assume that readers are familiar with those algorithms.
- My major concern is that the proposed benchmark can not fully evaluate “scalable” model selection and it is not clear how to scale the benchmark. The authors collected 8 source datasets to 6 target datasets across 4 different commonly used architectures. However, the size of the source datasets and target datasets in total is 32, which is not quite big. The architectures used (ResNet-50/18, GoogleNet, AlexNet) are not quite representative. Architectures with different input resolutions would be great to add. The target 6 datasets are also quite small, among which their performance is almost saturated, such as the accuracy of Pets/CIFAR10 is above 95%. There are Google VTAB datasets (19) which could be much larger as the target sets.

- It is not clear why the new proposed PARC has an advantage over existing methods such as RSA. The PARC seems pretty similar with the Label-Feature Correlation proposed in [1], which would be great to include for discussion. 

- It would be clearer if the authors can make clear what exactly are the models and datasets described in section 6.1. It is referenced but not stated anywhere.

- It is not clear how the benchmarks can be released with protocols and facilitate future comparisons.

[1] A linearized framework and a new benchmark for model selection for fine-tuning, arXiv 2021
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper proposes a model selection method wherein best experts are selected from a model zoo to fine-tune on the target task. The proposed method "Pairwise Annotation Representation Comparison" is an improvement over RSA [8], instead of using a small dnn trained on target task (or probe network) the authors use spearman correlation between RDM of features and label as model selection score (called PARC score). The proposed method is compared to many baselines NCE, LEEP, RSA, DDS etc. and shows better correlation to fine-tuning accuracy.
main_review: Strengths:
+ The proposed method is compared to many different baselines. In Tab. 2, 3 and 4 the proposed method shows better correlation to fine-tuning accuracy compared to RSA, LEEP, NCE, Nearest Neighbors model selection baselines. Tab. 2 also performs an ablation of the correlation score when architecture, source dataset and target dataset are changed.
+ The approach is extended to show that selection with the same scoring function works for object detection tasks. Model selection for object detection is not studied in any of the prior arts.

Weakness:
- The results on transfer between different models to the target task are missing. E.g. does imagenet trained expert or some of the candidate architecture always work better than the rest?
- Some other relevant baselines to the proposed method which are not discussed are below:
1. Cui et al "Large Scale Fine-Grained Categorization and Domain-Specific Transfer Learning" -- Uses EMD distance between average class vectors of source, target task to estimate transferability
2. Achille et al "Task2Vec: Task Embedding for Meta-Learning" -- Estimates an embedding space for tasks. If source and target task are close in the embedding space, the transfer is better
3. Deshpande et al "A linearized framework and a new benchmark for model selection for fine-tuning" -- Proposes a Label Feature Correlation method for model selection that uses correlation between label/feature similarity matrix. The score function is similar to proposed approach of Annotation (i.e. label), Representation (i.e. feature) Comparison.

limitations_and_societal_impact: Limitations are addressed in detail in Sec. 7.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper studies model selection for transfer learning: given a number of source or upstream models, and a new downstream or target data, how can we choose the best model to finetune? The paper offers two main contributions. First, it proposes a specific benchmark with source models, target tasks, and an evaluation metric to compare different approaches. Second, after testing on a number of methods from the recent literature, it identifies some algorithmic aspects that seem to hurt these methods and proposes an alternative approach called Pairwise Annotation Representation Comparison (PARC) which addresses those. Experiments suggest PARC outperforms the original algorithms.
main_review: I'll list strong (+) and weak (-) points, together with questions (?).

(+) I think model selection for transfer learning is indeed of very high practical importance.

(+) The code and benchmark (pre-trained models?) will be open-sourced, thus helping people to use and report the benchmark.

(-) The source datasets (and models) seem fairly small. It would make sense to add models trained on Imagenet (or parts of it).

(-) My main concern is related to the evaluation metric (presented in equation 2). The goal of model selection for transfer is to identify one model to transfer (or, say, 2 or 3 if one has more budget). We only care about the finetuned "quality" of the selected model. However, the correlation approach presented in eq. (2) measures agreement for all the models --including the long tail of "irrelevant" ones. I don't think we need to account (at least so heavily) for those sub-optimal models. If the benchmark has access to the \omega_s^t for all (s, t) --that is, the fine-tuned performance for source model s on target data t--, I think a more reasonable performance metric would be something like: for some small k >= 1, the top-k hindsight performance is the average \omega_s^t over the top-k models s with highest \alpha_s^t scores. In other words, we use the \alpha's to select the models, and then we look at the finetuned performance \omega we would get (we can normalize it somehow, to give all target datasets a similar importance). I'd add two columns (k=1 and k=3 for instance) to the tables in the paper, reporting a metric like this one (other work reports a notion of regret, [1]). In this case, I'm open to revisit my score (no need to remove the current mean PC, but discuss this).

(+) The proposed heuristic where some sort of model complexity (like the second term in equation 6) seems to work well. I wonder if this is indeed equally useful when looking at a metric that focuses on choosing the right model. In any case, it would be nice if this aspect is studied in follow-up work.

(+) PARC is relatively fast to compute, and it doesn't require access to a pre-trained model in the task data (as RSA or DDS do).

(+) I think previous work hasn't been able to properly deal with the fact that different source models may offer representations with very different dimensionality. Using PCA seems like a good idea (and results suggest so too).

(+) PARC can naturally deal with tasks other than classification (as g(.) unlocks the use of any relevant mapping).

Typos:

- line 71: methods.
- line 85: transferability.


[1] - Which Model to Transfer? Finding the Needle in the Growing Haystack (https://arxiv.org/abs/2010.06402)
limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper studies methods to select source models from a diverse set, considering both the quality of the proxy for downstream performance, and the speed of computing the proxy. The paper introduces a new benchmark for evaluating the ability to choose from diverse source models, studies a number of existing methods, and highlights some shortcomings. It then proposes some tweaks to improve these. Finally, the paper proposes a new proxy PARC, which performs well on their benchmark (also using the aforementioned tricks), and also on an extended evaluation on crowd-sourced models.
main_review: *Summary*

Overall I think that the paper is well written and organized. It makes a substantial contribution to a relevant practical problem. The experiments look very thorough. Overall, I recommend acceptance, with just a few small comments, that if addressed I might increase my score further.

*Originality*

The task of selecting from different source models has been studied before. However, this work introduces a substantially more systematic study than previous work. The paper also considers practical aspects carefully. I believe this is a significant original contribution. The literature is well cited, one extra relevant paper that studies different proxies that should probably be mentioned is [Which Model to Transfer? Finding the Needle in the Growing Haystack](https://arxiv.org/abs/2010.06402).


*Quality*

Overall, the benchmark and evaluation are thorough and informative. I have no major concerns, a few comments:

- I like the detailed practical considerations: in particular reporting real-world wall clock times. To compliment these timings, it would be useful to also have an analysis of the various methods that is not tied so strongly to the particular hardware used: such as reporting total FLOPs, or even an asymptotic analysis.
- I like the clear ablation of the tricks to improve existing methods, and the core PARC algorithm.
- The additional experiments on detection are a strength, even though PARC is not the best performing algorithm.
- How much does the analysis depend on the correlation measure, would the results be significantly different if Spearman's correlation was reported in Table 1?
- It would be nice to see a summary of which models are actually selected for the different methods, and identify qualitatively their failure modes.
- The paper mentions "Furthermore, practitioners are typically interested in the trade-off between performance and inference speed" -- however, as far as I can tell, the gold standard downstream performance is just accuracy (after finetuning) and does not take into account the speed of the resulting model? (I think this is OK, but needs to be clarified that this aspect is not included).

*Clarity*

The paper is well organized and clear. A couple of minor things
- PARC acronym is used before it is defined
- The notation in Eqns (2) and (5) seems confusing. It seems that correlation is being computed between two unpaired sets, whereas isn't it computed over a set of pairs?
-  typo: "We train all source models and transfers using SGD" -> transfer

*Significance*

I think the PARC method could be useful, although it is not a major extension over existing approaches, I think it is of moderate significance. The benchmark and thorough analysis of existing approaches I think is of high significance. Having a meaningful dataset to analyze and compare methods would be of great value.


POST REBUTTAL

The additional results are appreciated. It is interesting that under Spearman the rank of the methods is quite different. I think this should be acknowledged as a limitation, but I will keep my positive score.

limitations_and_societal_impact: Limitations are well acknowledged. There is not much discussion of societal impact, however, for this generic ML/CV paper there is no need for this.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper introduced a unified benchmark to test the performance of existing transferability estimation algorithms. The main contribution is firstly its unified benchmark, and then, its experiment models and datasets are large, which show a fair comparison for all methods. Finally, they modified an existing algorithm to propose a new one, which achieves better Person correlation score over others.
main_review: 1. My main concern is the applicability of the transferability estimation algorithms in real applications. From the recent results of LEEP, NCE, and the recent ones (LogME [1] and OTCE [2]), we see that although their evaluation scores (e.g., Person correlation) vary, but the results seem quite unified: no matter what algorithm you use, the most suitable source model will always be the biggest one (say, all methods will select ResNet-50 over ResNet-18 since we know that ResNet-50 has more capacity). So I question this line of research by the following questions:
- Is it suitable to use a simple Person correlation score as the evaluation metric? Selecting the best model is a 0/1 task, if all methods give the correct model, then it is not necessary to measure their scores.
- If all methods select the model with biggest capacity  / biggest pre-trained dataset, then what's the role of these selection algorithms? We can simply use the big models as the source model, why bother selecting?
- Last, the Person correlation score is not that convincing: e.g. it is hard to understand what an improvement of 3% in Person correlation stands for.

2. It is hard to understand the word "scalable". I think all other methods can also work in the situation of increasing source models / target models. Plus, from the results, the time complexity of PARC is just comparable to others, making me really confusing about this claim.

3. There is no theoretical justification why this selection method can work.

[1] You K, Liu Y, Wang J, et al. Logme: Practical assessment of pre-trained models for transfer learning[C]//International Conference on Machine Learning. PMLR, 2021: 12133-12143.

[2] Tan Y, Li Y, Huang S L. OTCE: A Transferability Metric for Cross-Domain Cross-Task Representations[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021: 15779-15788.

--- POST REBUTTAL --- 
Author's rebuttal did address some of my concerns. But I still think this paper has some issues that need to be addressed. Plus, the writing  should also be improved. Thus, I keep my score.
limitations_and_societal_impact: No negative societal impact
review_title: null

Meta-review: In summary, the reviewers rate this paper as follows:
* 3 of 5 reviewers view it as being close to the acceptance threshold (2 of 5 slightly below, 1 of 5 slightly above, not updated from "5" to "6" in the official rating, but in a discussion comment).
* 2 of 5 reviewers recommend the paper for acceptance (with scores "7" and "8").
So the reviewers overall are leaning towards acceptance with no reviewer opposing acceptance.

The paper discusses several methods for model selection and ranking and proposes a new method (PARC) for this task. Experiments compare the methods on a new benchmark created for that purpose.

Highlights of the paper mentioned by the reviewers include:
* The presented study is enlightening and thorough.
* The subject is of high practical importance. 
* Open-sourcing the benchmark will be valuable.
* The paper is well written.
* The authors' responses addressed several of the reviewers' initial concerns.

Some concerns expressed by the reviewers include the following:
* Limited improvements of the introduced method; similar performance of other approaches.
* Some over-claiming regarding novelty of the approach.
* A lack of the definition of "scalability".
* The value of ranking models vs selecting best models is debated.
* Some related work is missing or could be discussed more appropriately.

In case the paper is published, it would be appreciated if the authors could address the concerns as far as possible. Specifically, the results presented in the authors' responses would be valuable to include.

Overall, I would recommend the paper for acceptance as a poster.
Decision: Accept (Poster)