Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This work investigates the estimation of model reusability when the size of target data is small. Specifically, features are extracted for tasks and pre-trained models, respectively. Then, the metric distance between them is optimized as MRE score. The theoretical analysis and the empirical study confirm the effectiveness of the proposed method. 
strengths_and_weaknesses: Strong.  
1.	With the development of pre-training, evaluating the reusability of existing pre-trained models appropriately is important for applications.  
2.	The framework is elaborated with the example of classification.  
3.	The empirical study shows that the proposed method can learn better MRE score than existing methods.  

Weak.  
1.	Some notations are not clear. For example, the loss in Line 108 contains a term of $v(\mu’, h’)$ without any explanation about how to obtain it. Besides, it is better to elaborate the feature generator for models, i.e., $c_h$ and the validation MRE function $v_{valid}$ with examples to help illustrate.  
2.	For classification, the proposed method requires the label space for pre-trained models which is unavailable for models learned without labels. It may limit the application of the proposed method, which is only applicable for supervised pre-trained models.  
3.	The current experiments only contain models pre-trained on target data. It is better to include existing pre-trained models as in LogME to demonstrate the effectiveness for a diverse model zoo.  

questions: My major concern is about the application for unsupervised pre-trained models, which demonstrate better transfer performance than supervised counterparts.
limitations: Yes.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes synergistic learning, a new metric for Model Reusability Evaluation (MRE) of pre-trained models to downstream tasks with small data. This task-model metric is learned using a set of pre-trained models and one or more data providers. Given a validation MRE function, synergistic learning establishes a metric between data providers and pre-trained models so that this metric distance can be used as an MRE score. The authors propose four different types of synergistic models: 1) basic model, 2) task transform model, 3) model transform model, 4) ensembler model. Experiments are run on benchmark MNIST, CIFAR-10, dSprites, CIFAR-100 and MiniImagenet datasets, and show that the proposed method is competitive as a MRE metric compared to SOTA methods.
strengths_and_weaknesses: Strengths:
- The paper tackles the important problem of learning to select pre-trained models for downstream tasks with small training data. 
- The paper is technically sound and the claims are accompanied by theoretical and experimental analyses.
- The experiments show that the proposed method is successful as a MRE metric, compared to counterpars.

Weaknesses:
- The main weakness that I see is related to clarity and organization of the paper, especially in terms of definitions and assumptions. For example, it seems that feature extractors are an important part of the proposed method, however this assumption is not clearly stated until later in the paper. Also, important assumptions for the small data transfer scenario such as re-training of heads only (lines 187-195) are provided later in the paper. In general, the presence of definitions and assumptions in several parts of the paper make it difficult to read, and a big effort is required for understanding the ideas, assumptions and limitations of the proposed approach. I would suggest the authors to reorganize the paper in such a way that assumptions and definitions of the general approach are provided first, then an explanation of the proposed method (the theory and the algorithm). 

questions: Q1: In the experiments, can you clarify how many data providers are you actually using? 

Q2: How do the experimental results of the proposed MRE model relate to the bounds presented in Theorem 3.1, especially in terms of the number of the training datasets required when the feature variance is small?

limitations: Limitations are clearly stated.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper focuses on the reusability of pre-trained models with many source data. In particular, this paper aims to predict the transfer learning performance in advance when the target dataset is small. To evaluate model reusability, this paper proposes synergistic learning (metric distance converted to MRE score) as one of the metric learning methods. The proposed method experimentally shows that it is a more useful method to predict the transfer learning performance than baseline methods.
strengths_and_weaknesses: Strength

The proposed method can robustly calculate MRE scores even in situations where there is little learning data. The proposed method is also a generic method applicable to most problems.

Clarity

From the reader's point of view, the notation of this paper is somewhat confusing because it is different from that of other general transfer learning papers. It is also unclear what circumstances SQUERY and MQUERY assumed. Since there is a lot of space left in Figure 1, it is recommended to explain the situation of SQUERY and MQUERY in pictures.
Finally, some grammatical errors are seen. Therefore, this reviewer recommends that you organize your thesis well.

questions: 1. This reviewer intuitively believes that the proposed method is to perform metric learning, which consumes quite a computational cost. Isn't it more computationally expensive than the transfer learning method (fine-tuning the head of networks) performed in this paper?

2. The authors assume that the GT of model transferability depends only on the target task $u$ but does not depend on the random draw of $S_u$ from $u$ but this is not proven theoretically. This reviewer thinks this assumption is hard to realize typically.

Regarding the above questions, please correct me if I am wrong or misunderstood.
limitations: This reviewer thinks that the authors have properly stated the limitation of this paper.
review_title: null

Meta-review: metareview: There is a consensus among the expert reviewers that this paper tackles an important problem, is technically sound, and has a sufficient contribution for publication at NeurIPS2022. Synergistic learning is still in its infancy and as a result, requires visibility from the community. The proposed methods for calculating model reusability evaluation (MRE) metrics in this paper will serve as an important baseline for subsequent research in this direction. Personally, I also appreciate this research direction as it will be a crucial part of the democratization of AI.

The main reservation is the clarity and presentation of the paper. The authors attempted to address them by providing clarification during the discussion phase and by revising the manuscript accordingly, which I really appreciate. The reviewers also acknowledged and responded positively to the authors' responses, either maintaining their high scores or increasing them. Hence, I recommend that the authors take special care of this issue in the camera-ready version. 
confidence: Less certain
Decision: Accept