Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper is about retrieving a pre-trained network from a model zoo in a computationally effective manner. The authors proposed a network topology and parameter embedding that can be used to retrieve networks that perform well on a task represented by a dataset after fine-tuning.
main_review: I find the paper to be clearly written, interesting to read, the topic to be relevant and the contribution to be novel and significant. The paper contains a significant methodological contribution, defining a new task consisting of multiple datasets. The code is provided for reproducibility. A quick inspection shows that the code is well organised and contains all the ingredients to download and prepare the data.  I recommend accept and provide minor comments below.

- line 125, it took me some time to understand why neural network is denoted $M^\tau$. I realized later on that $M$ refers to model. Indeed, the proposed method could be generalized to arbitrary models, why not make it clear in line 125?
- it is unclear to me why authors chose a title that is different from the name of their core contribution. Why not have a title "TANS: Task-Adaptive Network Search." or "TANS: Task-Adaptive Network Search with Meta-Contrastive Learning." ?
- what is the total compute capacity that is needed to reproduce the results?
- datasets used in the study are public, but it would be a good idea to provide them from one location, because they may disappear with time. Have you thought about it?
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a new Task-Adaptive Neural network search framework (TANS). Such a task needs to retrieve a good network in well pre-trained dataset-network sets. In order to obtain a task-adaptive neural network, the amortized meta-learning method is employed. Meanwhile, this meta-learning framework is trained with contrastive learning to maximize the similarity between the positive dataset-network pairs and minimize the similarity between the negative pairs. The experiments show that TANS can rapidly retrieve a well-fitted network for unseen datasets. 
main_review: Pros:
(1)  The authors propose a new task, which retrieves a good network in well pre-trained dataset-network sets. Such a task is promising and novel. 
(2)  A meta-learning framework is trained with contrastive learning to maximize the similarity between the positive dataset-network pairs and minimize the similarity between the negative pairs.
(3)  The experiments in ten real-world datasets show the effectiveness of TANS.

Cons:
I understand the novel task of this manuscript. I have some questions as follows. I want to make sure again, is it the first work to propose architecture retrieval? I think this task is very promising because we can hardly search for a suitable network given any task. The second question is if the dataset is very large, e.g., ImageNet, how to sample images in this dataset? What kind of image is the suitable image that represents this dataset? Whether the simple random sampling method is applicable? Another question is whether such a method only tackles the image classification task. What if object detection or other regression tasks? 

To sum up, this manuscript introduces a new task, which is valuable in real-world architecture retrieval. The writing is clear and easy to understand. However, there are still some unclear descriptions and questions. Some tables and figures (e.g., Figure 7, Table 3) are also too small to see results clearly.

limitations_and_societal_impact: There are still some unclear descriptions in this manuscript. The specific contents are shown in the Main review. Judging from the paper, there is currently no direct negative impact.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes and tackles the following task: given an unseen dataset, selecting the optimal pre-trained neural network from a model zoo of networks which will obtain the best fine-tuned performance on the new dataset. To solve this problem they propose a novel contrastive learning method to learn a shared embedding space for the models and datasets which minimizes the distance of the dataset embeddings and model embeddings of models that can be well adapted to the datasets. They propose encoding the properties of a pretrained model using a combination of a topological and functional encoding and demonstrate the effectiveness of this encoding scheme. They also propose a novel method for efficient model-zoo construction and build a dataset and train their algorithm on a collection of Kaggle datasets. They demonstrate that they can achieve very high performance on unseen datasets with few training steps.
main_review: This paper tackles a novel problem of building a model zoo of pretrained networks and retrieving the optimal base model for fine tuning on an unseen dataset. This is a valuable direction of research and has strong significance for practical machine learning since picking an existing trained model from a model zoo and fine tuning it for your task is an extremely common method to quickly train an effective model on a new dataset. The methodology also includes a novel method of characterizing the model using a combination of topological and function encoding.

Unfortunately, I currently can't recommend acceptance for this paper. I believe that this work requires a stronger baseline to demonstrate the generalization of the dataset embedding.

Additional results would help ablate the effectiveness of the dataset embedding and the quality of the models and pretrained weights from the model zoo. While table 3 does ablate the choices made for the embedding, it's not entirely clear how this generalizes to unseen datasets from just those experiments. A strong baseline  that would significantly improve the quality of the paper would be for each unseen test dataset X, sample 5-10 random other test datasets, and retrieve and train the optimal models for those datasets on dataset X. This would help ablate the effectiveness of choosing random strong models from the model zoo with the effectiveness of the model embedding. In table 3, the author's do show some ablations of the different retrieval strategies which helps demonstrate their choices for the retrieval method. However, this could be strengthened by making sure if the models selected, but not trained on the specified data still performed adequate on the specified dataset.

The paper would also be much improved with more analysis of the efficiency of the methodology especially related to how much compute is used to generate the Model zoo. This is especially important to demonstrate the efficacy of their novel efficient model zoo construction method.

The results that they demonstrate on the unseen dataset are impressive, but difficult to directly compare. None of the existing models were pre-trained on the same training kaggle datasets and the model zoo was trained on both those and the imagenet dataset through the OFA initialization. It would also be helpful to compare the models if the flops of each model was included in Table 1. Something that help demonstrate their algorithm would be testing their ability to transfer to common unseen datasets like cifar-10/cifar-100/imagenet.

The paper is generally clear and easy to follow, but some of the details could be clearer. How are the positive and negative pairs selected? How exactly is the meta performance surrogate model used? How were models trained to create the model zoo? It seems both a 14,000 model model zoo and an efficiently constructed model zoo were created? It's not entirely clear which is used in which experiments. Was the model zoo trained off OFA in the meta-fashion or each architecture separately?

==================================================================================================

Post-Rebuttal Update:
I would like to thank the authors for addressing many of my concerns and especially including the additional baselines and experiments. I believe that the baseline exploring the capabilities of the models in just the model zoo significantly improves the papers and I think the paper would benefit from the inclusion of the full baseline. In addition, the inclusion of results with training the competitors on the same kaggle meta-training addressed my concerns on fairness with regards to pretraining on the kaggle datasets. The authors have definitely demonstrates the effectiveness of your task-adaptive retrieval mechanism and addressed my main concern. Given that and the inclusion of more discussion on the comparative pretraining costs, and additional experiments demonstrating the robustness of the method on other datasets, I would recommend acceptance for this paper and raise my score to a 7.
limitations_and_societal_impact: They discuss it in the appendix. They mostly address the societal impacts, but maybe could include some additional discussion of the possible impact of biased initializations.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper introduces an interesting problem of Neural Network Search (NNS), aiming at searching for the optimal pretrained network for a novel dataset from a model zoo. The authors propose a framework TANS to tackle this problem. The proposed method constructs a model encoder and a dataset encoder to calculate the embedding of model and query set respectively. By applying the contrastive loss to maximize the similarity between a dataset and a network that obtains high performance on it, the framework is trained jointly with a surrogate performance predictor. At testing time, the framework is able to retrieve the well-fit architectures from the model zoo given an unseen dataset.

The main contribution of this paper is that it introduces a novel problem Neural Network Search and also present a decent method to tackle the problem.
main_review: Pros:
1. The paper is well written and easy to follow.
2. The introduced problem NNS is novel to me. And the problem is practical and does exist in the real deployment scenario. 
3. The proposed method is technically sound and the authors also have provided sufficient details and ablation study to validate the effectiveness.

Cons:
1. I'm confused about the proposed functional embeddings. I'm not sure why introducing it since we've already have a topology embeddings. Does it play a significant role in the proposed framework. In table 2, the author has conducted an ablation w.r.t. the topol embedding but fail to conduct a similar one on functional embedding. I would like to see this ablation.

2. I would like to see an ablation study on the surrogate model. Does the predicted performance correlate with the GT acc well?

3. In Fig. 6c, the authors has visualized the the cross-model latent space to demonstrate the claim that "the network whose embeddings are the closest to the query dataset achieves higher performance on it" (L358). However, I think the best way to validate this point is to calculate the correlation between the latent distance and the real performance to see if they correlate well.

4. The authors also show the effect of model-zoo size on accuracy in Fig. 6d. I'm wondering the concrete data reduction method here? Do the author simply shrink the data size of each dataset or just simply remove some datasets? These two methods might have different impact on the performance. I would like the authors to clarify this and it would be better to see the results of both two methods.

limitations_and_societal_impact: The author has provided in the supp.
review_title: null

Meta-review: A strong paper on a subject that many people in the community are likely to find interesting.  Multiple reviewers agreed that this submission could be a Top-50 paper. But it was felt that the work could provide a better analysis, and possibly further experiments on the model zoo creation and corresponding properties. A healthy discussion occurred during the author response and discussion phase. New experiments and further explanations from the authors motivated one of the reviewers to upgrade their score and assessment of the paper to an accept, leading to all reviewers recommending acceptance. More discussions of comparative pretraining costs and additional experiments provided by the authors during the rebuttal helped to demonstrate the robustness of the method on other datasets.

The work addresses a subject that the reviewers felt is interesting and important, proposing a network topology and parameter embedding that can be used to retrieve networks that perform well on a task represented by a dataset after fine-tuning. The paper proposes an amortized meta-learning framework to learn a cross-modal latent space using a contrastive loss. Their goal is to maximize the similarity between a dataset and a high-performing network on it, and minimize the similarity between irrelevant dataset-network pairs. 

Given the good reception by the reviewers after the author response and the good level of discussion about this work, the AC recommends a Spotlight.
Decision: Accept (Spotlight)