Task-Adaptive Neural Network Search with Meta-Contrastive Learning
Authors: Wonyong Jeong
Hayeon Lee
Geon Park
Eunyoung Hyung
Jinheon Baek
Sung Ju Hwang
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: AutoML
Meta-Learning
Transfer Learning

Abstract: Most conventional Neural Architecture Search (NAS) approaches are limited in that they only generate architectures without searching for the optimal parameters. While some NAS methods handle this issue by utilizing a supernet trained on a large-scale dataset such as ImageNet, they may be suboptimal if the target tasks are highly dissimilar from the dataset the supernet is trained on. To address such limitations, we introduce a novel problem of Neural Network Search (NNS), whose goal is to search for the optimal pretrained network for a novel dataset and constraints (e.g. number of parameters), from a model zoo. Then, we propose a novel framework to tackle the problem, namely Task-Adaptive Neural Network Search (TANS). Given a model-zoo that consists of network pretrained on diverse datasets, we use a novel amortized meta-learning framework to learn a cross-modal latent space with contrastive loss, to maximize the similarity between a dataset and a high-performing network on it, and minimize the similarity between irrelevant dataset-network pairs. We validate the effectiveness and efficiency of our method on ten real-world datasets, against existing NAS/AutoML baselines. The results show that our method instantly retrieves networks that outperform models obtained with the baselines with significantly fewer training steps to reach the target performance, thus minimizing the total cost of obtaining a task-optimal network. Our code and the model-zoo are available at https://anonymous.4open.science/r/TANS-33D6

Task-Adaptive Neural Network Search with Meta-Contrastive Learning
Wonyong Jeong 1.2+f Hayeon Lee Geon Park
Eunyoung Hyung!.21 1.2t Jinheon Baek' Sung Ju Hwang!
KAIST AITRICS" Scoul, South Korea
wyjeongekaist.com ac. kr. hayeon926kkaist. ac. .r, geon- geon.parkkkais.cco ac.
ey0301 hyungdsamsunng.......o.... com, jinheon. baek@kaist.an kr, sjbwang@2@kaist..rr ac.kr kr
Abstract
Most conventional Neural Architecture Search (NAS) approaches are limited that they only generate architectures without searching for the optimal parameters. While some NAS methods handle this issue by utilizing supernet trained on a large-scale dataset such as ImageNet, they may be suboptimal if the target tasks are highly dissimilar from the dataset the supernet is trained on. To address such limitations, we introduce novel problem of Neural Network Search (NNS), whose goal is to search for the optimal pretrained network for a novel dataset and constraints (e.g. number of parameters), from model zoo. Then, we propose a novel framework to tackle the problem, namely Adaptide Neural Network Search (TANS). Given model-zoo that consists of network pretrained on diverse datasets, we use novel amortized meta meta-learning framework to learn a cross cross-modal latent space with contrastive loss, to maximize the similarity between dataset and a high-ieghiiee righ-performing network on it, and minimize the similarity between irrelevant dataset-network pairs. We validate the effectiveness and efficiency of our method on ten real-world datasets, against existing NAS//utoMM baselines. The results show that our method instantly retrieves networks that outperform models obtained with the baselines with significantly fewer training steps to reach the target performance, thus minimizing the total cost of obtaining a task-optimal network. Our code and the model-zoo are available http://ggthub..oo/wyjjoonn//////
Introduction
Neural Architecture Search (NAS) aims to automate the design process of network architectures by searching for nigh-performing architectures with RL. [76, 77], evolutionary algorithms [43, 11], parameter sharing [6, 42], or surrogate schemes [38], to overcome the excessive cost of trial-and-error approaches with the manual design of neural architectures [47, 23, 27]. Despite their success, existing NAS methods suffer from several limitations, which hinder their applicability to practical scenarios. First all, search for the optimal architectures usually requires a large amount of computation, which can take multiple GPU hours or even days to finish. This excessive computation cost makes difficult to efficiently obtain an optimal architecture for a novel dataset. Secondly, most NAS approaches only search for optimal architectures. without the consideration of their parameter values. Thus, they require extra computations and time for training on the new task, in addition to the architecture search cost, which is already excessively high.
For this reason, supernet-based methods [8, 37] that search for a sub-network (subnet) from a network pretrained on large-scale data, are attracting more popularity eliminates the need for additional
Equul contribution.
I This work was done while author was interning at AITRICS.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Conventional Neural Architecture Search
Task-Adaptive Network Search (Ours)
Query
Supernet Fretralned on Large-Szat Datarst
Networrk Rutrioval
Mode Zoe Consrrction with Optirnal Selection
Burtact
Nettieral
000
Additionaal Training Phooes Noural Aechitectore Search
Cren---iddaal Labert Space
Best Sited
Trained Networto
Networks
Figure Comparison between conventional NAS and our method: Conventional supernet- -based NAS approaches (Left) sample subnets from a fixed supernet trained on a single dataset. TANS (Right) can dynamically select the fitted neural networks that are trained on diverse datasets, adaptively for each query dataset.
training. However, this approach may be suboptimal when we want to find the subnet for dataset that is largely different from the source dataset the supernet is trained on (e.g. medical images or defect detection for semiconductors). This is common limitation of existing NAS approaches. although the problem did not receive much attention due to the consideration of only a few datasets in the NAS benchmarks (See Figure 1, left). However, in real-world scenarios, NAS approaches should search over diverse datasets with heterogeneous distributions, and thus it is important to task aask-adaptively search for the architecture and parameter values for a given dataset. Recently, MctaD2A [31] has utilized meta-learning to learn common knowledge for NAS across tasks, to rapidly adapt to unseen tasks. However it does not consider parameters for the searched architecture, and thus still requires additional training on unseen datasets.
Given such limitations of NAS and meta-NAS methods, we introduce a novel problem of Neural Network Search (NNS), whose goal is to search for the optimal pretrained networks for given dataset and conditions (e.g. number of parameters). To tackle the problem, we propose a novel and extremely efficient task-adaptive neural network retrieval framework that searches for the optimal neural network with both the architecture and the parameters for given task, based on cross cross-modal retrieval. In other words, instead of searching for an optimal architecture from scratch or taking sub-network from single super-net, we retrieve the most optimal network for a given dataset in a task-adaptive manner (See Figure 1, right), by searching through the model zoo that contains neural networks pretrained on diverse datasets. We first start with the construction of the model zoo, by pretraining state-of-the-a architectures on diverse real-world datasets.
Arg. Accuracy (%)
Then, we train retrieval model via amortized meta-learning of a pross-modal latent space with a contrastive learning objective. Specifically, we encode each dataset with a set encoder and obtain
TANS (Ours)
88
FBNct
OFA
97
Microsoft
Custom Vision
functional and topological embeddings of a network, such that a dataset is embedded closer the network that performs well it while minimizing the similarity between irrelevant dataset-network pairs. The learning process is further guided by a performance predictor, which predicts the model's performance on a given dataset.
96
MetaD2A Google
Vision Edge
95
25
Search & Training Times (Mins)
Figure 2: Comparison with NAS (orange) & AutoML (bluc) basc- lines &Rell-oorld Datasets
The proposed Task-Adaptive Network Search (TANS) largely outper- forms conventional NAS/AutoML methods (See Figure 2), while significantly reducing the search time. This is because the retrieval of a trained network can be done instantly without any additional architecture search cost, and retrieving a task-relevant network will further reduce the fine-tuning cost. To evaluate the proposed TANS, we first demonstrate the sample-efficiene of our model zoo construction method, over construction with random sampling of dataset-network pairs. Then, show that the TANS can adaptively retrieve the best-fitted models for an unseen dataset. Finally, we show that our method significantly outperforms baseline NAS/AutoML methods real-worid datasets (Figure 2), with incomparably smaller computational cost to reach the target performance. In sum, our main contributions are as follows:
We consider a novel problem of Neural Network Search, whose goal is to search for the optimal network for a given task, including both the architecture and the parameters. We propose novel cross-mi cross-modal retrieval framework to retrieve pretrained network from the model zoo for a given task via amortized meta- meta-learning with constrastive objective.
We propose an efficient model-zoo construction method to construct an effective database of
pairs, considering both the model performance and task diversity. We train and validate TANS on a newly collected large-scale database, on which our method outperforms all NAS/AutoMI. baselines with almost no architecture search cost and significantly fewer fine-tuning steps.
2

---

2
Related Work
Neural Architecture Search Neural Architecture Search (NAS), which aims to automate the design of neural architectures, is an active topic of research nowadays. Earlier NAS methods use non-differentiable search techniques based on RL [76. 77] or evolutionary algorithms [43, 11]. However, their excessive computational requirements [44] in the search process limits their practical applicability to resource-limited settings. To tackle this challenge, one-shot methods share the parameters [42, 6, 35, 65] among architectures, which reduces the search cost by orders of magnitude. The surrogate scheme predicts the performance of architectures without directly training them [38, 75, 54], which also cuts down the search cost. Latent space-based NAS methods [38, 54, 67] learn latent embeddings of the architectures to reconstruct them for specific task. Recently, supernet- based approaches, such as OFA [8]. receive the most attention due to their high-performances. OFA generates the subnet with its parameters by splitting the trained supernet. While this eliminates the need for costly re-training of each searched architecture from scratch, but, it only trains a fixed- supernet single dataset (ImageNet-IK) which limits their effectiveness diverse tasks that largely different from the training set. Whereas our TANS task-adaptively retrieves a trained neural network from database of networks with varying architectures trained on diverse datasets.
Meta-Learning The goal meta-learning [55] learn a model to generalize over the distribution of tasks, instead of instances from a single task, such that meta-learner trained across multiple tasks can rapidly adapt to a novel task. While most meta-learning methods consider few-shot classification with a fixed architecture [56, 20, 48, 40, 33, 30), there are few recent studies that couple NAS with meta-learning [46, 34, 17] to search for the well-fitted architecture for the given task. However, these NAS approaches are limited to small-scale tasks due to the cost of roll-out gradient steps. To tackle this issue, MetaD2A [31] proposes to generate task-dependent architectures with amortized leatallennning but does not consider parameters for the searched architecture, and thus requires additional cost of training it on unseen datasets. To overcome these limitations, our method retrieves the best-fitted architecture with its parameters for the target task, by learning cross modal latent space for dataset-network pairs with amortized meta-learning.
Neural Retrieval Neural retrieval aims to search for and return the best-fitted item for the given query, by learning to embed items in latent space with neural network. Such approaches can be broadly classified into models for image retrieval [21, 14, 66] and text retrieval [73, 9, 63]. Cross- modal retrieval approaches [32, 74, 58] handle retrieval across different modalities of data (e.g. image and text). by learning common representation space to measure the similarity across the instances from different modalities. To our knowledge, none of the existing works is directly related to our approach that performs cross-modal retrieval of neural networks given datasets.
3 Methodology
We first define the Neural Network Search (NNS) problem and propose a meta-contrastive learning framework to learn a cross-modal retrieval space. We then describe the structural details the query and model encoders, and an efficient model-zoo construction strategy.
3.1 Problem Definition
Our goal in NNS is search for an optimal network (with both architectures and parameters) for a dataset and constraints, by learning a cross- moosllmoda latent space over the dataset-network pairs. We first describe the task-adaptive network retrieval with amortized meta-contrastive learning.
3.1.1 Meta-Trainin
We assume that we have a database of neural networks (model zoo) pretrained over distribution of tasks p(T) with each task 7 = [D,M",, network (or model) trained network to be found (e.g. number of parameters or the inference latency). Then, our goal is to learn a cross-modal latent space for the dataset-network pairs 1D,M" while considering the constraints s" over the task distribution p(T), as illustrated in Figure 1. In this way, meta-trained model from
M",s"); ,s"}, where D" denotes a dataset, M denotes a neural the dataset, and s" denotes
of auxiliary constraints for the
diverse (DT, .M) pairs, will rapidly generalize on an unseen dataset D; DT O D 0 by retrieving a well-fitted neural network on the dataset that satisfies the conditions s².
3

---

Crass Modal Hartrional Space via Marta-Contratiie Learning
Dataset Network Fair Accuracy Extimator
Contantie
Mo- hopehte
Bog
Auchilectoree Trpeings
Model Encoder
eRt II
Selection Construction
Real-world Datasets
tedenciing
Qarn Deseniing
Encoder
Set
Encoder
Query
Performante Predictor
(SE ( (.rru ace)
Performance Estimation Query Embeddinngs
0.98
0.97
0.98
Figure 3: Illustration for overall framework of our propased method (TANS): We first construct our model- with pareto-optimal pairs of dataset and network, rather than exhaustively train all possible pairs. We then embed model and dataset with cunptioontii model and set encoder. After that, we meta-learn the cross-modal retrieval network over multiple model-query pairs, guided by our performance predictor.
ddspive Neural Network Retrieval To learn a cross cross-mode modal latent space for dataset-networt pairs over task distribution, we first introduce novel task-adaptive neural network retrieval problem. The goal of task-adaptive retrieval is to find an appropriate network M given the query dataset D for task т. To this end, need to calculate the similarity between the dataset-network pair (D',M7 E Q x M, with a scoring function f that outputs the similarity between them as follows:
E TÉP(F)
m)) q EQ(D"; and m EM (MM
max 0.6
(1)
where Eq Q + Rdd is a query (dataset) encoder, EMs . + Idd is a model encoder, which are parameterized with the parameter & and & respectively. and feim : R x Rd + R is a scoring function for the query-mos model pair. In this way, we can construct the cross moosll latent space for dataset-network pairs over the task distribution with equation 1, and use this space to rapidly retrieve the well-fitted neural network in response to the unseen query dataset.
We can learn such a cross-modal latent space of dataset-network pairs for rapid retrieval by directly solving for the above objective, with the assumption that we have the query and the model encoder: Q and M. However, we further propose a contrastive loss to maximize the similarity between a dataset and network that obtains high performance on it in the learned latent space, and minimize the similarity between irrelevant dataset-network pairs. inspired by Faghri et al. [19]. Engilberge al. [18]. While existing works such as Faghri et al. [19], Engilberge al. [18] target image-to-texx retrieval, we tackle the problem of cross-mode modal retrieval across datasets and networks, which is a nontrivial problem as requires task-level meta-learning.
Retrieval with Meta-Contrastive Learning Our mela-contrastive learning objective for each task E P(T) consisting of a dataset-model pair M") E x M, aims to maximize the similarity between positive pairs: seim (q, mm)) while minimizing the similarity between negative pairs: fsim (q. m ), where m is obtained from the sampled target task E p(r) and m is obtained from other tasks y E p(T);+ Y & т, which is illustrated in Figure 3. This meta-contrastive
learning loss can be formally defined as follows:
Lm 0, (b) L(Jsim (q, m. ),m"t,,,, Siim (g. m 0;
= (D; m = EM (MT:)) m = (M
We then introduce C for the meta-contrastive learning:
exp(fsim (q. m mn))
max 0, a
log exp EEep(r) fsim (q. m
(3)
where E IR. is margin hyper-parameter and the score function fsim is the cosine similarity. The contrastive loss promotes the positive (q, m embedding pair to be close together, with at most margin closer than the negative (g. m=) embedding pairs in the learned cross-modal metric space. Note that, similar to this, we also contrast the query with its corresponding model: E, (fsim (q m)) fsim (q m)). which we describe in the supplementary material in detail.
With the above ingredients, we minimize the meta-contrastive learning loss over a task distribution p(T), defined with the model Lm and query Lg contrastive losses, follows:
min 0.0
Σ (m 0, x) + (T; 0.
(4)
TEP(T)

---

Mleta-Performane Surrogate Model We propose the meta- performance surrogate model to pre- dict the performance on an unseen dataset without directly training on it, which is highly practical in real-world scenarios since it is expensive to iteratively train models for every dataset to measure the performance. Thus, we meta-train a performance surrogate model S(; xb) over a distribution of tasks p(r) on the model-zoo database. This model not only accurately predicts the performance of a network M on an unseen dataset Dr, but also guides the learning of the cross-modal retrieval space, thus embedding neural network closer to datasets that it performs well on.
Specifically, the proposed surrogate model S takes a query embedding q" and model embedding m" as an imput for the given task and then forwards them to predict the accuracy of the model for the query. We train this performance predictor S(; +) minimize the mean mean-square square error loss (T; +b) = [ S(;; 2)))² between the predicted accuracy S(; a)) and the true accuracy s for the model on each task т, which is sampled from the task distribution p(T). Then, we combine this objective with retrieval objective in equation 4 to train the entire framework as follows:
min 0.0.1b TEP(T)
Σ
(T; Lg( A CT(
(5)
where X is Slyper-parameter for weighting losses.
3.1.2 Meta-Test
By leveraging the meta- leannee cross-modal retrieval space, we can instantly retrieve the best-fitted pretrained network M .M, given an unseen query dataset D E å, which is disjoint from the meta-training dataset D Q. Equipped with meta-training components described in the previous subsection, we now describe the details of our model at inference time, which includes the following: amortized inference, performance prediction, and task- and constraints- adaatiiv initialization.
Amortized Inference Most existing NAS methods are slow as they require several GPU hours for training, to find the optimal architecture for a dataset D. Contrarily, the proposed Task-Adaptive Network Search (TANS) only requires a single forward pass per dataset, to obtain a query embedding q for the unseen dataset using the query encoder Q(1;; o*) with the meta-trained parameters 0', since we train our model with amortized meta-learning over distribution of tasks p(r). After obtaining the query embedding, we retrieve the best-fitted network M for the query based on the similarity:
M
max (Jsim (q, (q, T E p(T)), Mr
(6)
p(T)} is pre-computed by the meta-trained model where a set of model embeddings {m _ | encoder Eas (MT:; o²).
Performance Prediction While we achieve the competitive performance on unseen dataset only with the previously defined model, we also use the meta-learned performance predictor S to select the best performing one among top candidate networks {M, }.K based on their predicted per- formances. Since this surrogate model with module to consider datasets is meta-learnet over the distribution of tasks P(T), we predict the performance on an unseen dataset D without training on it. This is different from the conventional surrogate models [38, 75, 54] that additionally need train on an unseen dataset from scratch, to predict the performance on it.
Task-adaptive Initialization Given an unseen dataset, the proposed TANS retrieve the network trained on a training dataset that is highly similar to the unseen query dataset from the model zoo (See Figure 4). Therefore, fine-tume tuning time of the trained network for the unseen target dataset D is effectively reduced since the retrieved network M has task-relevant initial parameters that already trained on a similar dataset. If we need to further consider constraints 8, such as parameters and FL.PS, then we can easily check if the retrieved models meet the specific constraints by sorting them in the descending order of their scores, and then selecting the constraint-fice best accuracy model.
3.2 Encoding Datasets and Networks
Query Encoder The goal of the proposed query encoder (D( - Rd is to embed dataset as a single query vector q onto the cross-modal latent space. Since each dataset consists of n data instances D = ) Q, we need to fulfill the permutation inviaiaiiiioinivaininaee condition over
5

---

the data instances X,, to output a consistent representation regardless of the order of its instances. To satisfy this condition, we first individually transform randomly sampled instances for the dataset D with continuous learnable function p. and then apply a pooling operation to obtain the query vector q = ==ExEe p (x), adopting Zaheer al. [69].
Model Encoder To encode a neural network M we consider both its architecture and the model parameters trained on the dataset D for each task т. Thus, we propose generate model embedding with two encoding functions: 1) topological encoding and 2) functional encoding.
Following Cai et al. [8], we first obtain a topological embedding os with auxiliary information about the architecture topology, such as the numbers of layers, channel expansion ratios, and kernel sizes. Then, next goal is to encode the trained model parameters for the given task, to further consider parameters on the neural architecture. However, a major problem here is that directly encoding millions of parameters into vector is highly challenging and inefficient. To this end, we use functional embedding, which embeds a network solely based on its input-output pairs. This operation generates the embedding of trained networks, by feeding fixed Gaussian random noise into each network M", and then obtaining an output vý of it. The intuition behind the functional embedding is straightforward: since networks with different architectures and parameters comprise different functions, we can produce different outputs for the same input.
With the two encoding functions, the proposed model encoder generates the model representation by concatenating the topology and function embeddings [o] v7). and then transforming the concatenated vector with non-linear function as follows: m alv] e7]) EM (MF;; o). Note that, the two encoding functions satisfy the injectiveness property under certain conditions, which helps with the accurate retrieval of the embedded elements a condensed continuous latent space. We provide the proof of the injectiveness of the two encoding functions in Section C of the supplementary file.
3.3 Model-Zoo Construction
Given a set of datasets D (Di,.... Dx) } and a set of architectures M = {Mi,... Mr). the most straightforward way to construct a model zoo 2, is by training all architectures on all datasets, which will yield model zoo 2 that contains N x K pretrained networks. However, we may further reduce the construction cost by collecting P dataset-model pairs, {D, M M where P < N x K, by randomly sampling an architecture M E .M and then training it on D E D. Although this works well in practice (see Figure 8 (Bottom)), we further propose an efficient algorithm to construct it in a more sample-efficient manner, by skipping evaluation of dataset-model pairs that are certainly worse than others in all aspects (memory consumption, latency, and test accuracy). We start with an initial model zoo (o that contains small amount of randomly selected pairs and its test accuracies. Then, at each iteration t, among the set of candidates C(t). find a pair {D, M) that can expand the currently known set of all Parcto-optimal pairs w.r.t. all conditions (memory. latency, and test accuracy on the dataset D), based on the amount of the Pareto front expansion estimated by fzoo(; (()):
(D,M)++
arg max I200 ((D, M); Z(e)
(D.M)EC()
where
= E (9D 8mec)) - gD Baco S(D,, M 2000 with Suce
parameter 22200 trained on 2, and the function gD measures the volume under the Pareto curve, also known as the Hypervolume Indicator [41], for the dataset D. We then train M on D, and add it to the current model zoo (zz. For the full algorithm, please refer to Appendix A.
4
Experiments
In this section, we conduct extensive experimental validation against conventional NAS methods and commercially available AutoML platforms, to demonstrate the effectiveness of our proposed method.
4.1 Experimental Setup
Datasets We collect 96 real-world image classification datasets from Kaggle". Then we divide the datasets into two non-overlapping sets for 86 meta-training and 10 meta-test datasets. As some datasets contain relatively large number of classes than the other datasets, we adjust each dataset
Phttps://www http://wwwwwgggg..ee/ com/

---

Table 1: Performance of the searched networks on 10 unseen real-world datasets. We report the averaged accuracy on ten unseen meta-test datasets over 3 different runs with 95% confidence intervals.
Target Dataset
Method
Epochs
FLOPs (M)
Purams (M)
Search Time
(GPU
Truining Time
(GPU
Speed Up
Accuracy (%)
MohileNerV3 [26]
132.94
4.00
257.28
1.00
94.20 94.20_.03
PC-DARTS [65]
500
566.55
3.54
1100.37
5721. 13=7w41
0.04x
79:22=1m9
Averaged
DrNAS1101
500
623,43
4.12
1501,75-4m
5659.77==444
0.04%
84.06-192
Performance
FiNNei-A [60]
50
246.69
4.3
293. 291.42:55.
0.8x
92.00-11
OFA [8]
50
148.76
6.74
121 120
226. 226.58-10.2
0.74x
91.89-u4
MetaD2A [3]]
50
512.67
6.56
2.59
145.399.285
0.74%
95.24_114
TANS (Ours)
181.74
5.51
0.002
40. 40.19-mm.
95 17:300
IANS (Ours)
50
181.74
5,51
0,002-aue
200.93-1.85
1,28x
9..28-
MohileNerV3
132.94
4.00
577.18
100%
_6
Colorectal
PC-DARTS [65]
500
534.64
4.02
2062,42aamn
12124.18=
0.04x
96 17
Histology
DrNAS1101
500
614.23
4.12
4183.20
11355
1155_18-18842
0.04X
97.51-mu
Dataset
OFA [8]
FBNNei-A [60]
50
50
215.45
134.55
6.74
4.3
121 +00
696.00.2a..
537.61
0,8x
083x
96.40-uu waz
95.43 um
(Easy)
MetaD2A [3]]
50
506.88
5.91
2.58
784
0.73%
96.57
TANS (Ours)
10
171.74
4.95
0.001 2000
98. 56
96, 87 =na
TANS (Ours)
50
171.74
4,95
0.001
492.81-21.9
1.17x
97.67 77.67-68
MohileNerV3
132.94
4.00
235
1.00
27.52.00
Classification
Food
PC-DARTS [65]
DrNAS [10]
500
500
632.67
561.8
4.12
3.62
1018.49
1276.382010
6323 40
1079.8921118
0.03x
0.04%
61 61.45
55.42-14
Dataset
(Hard)
OFA [8]
FBNer-A [60]
MetaD2A [3]]
50
50
50
251
521,11
152.34
8.23
6.74
4.3
121 900
2,60go21
190.86 1o.45
224.624344
251.24-331
0.72
0.75x
0.94x
87.43-m9
89.72-111
84.33
TANS (Ours)
179.83
5.07
0.002
40 29.59
93. 2024
TANS (Ours)
50
179.83
5.07
0.002
202. -22-93322221
1.16x
93.71 0.24
Query
Retrieval
Query
Retrieval
Query
Retrieval
Figure 4: Retrieved examples for meta-test datasets. We visualize the dataset used for pretraining the retrieved model with the unseen query datasct. For more examples, see Figure 9 in Appendix.
to have up to 20 classes, yielding 140 and 10 datasets for meta- meta-training and meta-test datasets, respectively (Please see Table for detailed dataset configuration). For each dataset, we use randomly sampled 80/20% instances as a training and test set. To be specific, our 10 meta-test datasets include Colorectal Histology, Drawing, Dessert, Chinese Characters, Speed Limit Signs, Alien vs Predator, 19, Gemstones, and Dog Breeds. We strictly verified that there is no dataset-, class-, and instance-level overlap between the meta-training and the meta-test datasets, while some datasets may contain semantically similar classes.
Baseline Models We consider MobileNetV3 [26] pretrained on ImageNet as our baseline neural network. We compare our method with conventional NAS methods, such as PC-DARTS [65] and DrNAS [10], weight-sharing approaches, such as FBNet [60] and More-Foo-All [8], and data- driven meta-NAS approach, e.g. MetaD2A [31]. All these NAS baseline approaches are based on MobileNetV3 pretrained on ImageNet, except for the conventional NAS methods that are only able to generate architectures. As such conventional NAS methods start from the scratch, we train them for sufficient amount of training epochs (10 times more training steps) for fair comparison. Please see Appendix B for further detailed descriptions of the experimental setup.
Model-zoo Construction We follow the OFA search space [8], which allows us design resource- efficient architectures, and thus we obtain network candidates from the OFA space. We sample 100 networks condidates per meta-training dataset, and then train the network-dataset pairs, yielding 14,000 dataset-network pairs in our full model-zoo. We also construct smaller-sized efficient model- zoos from the full model-zoo (14,000) with our efficient algorithm described in Section 3.3. We use the full-sized model-zoo as our base model-zoo, unless otherwise we clearly mention the size of the used model-zoo. Detailed descriptions, e.g. training details and costs, are described in Appendix D.2.
4.2 Experimental Results
Meta-test Results We compare 50-epoch accuracy between TANS and the existing NAS methods on 10 novel real-world datasets. For fair comparison, we train PC-DARTS [65] and DrNAS [10] for 10 times more epochs (500), as they only generate architectures without pretrained weights, so we train them for a sufficient amount of iterations. For FBNet and OFA (weight-sharing methods) and
7

---

(Oun)
Onoe fo for-lllll ATIONN
TANG (0am)
Orcc-fan-dl (074)
98.0
Colorcctil HE500007
3
F3or Clazillsation
cotal Hizaboy
Closs
B07.5
Acuuracy 260
296.2
37.0
2
90.2
20.5
8
90.9
96.91
280
895
The Number Parattuatara
The Number Permmeters (M)
The Number of Hlaps
The humber (10
(a) Number of Paramcters
(b) FLOPPs
Method
Averaged Acc.
Method (1/20 Model-Zoo)
Averaged Acc.
FOOD
FBNer-A [60]
92.00=1 195
TANS (Ours)
95.07%
OFA [8]
92.89.00.4
MutaD2A [3]]
95.24111.4
TANS w/ Random Init.
74.89%
TANS w/ ImageNet
94.84%
TANS w/o Topol.
98 24-221
Merros Model Fulm: Warthct Macded Padi
TANS (Ours)
96.28:0.30
TANS w/ MetaDZA Arch.
95.38%
BB
(c) The Mooss Latent Space (d) Effectiveness of Topology
(c) Analysis on Arch. & Parameters
Figure 6: In-depth analysis of TANS: Performance comparison with constraints, such as (a) the number of parameters and (b) FLOPs. (c) Visualization for the cross-modal latent space using T-SNE and performance
comparison depending on the distance (d) Ablation study on topology information. (e) Analysis on the architecture and pretrained knowledge.
MetaD2A (data-driven meta-NAS), which provide the searched architectures as well as pretrained weights from ImageNet-1K, I, we fine-tune them on the meta-test query datasets for 50 epochs.
As shown in Table 1, we observe that TANS out-
sAs Viumi
(OON))
performs all baselines, with incomparably smaller search time and relatively smaller training time. Conventional NAS approaches such as PC-DARTS
Colorectal Histolosy
Food fiatinnatio
Accuracy (06)
90
94-
and DrNAS repeatedly require large search time for every dataset, and thus are inefficient for this practical setting with real-world datasets. FBNet, OFA, and MetaD2A are much more efficient than general NAS methods since they search for subnets within a given supernet, but obtain suboptimal performances on unseen real- real-world datasets as they may have largely different distributions from the dataset the supernet is trained on. In contrast, and reduced training time as fine-tunes a network pretrained on a relevant dataset. In Figure 5, we show the test performance curves and observe that TANS often starts with higher starting point, and converges faster to higher accuracy.
80
5 Epochs
Epochs
Figure 5: Meta-test Accuracy Curves
method achieves almost zero cost in search time,
In Figure 4, we show example images from the query and training datasets that the retrieved models are pretrained on. In most cases, our method matches semantically similar datasets to the query dataset. Even for the semantically-dissmiiia cases (right column), for which our model-zoo does not contain models pretrained on datasets similar to the query, our models still outperform all other base NAS models. As such, our model effectively retrieves not only relevatl models, but also potentially best-fitted models even trained with dissimilar datasets, for the given query datasets. We provide detailed descriptions for all query and retrieval pairs in Figure 9 of Appendix.
We also compare with commercially available AutoMI. platforms, such as Microsoft Azure Custom Vision [1] and Google Vision Edge [2]. For this experiment, we evaluate on randomly chosen five datasets (out of ten), due to excessive training costs required on AutoML platforms. As shown in Figure 2. our method outperforms all commercial NAS/AutoML methods, with significantly smaller total time cost. We provide more details and additional experiments, such as including real-world architectures, in Appendix E.
Analysis of the Architecture & Parameters To see that our method effective in retrieving networks with both optimal architecture and relevant parameters, We first report the results of base models that only search for the optimal architectures. Then we provide the results of the network retrieved using variant of our method which does not use the topology (architecture) embedding, and only uses the functional embedding of (Tans w/o Topol). As shown in Figure (d), TANS w/o Topol outperforms base NAS methods (except for MetaD2A) without considering the architectures, which shows that the performance improvement mostly comes from the knowledge transfer from the weights of the most relevant pretrained networks. However, the full TANS obtains about 1% higher performance over TANS w/o Topol., which shows the importance
conduct several ablation studies.
8

---

UEF-N Dataset
Performance of Thhe Same Pair Retrieval
Honey Boe Dataset
Model
R@1
R@5
R@
Mean
Median
Gemstone Datoset-
Random
2.14
2,86
8,57
69.04
70.0
Fast Dataset-
1 Largest Parameter
7.14
10.00
51.85
52.0
vs Presator
Chinase Characters
TANS Cosine
9.29
12.86
22.14
46.02
38.0
COVID-19 Dataset
TA NS Hard Neg.
72.14
84.29
88.57
4.86
1.0
Troffic Signs bataset-
TANS + Contrastive
80.71
96.43
99.29
1,90
1.0
Coloraciral Hisinoay
TANS w/o Func. Emb.
5.00
11,43
18.57
63.20
63.0
Performance Gain (%)
TANS w/o Predictor
80.00
96.43
97.86
2.23
1.0
(a) Effectiveness of Performance Predictor
(b) Analysis of Retrieval Performance
Method
Food
MSE on Metariest Datasets Genstones
Dogs
A. P.P.
COVID~19
Method
Food
Grasstones
MSE o
Meta Datasets Doga
A. v.
COVID-19
Prodictor wlogg 2
0.0178
0,0782
0.0194
0.01R5
0.0418
Rasdom Estanation
a.1619
0.3081
0.154M
0.2009
0L2928
Prodictor w/o
0.0188
0,0323
0.0016
0,0652
0.0928
1/100 Model-Zao
0,088
0.0069
0.0034
0,0077
0.0241
Predictor (Ours)
0.0036
0,0338
0.0013
0,0028
0.0233
Top 10 Retrievals
0.0036
0,0338
0,0013
0,0028
0.0233
(c) Analysis of Performance Predictor
(d) Fffectiveness of Performance Predictor
Figure 7: In-depth analysis of TANNS (2): (a) The effectiveness of meta- pecea-performmnce predictor. (b) Ablation study on retrieval performance. Additional (c) analysis and (d) the effectiveness of our performance predictor.
of the architecture and the effectiveness of our architecture embedding. In Figure 6 (e), we further experiment with a version of our method that initializes the retrieved networks with both random weights and ImageNet pre-trained weights, using 1/20 sized model-zoo (700). We observe that they achieve lower accuracy over TANS on 10 datasets, which again shows the importance of retrieving relevant tasks' knowledge. We also construct the model-zoo by training on the architectures found by an existing NAS method (MetaD2A), and see that it further improves the performance of TANS.
Contraints-conditione Retrieval TANS can retrieve models with a given dataset and additional constraints, such as the number of the parameters or the computations (in FLOP). This is practically important since we may need a network with less memory and computation overhead depending on the hardware device. This can be done by filtering networks that satisfy the given conditions among the candidate networks retrieved. For this experiment, we compare against OFA that performs the same constrained search, as other baselines do not straightforwardly handle this scenario. As shown in Figure 6 (a) and (b), we observe that the network retrieved with TANS consistently outperforms the network searched with OFA under varying parameters and computations constraints. Such constrained search is straightforward with our method since our retrieval- based method allows us to search from the database consisting of networks with varying architectures and sizes.
Analysis of the Cross-Modal Retrieval Space We further examine the learned cross-modal space. We first visualize the meta-learnd latent space in Figure 6 (c) with randomly sampled 1,400 models among 14,000 models in the model-zoo. We observe that the network whose embeddings are the closest to the query dataset achieves higher performance on it, compared to networks embedded the farthest from it. For example, the accuracy of the closet network point for UCF-AI 98.94% while the farthest network only achieves 91.53%.
Table 2: Latent Distance and Performance
We also show Spearman correlation scores on 5 meta-test datasets in Table 2. Measuring correlation with the distances is not directly compatible with our contrastive loss, since the negative examples (models that achieve low performance the target dataset) are pushed away from the query point, without a meaningful ranking between the negative instances. To obtain a latent space where the negative examples are also well-ranked, we should replace the contrastive loss with a ranking loss instead, but this will not be very meaningful. Hence, to effectively validate our latent space with correlation metric, we rather select clusters, such that 50 models around the query point and another 50 models around the farthest points, thus using a total of 100 models to report the correlation scores. In the Table 2, we show the correlation scores of these 100 models on the five unseen datasets. For Food dataset (reported as "hard" dataset in Table 1), the correlation scores are shown to be high. On the other hand, for Colorectal Histology dataset (reported as "easy" dataset), the correlation scores are low as any model can obtain good performance on it, which makes the performance gap across models small. In sum, as the task (dataset) becomes more difficult, we can observe higher correlation in the latent space between the distance and the rank.
Spearman's 's Correlation on Meta-test Datasets
Food Drawing
Chinese
A. vs P. Colorectal
0.752
0.583
0.322
0.214
0.213
9

---

Meta- Meeaa-perorranne Predictor The role of the proposed performance predictor is not only guiding the model and query encoder learn the better latent space but also selecting the best model among retrieval candidates. To verify its effectiveness, we measure the performance gap between the top- retrieved model w/o the predictor and the model with the highest scores selected using the predictor among retrieved candidates. As shown in Figure 7 (a), there are 1.5%p 8%p performance gains on the meta-test datasets. The top-1 retrieved model from the model zoo with TANS may not be always optimal for unseen dataset, and our performance predictor remedies this issue by selecting the best-fitted model based on its estimation. We also examine ablation study for our performance predictor. Please note that we do not use ranking loss which does not rank the negative examples. Thus we use Mean Squared Error (MSE) scores. We retrieve the top 10 most relevant models for an unseen query datasets and then compute the MSE between the estimated scores and the actual ground truth accuracies. As shown in Figure (c), we observe that removing either query or model emheddings degrades performance compared to the predictor taking both embeddings. It is natural that, with only model or query information, it is difficult to correctly estimate the accuracy since the predictor fails recognize what where to learn. Also, we report the MSE between the predicted performance using the predictor and the ground truth performance of each model for the entire set of pretrained models from a smaller model zoo in Figure (d). Although the performance predictor achieves slightly higher MSE scores for this experiment compared to the MSE obtained on the top-10 retrieved models (which are the most important), the MSE scores are still meaningfully low, which implies that our performance model successfully works even toward the entire model-zoo.
Retrieval Performance We also verify whether our model successfully retrieves the same paired models when the correspondent meta-train datasets are given (we use unseen instances that are not used when training the encoders.) For the evaluation metric, we use recall at & (R ((kk) which indicates the percentage of the correct models retrieved among the top-k candidates for the unseen query instances, where & is set to 1, 5, and 10. Also, we report the mean and median ranking of the correct network among all networks for the unseen query. In Figure (b), the largest parameter selection strategy shows poor performances on the retrieval problem, suggesting that simply selecting the largest network is not a suitable choice for real-world tasks. In addition, compared to cosine similarity learning, the proposed meta-contrastive learning allows the model to learn significantly improved discriminative latent space for cross-modal retrieval. Moreover, without our performance predictor, we observe that TANS achieves slightly lower performance, while it is significantly degenerated when training without functional embeddings.
Analysis of Model-Zoo Construction Unlike most existing NAS methods, which repeatedly search the optimal architec- tures per dataset from their search spaces, TANS do not need to perform such repetitive search procedure, once the model- zoo is built beforehand. We are able to adaptively retrieve the relevant pretrained models for any number of datasets from our model-zoo, with almost zero search costs. Formally,
Method
Aschitecture Search Cost
Retrieved Dataset
Pre training Cost
Conventional NAS MesaD2AA
OIN)
ONN)
((1)
(KN)
TANS (Ours)
O(1)
O(1)
Accuracy (%)
95.0
Ours
95.5
Random
TANS reduces the time complexities of both search cost and
95.0
pre- veettraining cost from O(N) to O(1), where N is the num-
94.5
ber of datasets, as shown in Figure (Top). Furthermore, model zoo constructed using our efficient construction algo- rithm, introdeued in Section 3.3, yields models with higher performance on average, compared to the random sampling strategy when the size of the model-zoo the same as shown in Figure (Bottom).
1000
2500
5000
Model-Zao Size
Figure 8: Scarch & Pretraining Costs (Top), Model-Zoo Analysis (Bottom)
5 Conclusion
We propose novel Adsp-Ade Neural Network Search framework (TANS), that instantly retrieves a relevant pre-trained network for an unseen dataset, based on the cross-modal retrieval of dataset- network pairs. We train this retrieval model via amortized meta-learning of the cross-modal latent space with contrastive learning, to maximize the similarity between the positive detaset-netwon pairs, and minimize the similarity between the negative pairs. We train our framework on a model zoo consisting of diverse pretrained networks, and validate its performance on ten unseen datastes. The results show that the proposed TANS rapidly searches and trains a well-fitted network for unseen datasets with almost no architecture search cost and significantly fewer fine-tuning steps to reach the target performance, compared to other NAS methods. We discuss the limitation and the societal impact of our work in Appendix F.
10

---

Acknowledgement This work was supported by AITRICS and by Institute of Information & communica- tions Technology Planning & Evaluation (IITP) grant funded by the Korea (SST) (No. 20..001--0-00075 Artificial Intelligence Graduate School Program(KAIST)
References
[1] Microsoft azure custom vision. http::::/ww...uuutcm..........a ai/.
[2] Google automl vision edge. https://cloud. com.vision/aatooniioo.ooooo htwpp///aa..aoooooooooooooooooo.
[3] Alekh Agarwal, Alina Beygelzimer, Miroslav Dudík, John Langford, and Hanna Wallach, reductions approach to fair classification, 2018.
[4] Irwan Bello. Lambdanetworks: Modeling long-range interactions without attention. In International Confer- ence Learning Repreventations, 2021. URL pentps::/opprreeee http:p/awrrreee...oooo///rrr//aa net/foorm/id=xxtEn ggllb.
[5] Francesco Biscani and Dario Izzo. A parallel global multiobjectivee framework for optimization: pagmo. Journal of Open Source Software, 2020.
[6] Andrew Brock, Theodore Lim, James M. Ritchic, and Nick Weston. SMASH: one-shot model architecture search through hypemetworks. 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings, 2018
[7] Andrew Brock, Soham De, Samuel Smith, Karen Simonyan. performefrrmnce large-scale image recognition without normalization. arXiv preprint arXiv:2102.0611 2021.
[8] Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once for all: Train one network and specialize it for efficient deployment. In International Conference on Learning Representations, 2020.
[9] Wei-Cheng Chang, Felix X. Yu, Yin-Wen Chang, Yiming Yang, and Sanjiv Kumar. Pre-training tasks for saahedding-baseee large-scale retrieval. In 8th International Conference on Learning Representations, ICI. 2020, Addis Ababo, Ethiopia, April 26-30, 2020, 2020.
[10] Xiangning Chen, Ruochen Wang, Minhao Cheng, Xiaocheng Tang, and Cho-Jui Hsieh. Dr(nas): Dirichlet neural architecture search. International Conference on Learning Representations. 2021.
[11] Yukang Chen, Gaofeng Meng, Qian Zhang. Shiming Xiang, Chang Huang, Lisen Mu, and Xinggang Wang. RENAS:
reinforced evolutionary neural architecture search. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 4787-4796. Computer Vision Foundation/ / IEEE, 2019.
[12] Andrew Cotter, Maya Gupta, Heinrich Jiang, Nathan Srebro, Karthik Sridharan, Serena Wang, Blake Woodworth, and Seungil You. Training well-generalizing classifiers for fairness metries and other data- dependent constraints, 2018
[13] Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, Fei-Fei. Imagenet: large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Icce, 2009.
[14] A Francis Saviour Devaraj, G Murugaboopathi, Mohamed Elhoseny, K Shankar, Kyungbok Min, Hyeon- joon Moon, and Gyanendra Prasad Joshi. An efficient framework for secure image archival and retrieval system using multiple secret share creation scheme. IEFE Access, 8:444300-144320, 2020.
[15] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL /ttpps://opereviiiii net/forum? id-YicbFaNTIy.
[16] Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. Proceedings of the 3rd Innovations Theoretical Computer Science Conference, ITCS '12 page 214-226, New York, NY, USA, 2012. Association for Computing Machinery. ISBN 9111.50311151. doi: 10.1 145/2090026.22902555 URL https: h/ap:::://ds /doi. org/10 100 1145/2090236, 2090000000222202225....
[17] Thomas Elsken, Benedikt Staffler, Jan Hendrik Metzen, and Frank Hutter. Meta-learning of neural architectures for few-shot learning. In 2020 JEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 12362 12372. IEEE, 2020.
11

---

[18] Martin Engilberge, Louis Chevallier. Patrick Pérez, and Matthieu Cord. Finding beans in burgers: Deep semantic-visual embedding with localization. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[19] Fartash Faghri, David J. Fleet, Jamie Kiros, and Sanja Fidler. VSF++: improving visual-semantic embeddings with hard negatives. In British Machine Vision Conference 2018, BMVC, 2018
[20] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning (ICML.) 2017.
[21] Albert Gordo, Jon Almazán, Jerome Revaud, and Diane Larlus. End-to-end learning of deep visual representations for image retrieval. International Journal of Computer Vision, 124(2)2377225 September 2017.
[22] Moritz Hardt, Eric Price, and Nathan Srebro. Equality of opportunity in supervised learning. In NIPS, 2016.
[23] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2016.
[24] Kurt Horiik Approximation capabilities of multilayer feedforward networks. Neural Networks, 4(2): 251-257, 1991.
[25] Kurt Homik, Maxwell B. Stinchcombe, and Halbert White. Multilayer feedforward networks universal approximators. Neural Networks, 2(5):399-333 366, 1989.
[26] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chiell Chen, Bo Chen, Mingxing Tan, Weijun Wang. Yukun Zhu, Ruoming Pang. Vijay Vasudevan, al. Searching for mobilenetv3. In Proceedings of the IEEE International Conference on Computer Vision, pages 1314-1324, 2019.
[27] Andrew G. Howard, Menglong Zhu. Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 174.0066... 2017.
[28] Forrest N landola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and< 0.5 mb model size. arXiv preprint arXiv:1602.07360. 2016.
[29] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25:1097-1105, 2012.
[30] Hac Boom Lec, Haycon Lec, Donghyun Na, Sachoon Kim, Minscop Park, Eunho Yang, and Sung Ju Hwang. Learning to balance: Bayesian meta-learning for imbalanced and out-of-distribution tasks. In International Conference on Learning Representations (ICLR), 2020.
[31] Hayeon Lee, Eunyoung Hyung. and Sung Ju Hwang. Rapid neural architecture search by learning to generate graphs from datasets. In International Conference Learning Representations, 2021.
[32] Kuang-Hus Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He. Stacked cross attention for image-text matching. In Computer Vision ECCV 2018 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part IV. volume 11208 of Lecture Notes Computer Science, pages 212-228. Springer, 2018
[33] Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differ- entiable convex optimization. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2019.
[34] Dongze Lian, Yin Zheng, Yintao Xu, Yanxiong Lu, Leyu Lin, Peilin Zhao, Junzhou Huang, and Shenghua Gao. Towards fast adaptation of neural architectures with meta learning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, 2020.
[35] Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. In In International Conference on Learning Representations (ICLR), 2019.
[36] Christos Louizos, Kevin Swersky, Yujia Li, Max Welling, and Richard Zemel. The variational fair autoencoder, 2017.
[37] Zhichao Lu, Kalyanmoy Deb, Erik Goodman, Wolfgang Banzhaf, and Vishnu Naresh Boddeti. Nsganetv2: Evolutionary multi-objective sarooate--ssisted neural architecture search. In European Conference on Computer Vision, pages 35 51. Springer, 2020.
12

---

[38] Renqian Luo. Fei Tian, Tao Qin, Enhong Chen, and Tie-Yan Liu. Neural architecture optimization. In Advances in neural information processing systems (NeurIPS). 2018.
[39] Ningning Ma, Xiangyu Zhang, Tai-Tao Zheng, and Jian Sun. Shufilenet v2: Practical guidelines for efficient emn architecture design. In Proceedings of the European conference on computer vision (ECCV), pages 116-131, 2018.
[40] Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint ArXiv:1803.000 wew::8803300000 029999 2018.
[41] Krzysztof Nowak, Marcus Märtens, and Dario Izzo. Empirical Performance the Approximation of the Least Hypervolume Contributor. In Thomas Bartz-Beiclstein, Jürgen Branke, Bogdan Filipić, and Jim Smith, editors, Parallel Prohlem Solving from Nature PPSN XIII. Lecture Notes in Computer Science. pages 662-671, Cham, 2014. Springer International Publishing. ISBN 978-3-319-10762-2. doi: 10.1007/978-3 3-1199998---9---11 1099-0776-22666
[42] Hieu Pham, Melody Y. Guan, Barret Zoph, Quoe V. Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. In Proceedings of the 35th International Conference on Machine Learning, ICML. 2018, Stockholmsmāsssn, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 4092-4101. PMLR, 2018.
[43] Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Le. Regularized evolution for image classifier architecture search. In Proceedings of the aaai conference artificial intelligence, volume 33, pages 4780-4788 4089, 2019.
[44] Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po- Yao Huang, Zhihui Li, Xiaojiang Chen, and Xin Wang. comprehensive survey of neural architecture search: Challenges and solutions, 2020.
[45] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang Chich Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510 4520, 2018
[46] Albert Shaw. Wei Wei, Weiyang Liu, Le Song, and Bo Dai. Meta architecture search. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019. December 8-14, 2019, Vancouver. BC, Canada, pages 11225-11235. 2019.
[47] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. In 3rd International Conference Learning Representations, ICIR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
[48] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Advances in neural information processing systems (NIPS), 2017.
[49] Aravind Srinivas, Tsung-Y Yi Lin, Niki Parmar, Jonathon Shlens, Pieter Abbeel, and Ashish Vaswani. Bottleneck transformers for visual recogmition. arXiv preprint arXiv:2101. 2021.
[50] Christian Szegedy, Wei Liu, Yangging Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9, 2015.
[51] Mingxing Tan and Quoc V Lee. Efficientnet: Rethinking model scaling for convolutional neural networks. arXiv preprint arXiv:1905.1 1946. 2019.
[52] Mingxing Tan and Quoe V Le. Efficienteet22: Smaller models and faster training. 2021.
[53] Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoe V Le. Mnasnet: Platform-awar neural architecture scarch for mobile. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 2820-2828. 2019.
[54] Yehui Tang, Yunhe Wang, Yixing Xu, Hanting Chen, Boxin Shi, Chao Xu, Chunjing Xu, Qi Tian, and Chang Xu. A suni-super assessor of neural architectures. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1810-1819, 2020.
[55] Sebastian Thrun and Lorien Pratt, editors. Learning to Leurn. Kluwer Academic Publishers, Norwell, MA, USA, 1998. ISBN 0-7923-8047-9.
[56] Oriol Vinyals, Charles Blundell, Timothy Lillicrap. Daan Wierstra, et al. Matching networks for one shot learning. In Advances in neural information processing systems (NIPS), 2016.
13

---

[57] Edward Wagstaff. Fabian Fuchs, Martin Engelcke. Ingmar Posner, and Michael A. Osborne. On the limitations of representing functions sels. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 Proceedings of Machine Learning Research, pages 6487 6494. PMLR, 2019.
[58] Sijin Wang, Ruiping Wang, Ziwei Yao, Shiguang Shan, and Xilin Chen. Cross-modat scene graph matching for relationship-ave aware image-text retrieval. In IEEE Winter Conference on Applications of Computer Vision, WACV 2020, Snowmass Village, CO, USA, March 1-5, 2020, pages 1497-1506. IEEE, 2020.
[59] Blake Woodworth, Suriya Gunasekar, Mesrob Ohannessian, and Nathan Srebro. Learning non- discriminatory predictors, 2017.
[60] Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wa, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 10742, 2019.
[61] Bichen Wu. Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. FBNet: Hardware- Aware Efficient ConwNet Design via Differentiahle Neural Architecture Scarch. arXiv:/812.00343 [cs], May 2019. URL Utt.att::/arriiv.rrr arXiv: 1812.03443.
Aba/1812. 03443.
[62] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transforma- tions for deep neural networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1492-1500, 2017.
[63] Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwikj. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In International Conference on Learning Representations. 2021.
[64] Keyulu Xu, Weihua Hu. Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks?
In International Conference on Learning Representations, 2019.
[65] Yuhui Xu, Lingxi Xie, Xiaopeng Zhang. Xin Chen, Guo-Jun Qi. Qi Tian, and Hongkai Xiong. Pa-darts: Partial channel connections for memory-efficiien architecture search. In International Conference on Learning Representations (ICLR), 2020.
[66] Chenggang Yan, Biao Gong, Yuxuan Wei, and Yue Gao. Deep multi-view enhancement hashing for image retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
[67] Shen Yan, Yu Zheng, Wei Ao, Xiao Zeng, and Mi Zhang. Does unsupervised architecture representation learning help neural architecture search? Advances in Neural Information Processing Systems, 33, 2020.
[68] Mikhail Yurochkin, Amanda Bower, and Yuekai Sun. Training individually fair ml models with sensitive subspace rohustness. In International Conference on Learning Representations, 2020. URL https: / /oppnrevview.oe www.e/www.eeeeeeeeeeeeeeeeeeeeee wwt/fformmiiiee wi-.rrniddiiidiiffiie
[69] Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Ruslan R Salakhutdinov, and Alexander J Smola. Deep sets. In Advances in Neural Information Processing Systems (NIPS), 2017.
[70] Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. Leanning fair representations. In Sanjoy Dasgupta and David McAllester, editors, Proceedings of the 30th International Conference on Machine Learning. volume 28 of Proceedings of Machine Learning Research, pages 325-333. 333, Atlanta, Georgia, USA, 17-19 Jun 2013. PMLR. URL https: https://proceeding mlr, aar.pea pwwss/eeseeee v28/ 202///eme11... html.
[71] Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell, Mitigating unwanted biases with adversarial learning. 2018
[72] Muhan Zhang, Shali Jiang, Zhicheng Cui, Roman Garnett, and Yixin Chen. D-vae: A variational autoencoder for directed acyclic graphs. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
[73] Weinan Zhang, Xiangyu Zhao, Li Zhao, Dawei Yin, Grace Hui Yang. and Alex Beutel. Deep reinforcement learning for information retrieval: Fundamentals and advances. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2468 2471, 2020.
14

---

[74] Liangli Zhen, Peng Hu, Xu Wang, and Dezhong Peng. Deep supervised cross-modal retrieval. In IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 10394-10403. Computer Vision Foundation IEEE, 2019.
[75] Dongzhan Zhou, Xinchi Zhou, Wenwei Zhang, Chen Change Loy, Shuai Yi, Xuesen Zhang, and Wanli Ouyang. Econas: Finding proxies economical neural architecture search. Proceedings of IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 1339--1144, 2020.
[76] Barret Zoph and Quoe Conference on Learning Representations (ICLR), 2017.
Neural architecture search with reinforcement learning. In International
[77] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and puttern recognition, pages 8697-8710. 10, 2018.
15


Reviews:
[{"review_id": "x-AiHb67Sgn", "reviewer": "Reviewer_ZAGc", "summary": "This paper introduces an interesting problem of Neural Network Search (NNS), aiming at searching for the optimal pretrained network for a novel dataset from a model zoo. The authors propose a framework TANS to tackle this problem. The proposed method constructs a model encoder and a dataset encoder to calculate the embedding of model and query set respectively. By applying the contrastive loss to maximize the similarity between a dataset and a network that obtains high performance on it, the framework is trained jointly with a surrogate performance predictor. At testing time, the framework is able to retrieve the well-fit architectures from the model zoo given an unseen dataset.\n\nThe main contribution of this paper is that it introduces a novel problem Neural Network Search and also present a decent method to tackle the problem.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. The paper is well written and easy to follow.\n2. The introduced problem NNS is novel to me. And the problem is practical and does exist in the real deployment scenario. \n3. The proposed method is technically sound and the authors also have provided sufficient details and ablation study to validate the effectiveness.\n\nCons:\n1. I'm confused about the proposed functional embeddings. I'm not sure why introducing it since we've already have a topology embeddings. Does it play a significant role in the proposed framework. In table 2, the author has conducted an ablation w.r.t. the topol embedding but fail to conduct a similar one on functional embedding. I would like to see this ablation.\n\n2. I would like to see an ablation study on the surrogate model. Does the predicted performance correlate with the GT acc well?\n\n3. In Fig. 6c, the authors has visualized the the cross-model latent space to demonstrate the claim that \"the network whose embeddings are the closest to the query dataset achieves higher performance on it\" (L358). However, I think the best way to validate this point is to calculate the correlation between the latent distance and the real performance to see if they correlate well.\n\n4. The authors also show the effect of model-zoo size on accuracy in Fig. 6d. I'm wondering the concrete data reduction method here? Do the author simply shrink the data size of each dataset or just simply remove some datasets? These two methods might have different impact on the performance. I would like the authors to clarify this and it would be better to see the results of both two methods.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The author has provided in the supp.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.0"}, "extra_text": {}}, {"review_id": "VMPitNUWrmN", "reviewer": "Reviewer_orau", "summary": "This paper proposes a new Task-Adaptive Neural network search framework (TANS). Such a task needs to retrieve a good network in well pre-trained dataset-network sets. In order to obtain a task-adaptive neural network, the amortized meta-learning method is employed. Meanwhile, this meta-learning framework is trained with contrastive learning to maximize the similarity between the positive dataset-network pairs and minimize the similarity between the negative pairs. The experiments show that TANS can rapidly retrieve a well-fitted network for unseen datasets. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n(1)  The authors propose a new task, which retrieves a good network in well pre-trained dataset-network sets. Such a task is promising and novel. \n(2)  A meta-learning framework is trained with contrastive learning to maximize the similarity between the positive dataset-network pairs and minimize the similarity between the negative pairs.\n(3)  The experiments in ten real-world datasets show the effectiveness of TANS.\n\nCons:\nI understand the novel task of this manuscript. I have some questions as follows. I want to make sure again, is it the first work to propose architecture retrieval? I think this task is very promising because we can hardly search for a suitable network given any task. The second question is if the dataset is very large, e.g., ImageNet, how to sample images in this dataset? What kind of image is the suitable image that represents this dataset? Whether the simple random sampling method is applicable? Another question is whether such a method only tackles the image classification task. What if object detection or other regression tasks? \n\nTo sum up, this manuscript introduces a new task, which is valuable in real-world architecture retrieval. The writing is clear and easy to understand. However, there are still some unclear descriptions and questions. Some tables and figures (e.g., Figure 7, Table 3) are also too small to see results clearly.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There are still some unclear descriptions in this manuscript. The specific contents are shown in the Main review. Judging from the paper, there is currently no direct negative impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "NriwDbtmCKG", "reviewer": "Reviewer_6K2a", "summary": "This paper proposes and tackles the following task: given an unseen dataset, selecting the optimal pre-trained neural network from a model zoo of networks which will obtain the best fine-tuned performance on the new dataset. To solve this problem they propose a novel contrastive learning method to learn a shared embedding space for the models and datasets which minimizes the distance of the dataset embeddings and model embeddings of models that can be well adapted to the datasets. They propose encoding the properties of a pretrained model using a combination of a topological and functional encoding and demonstrate the effectiveness of this encoding scheme. They also propose a novel method for efficient model-zoo construction and build a dataset and train their algorithm on a collection of Kaggle datasets. They demonstrate that they can achieve very high performance on unseen datasets with few training steps.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper tackles a novel problem of building a model zoo of pretrained networks and retrieving the optimal base model for fine tuning on an unseen dataset. This is a valuable direction of research and has strong significance for practical machine learning since picking an existing trained model from a model zoo and fine tuning it for your task is an extremely common method to quickly train an effective model on a new dataset. The methodology also includes a novel method of characterizing the model using a combination of topological and function encoding.\n\nUnfortunately, I currently can't recommend acceptance for this paper. I believe that this work requires a stronger baseline to demonstrate the generalization of the dataset embedding.\n\nAdditional results would help ablate the effectiveness of the dataset embedding and the quality of the models and pretrained weights from the model zoo. While table 3 does ablate the choices made for the embedding, it's not entirely clear how this generalizes to unseen datasets from just those experiments. A strong baseline  that would significantly improve the quality of the paper would be for each unseen test dataset X, sample 5-10 random other test datasets, and retrieve and train the optimal models for those datasets on dataset X. This would help ablate the effectiveness of choosing random strong models from the model zoo with the effectiveness of the model embedding. In table 3, the author's do show some ablations of the different retrieval strategies which helps demonstrate their choices for the retrieval method. However, this could be strengthened by making sure if the models selected, but not trained on the specified data still performed adequate on the specified dataset.\n\nThe paper would also be much improved with more analysis of the efficiency of the methodology especially related to how much compute is used to generate the Model zoo. This is especially important to demonstrate the efficacy of their novel efficient model zoo construction method.\n\nThe results that they demonstrate on the unseen dataset are impressive, but difficult to directly compare. None of the existing models were pre-trained on the same training kaggle datasets and the model zoo was trained on both those and the imagenet dataset through the OFA initialization. It would also be helpful to compare the models if the flops of each model was included in Table 1. Something that help demonstrate their algorithm would be testing their ability to transfer to common unseen datasets like cifar-10/cifar-100/imagenet.\n\nThe paper is generally clear and easy to follow, but some of the details could be clearer. How are the positive and negative pairs selected? How exactly is the meta performance surrogate model used? How were models trained to create the model zoo? It seems both a 14,000 model model zoo and an efficiently constructed model zoo were created? It's not entirely clear which is used in which experiments. Was the model zoo trained off OFA in the meta-fashion or each architecture separately?\n\n==================================================================================================\n\nPost-Rebuttal Update:\nI would like to thank the authors for addressing many of my concerns and especially including the additional baselines and experiments. I believe that the baseline exploring the capabilities of the models in just the model zoo significantly improves the papers and I think the paper would benefit from the inclusion of the full baseline. In addition, the inclusion of results with training the competitors on the same kaggle meta-training addressed my concerns on fairness with regards to pretraining on the kaggle datasets. The authors have definitely demonstrates the effectiveness of your task-adaptive retrieval mechanism and addressed my main concern. Given that and the inclusion of more discussion on the comparative pretraining costs, and additional experiments demonstrating the robustness of the method on other datasets, I would recommend acceptance for this paper and raise my score to a 7.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "They discuss it in the appendix. They mostly address the societal impacts, but maybe could include some additional discussion of the possible impact of biased initializations.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "J_V3IFMRkQN", "reviewer": "Reviewer_5Ssa", "summary": "The paper is about retrieving a pre-trained network from a model zoo in a computationally effective manner. The authors proposed a network topology and parameter embedding that can be used to retrieve networks that perform well on a task represented by a dataset after fine-tuning.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I find the paper to be clearly written, interesting to read, the topic to be relevant and the contribution to be novel and significant. The paper contains a significant methodological contribution, defining a new task consisting of multiple datasets. The code is provided for reproducibility. A quick inspection shows that the code is well organised and contains all the ingredients to download and prepare the data.  I recommend accept and provide minor comments below.\n\n- line 125, it took me some time to understand why neural network is denoted $M^\\tau$. I realized later on that $M$ refers to model. Indeed, the proposed method could be generalized to arbitrary models, why not make it clear in line 125?\n- it is unclear to me why authors chose a title that is different from the name of their core contribution. Why not have a title \"TANS: Task-Adaptive Network Search.\" or \"TANS: Task-Adaptive Network Search with Meta-Contrastive Learning.\" ?\n- what is the total compute capacity that is needed to reproduce the results?\n- datasets used in the study are public, but it would be a good idea to provide them from one location, because they may disappear with time. Have you thought about it?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "No concerns", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "1 hour"}, "extra_text": {}}]

Author rebuttal:

Meta-review: A strong paper on a subject that many people in the community are likely to find interesting.  Multiple reviewers agreed that this submission could be a Top-50 paper. But it was felt that the work could provide a better analysis, and possibly further experiments on the model zoo creation and corresponding properties. A healthy discussion occurred during the author response and discussion phase. New experiments and further explanations from the authors motivated one of the reviewers to upgrade their score and assessment of the paper to an accept, leading to all reviewers recommending acceptance. More discussions of comparative pretraining costs and additional experiments provided by the authors during the rebuttal helped to demonstrate the robustness of the method on other datasets.

The work addresses a subject that the reviewers felt is interesting and important, proposing a network topology and parameter embedding that can be used to retrieve networks that perform well on a task represented by a dataset after fine-tuning. The paper proposes an amortized meta-learning framework to learn a cross-modal latent space using a contrastive loss. Their goal is to maximize the similarity between a dataset and a high-performing network on it, and minimize the similarity between irrelevant dataset-network pairs. 

Given the good reception by the reviewers after the author response and the good level of discussion about this work, the AC recommends a Spotlight.
Decision: Accept (Spotlight)