Neural Architecture Retrieval
Authors: Xiaohuan Pei
Yanxi Li
Minjing Dong
Chang Xu
Venue: ICLR 2024 poster (2024)
Keywords: Information Retrieval
Vector Database
Neural Architecture Search
Primary area: general machine learning (i.e., none of the above)

Abstract: With the increasing number of new neural architecture designs and substantial existing neural architectures, it becomes difficult for the researchers to situate their contributions compared with existing neural architectures or establish the connections between their designs and other relevant ones. To discover similar neural architectures in an efficient and automatic manner, we define a new problem Neural Architecture Retrieval which retrieves a set of existing neural architectures which have similar designs to the query neural architecture. Existing graph pre-training strategies cannot address the computational graph in neural architectures due to the graph size and motifs. To fulfill this potential, we propose to divide the graph into motifs which are used to rebuild the macro graph to tackle these issues, and introduce multi-level contrastive learning to achieve accurate graph representation learning. Extensive evaluations on both human-designed and synthesized neural architectures demonstrate the superiority of our algorithm. Such a dataset which contains 12k real-world network architectures, as well as their embedding, is built for neural architecture retrieval.

Published as a conference paper at ICLR 2024
NEURAL ARCHITECTURE RETRIEVAL
Xiaohuan Pei
Yanxi Li
Department of Computer Science The University of Sydney, Australia apei831888 8@uni sydney. edu. au
Department of Computer Science The University of Sydney, Australia yan110722@uni sydney edu au
Minjing Dong
Chang Xu
Department of Computer Science City University of Hong Kong, China min cityu. hk
Department of Computer Science The University of Sydney, Australia xuê sddney. edu. au
ABSTRACT
With the increasing number of new neural architecture designs and substantial ex- isting neural architectures, it becomes difficult for the researchers to situate their contributions compared with existing neural architectures or establish the connec- tions between their designs and other relevant ones. To discover similar neural ar- chitectures in an efficient and automatic manner, we define a new problem Neural Architecture Retrieval which retrieves set of existing neural architectures which have similar designs to the query neural architecture. Existing graph pre-training strategies cannot address the computational graph in neural architectures due the graph size and motifs. To fulfill this potential, we propose to divide the graph into motifs which are used to rebuild the macro graph to tackle these issues, and introduce multi-level contrastive learning to achieve accurate graph representation learning. Extensive evaluations both human-designed and synthesized neural architectures demonstrate the superiority of our algorithm. Such a dataset which contains 12k real-world network architectures, as well their embedding, is built for neural architecture retrieval. Our project is available at www.teryypei.com/mn retrieval
INTRODUCTION
Deep Neural Networks (DNNs) have proven their dominance in the field of computer vision tasks,
including image classification (He et al., 2016: Zagoruyko & Komodakis 2016 Liu et al.) 2021
Dong et al. 2021 Wang al. 20188) object detection (Tian et al. 2019 Carion al. [2220] Tan et al. 2020). etc. Architecture designs play an important role in this success since each innovative and advanced architecture design always lead to a boost of network performance
various tasks.
For example, the ResNet family is introduced to make it possible to train extremely deep networks via residual connections (He al.) 2016), and the Vision Transformer (ViT) family proposes to split the images into patches and utilize multi-head self-attentions for feature extraction, which shows superiority over Convolutional Neural Networks (CNNs) in some tasks (Dosovitskiy et al., 2020). With the increasing efforts in architecture designs, an enormous number of neural architectures have been introduced and open-sourced, which are available on various platforms]
Information Retrieval (IR) plays an important role in knowledge management due to its ability to store, retrieve, and maintain information. With access to such a large number of neural architec- tures various tasks, it is natural to look for retrieval system which maintains and utilizes these valuable neural architecture designs. Given a query, the users can find useful information, such as relevant architecture designs, within massive data resources and rank the results by relevance in low latency. To the best of our knowledge, this is the first work to setup the retrieval system for neural architectures. We define this new problem as Neural Architecture Retrieval (NAR). which returns a set of similar neural architectures given a query neural architecture. NAR aims at maintaining both
https:/huggiingfccc..o co/, https//pptoocchorgghubb

---

Published as a conference paper at ICLR 2024
existing and potential neural architecture designs, and achieving efficient and accurate retrieval, with which the researchers can easily identify the uniqueness of a new architecture design or check the existing modifications specific neural architecture.
Embedding models which jointly embed documents and queries in the same embedding space for similarity measurement widely adopted in retrieval algorithms (Huang et al.) 2020 Chang et al. 2020). With accurate embedding of all candidate documents, the results can be efficienlly computed via nearest neighbor search algorithms in the embedding space, At first glance of NAR, is easy to come up with the graph pre-training strategies via Graph Neural Networks (GNNs) since the computational graphs of networks can be easily derived represent the neural architectures. However, existing graph pre-training strategies cannot achieve effective learning of graph embed- ding directly due to the characteristic of neural architectures. One concern lies in the dramatically varied graph sizes among different neural architectures, such as LeNet-5 versus ViT-L. Another concern lies in the motifs in neural architectures. Besides the entire graph, the motifs in neural architectures are another essential component to be considered in similarity measurement. For ex- ample, ResNet-50 and ResNet-101 are different in graph-level, however, their block designs are exactly the same. Thus, it is difficult for existing algorithms learn graph embedding effectively.
In this work, we introduce a new framework to learn accurate graph embedding especially for neural architectures to tackle NAR problem. To address the graph size and motifs issues, we propose to split the graph into several motifs and rebuild the graph by treating motifs as nodes in a macro graph to reduce the graph size well take motifs into consideration. Specifically, we introduce new motif's sampling strategy which encodes the neighbours of nodes to expand the receptive field of motif's in the graph convert the graph an encoded sequence, and the motif's can be derived by discovering the frequent subsequences. To achieve accurate graph embedding learning which can be easily generalized to potential unknown neural architectures, we introduce motifs- and graph- level pre-train rre-train tasks. We include both human-designed neural architectures and those from NAS search space as datasets to verify the effectiveness of proposed algorithm. For real-world neural architectures, we build dataset with 12k different architectures collected from Hugging Face and PyTorch Hub, where each architecture is associated with an embedding for relevance computation.
Our contributions can be summarized as: 1. A new problem Neural Architecture Retrieval which benefits the community of architecture designing. 2. A novel graph representation learning algo- rithm tackle the challenging NAR problem. 3. Sufficient experiments on the neural architectures from both real-world ones collected from various platforms and synthesized ones from NAS search space, and our proposed algorithm shows superiority other baselines. 4. A new dataset of 12k real-world neural architectures with their corresponding embedding.
2 RELATED WORK
Human-designed Architecture Researchers have proposed various architectures for improved performance on various tasks (Li et al.) 2022aa Dong et al. (023). GoogLeNet uses inception mod- ules for feature scaling (Szegedy et al. 2015). ResNet employs skip connections (He et al. 2016), DenseNet connects all layers within blocks (Huang et al. 2017). and SENet uses squeeze-and- excitation blocks for feature recalibration (Hu al. 2018)) ShuffleNet, GhostNet and MobileNet aim for efficiency by shuffle operations (Zhang et al. 2018)). cheap feature map generation (Han)
al. 2020 2022) and depthwise separable convolutions (Howard et al., 2017). In addition to
CNNs. transformers have been explored in both CV and NLP. BERT pre-trains deep bidirectional representations (Devlin et al.) 2018), while (Dosovitskiy et al. 2020) and (Liu et al. 202T) apply transformers to image patches and shifted windows, respectively.
Neural Architecture Search Neural Architecture Search (NAS) automates the search for optimal
CNN designs, as evidenced by works such as (Dong et al.) |020 Li et al., 2022b Baker et al.) 2016
Vahdat et al. 2020 Guo et al. 22020: Chen et al. [202 Niu et al. 202 Guo et al.
2021). Recently,
has been extended to ViTs (Su et al. [2022). AmoebaNet evolves network blocks (Real et al.) 2019).
while (Zoph et al., 2018) use evolutionary algorithms to optimize cell structures. DARTS employs
differentiable searching (Liu et al.) 2018), and PDARTS considers architecture depths (Chen et al.)
2019). (Dong & Yang] 2019) use the Gumbel-Mar trick for differentiable sampling over graphs. NAS benchmarks have also been developed (Ying et al. 2019 Dong & Yang. 2020). This work
2

---

Published as a conference paper at ICLR 2024
aims at the neural architecture retrieval of all existing and future neural architectures instead of those within a pre-defined search space.
Graph Pre-training Strategy Graph neural networks become effective method for graph repre- sentation learning (Hamilton et al. 2017: Li et al., 2015 Kipi' & Welling 2016). To achieve gen- eralizable and accurate representation learning
graph, self-supervised learning and pre-training strategies have been widely studied (Hu et al. 2019 You et al., 2020) Velickovic et al.] 2019). Velickovic al. (2019) used mutual information maximization for node learning. |Hamilton et al. (217)) focused on edge prediction. Hu et al. (2019) employed multiple pre-training tasks at both node and graph levels. You al. (2020) use data augmentations for contrastive learning. Different from previous works which focused on graph node pre-train pre-training. this work pay more attention to the motifs and macro graph of neural architectures pre-train tasks designing due to the character- istic of neural architectures.
3 METHODOLOGY
3.1 PROBLEM FORMULATION
Given query of neural architecture A,, our proposed neural architecture retrieval algorithm returns a set of neural architectures {Ax 1 E A. which have similar architecture as Ag- In order to achieve efficient searching of similar neural architectures, map each neural architecture Ag E AA to an embedding Hg where the embedding of similar neural architectures are clustered. We denote the set of embedding of existing neural architectures H. Through the derivation of embedding Hg of the query neural architecture Ag. set of similar neural architectures (A& [A]k = A2, As} can be found through similarity measurement:
propose to utilize network J to
Hg H, K
Hg + F(A): (
argsort HECH
|| ||
where argsort|-; K] denotes the function to find the indices Is the maximum K val- ues given pre-defined similarity measurement, and With the top-k similarity indices, we can acquire the {Ax] from the candidates A successful searching of similar neural archi- tectures requires an accurate, effective, and ef- ficient embedding network F. Specifically, the network F is expected to capture the architec- ture similarity and be generalized to Out-of- Distribution (OOD) neural architectures. To design network F, we first consider the data structure of neural architectures. Given the model definition, the computational graph can be derived from initialized model. With the computational graph the neural architec- tures can be represented by directed acyclic graph where each node denotes the operation and edge denotes the connectivity. It is natural to apply GNNs to handle this graph-based data. However, there exists some risks when it comes neural architectures. First, the sizes of the neural architecture graphs vary significantly from one to another, and the sizes of models with state-of-the art performance keep expanding. For example, small number of operations are involved AlexNet whose computational graph is in small size Krizhevsky et al. (2012), while recent vision transformer models contain massive operations and their computational graphs grow rapidly Dosovitskiy et al. 2020). Thus, given extremely large computational graphs, there exists an increasing computational burden encoding neural architectures and could be difficult for GNNs to capture valid architec- ture representations. Second, different from traditional graph-based data, there exist motifs in neural architectures. For example, ResNets contains the block design with residual connections and vision transformers contain self-attention modules He et al. (2016); Dosovitskiy et al. (2020), which stacked for multiple times in their models. Since these motifs reflect the architecture designs, taking motifs into consideration becomes essential step in neural architecture embedding.
cosine similarity in Eq. T
Av
Embeddings
ALEA
Encader
H
0
f Faiss (Similanity Search)
Query Embedding.
A A
Figure I: Retrieval (NAR). This paper explores pre-training an encoder F. to build a neural network embed- ding database H. based on the architecture designs.
The definition of Neural Architecture
3

---

Published as a conference paper at ICLR 2024
Input Graph R
Output Sequence
Output Subgraphs
b
b
1
0
0
0
0
0
0
0
0
0
0
0
01
0
0
0
0
Repeat steps
Consider urderss neighhours
Figure 2: An illustration of motifs sampling strategy. The graph nodes encode their neighbours in adjacent matrix via an iterative manner to form the encoded node sequence where each node denotes subgraph and motifs denote the subsequence.
3.2 MOTIFS IN NEURAL ARCHITECTURE
To capture the repeated designs in the neural architectures, we propose to discover the motifs in computational graphs G. The complexity of searching motifs grows exponentially since the size and pattern of motifs in neural architectures are not fixed. For efficient motifs mining, we introduce a new motifs sampling strategy which encodes the neighbours for each node expand the receptive field in the graph. An illustration is shown in Figure 2 Specifically, given the computational graph G with m nodes, we first compute the adjacent matrix M m xm and label the neighbour pattern for each node via checking the columns of adjacent matrix. As shown in the left part of Figure 2. a new label is assigned to each new pattern of sequence in Mism,i where we denote the label of node N, C.. With C' encoding the first order neighbours, each node can be represented by motif. Through performing this procedure by s steps in an iterative manner, the receptive field can be further expanded. Formally, the node encoding can be formulated as
CK- M
if M + J.
M i = ENM(MMLL))) c..m = (M), ENM(M)
otherwise
(2)
where it denotes the encoding step, and ENg denotes the label function which assigns new or ex- isting labels the corresponding sequence in M With Eq. [ after steps, the computational graph is converted to sequence of encoded nodes C and each node encodes order-s neighbours the graph. The motifs can be easily found in C through discovering the repeated subsequences. In Figure? 2 we illustrate with a toy example which only considers the topology in adjacent matrix without node labels and takes the parents as neighbours. However, we can easily generalize to the scenario where both parents and children are taken into consideration as well as node labels through the modification of adjacent matrix M at the first step.
3.3 MOTIFS TO MACRO GRAPH
With the motifs in neural architectures, the aforementioned risks including the huge computational graph size and the involvement of motifs in neural architectures can be well tackled. Specifically, we propose represent each motif G, node with embedding Hag and recover the computational graph G to form a macro graph Gm through replacing the motifs in G by the motifs embedding according the connectivity among these motifs. An illustration of macro graph setup is shown in Figure 3 (a). All the motif's are mapped from G, to the embedding Hag respectively through a multi-layer graph convolutional network as
H!'l ((+1)) = "
(3)
where 1 denotes the layer, a denotes the activation function, W denotes the weight parameters, D denotes diagonal node degree matrix of Mag. and Ma, Mag + where Mag is the adjacent matrix of motif G, and / is the identity matrix. Since we propose to repeat s steps in Eq. to cover the neighbours in the graph, the motifs have overlapped edges, such as edge 0 - 1 and edge 3 - in Figure (3 (a), which can be utilized to determine the connectivity of the nodes H., in the macro graph. Based the rule that the motifs with overlapped edges connected, we build the macro
4

---

Published as a conference paper at ICLR 2024
Input Graph
Embedding
Graph
Motifs
P5g1
Hsg1
Pos
New
Nog
Pin
Contest
Gruph
GCN
High
High
(b). Motih Contrastive Learning
ReaNet-18
WRN-36-00
Hlgg
Macro Graph
VIT-S
Swind
DeiT-
(a). Marco Graph Metifs Embedding
(c). Marco Graph Contrastive Learning
Figure 3:
An illustration of macro graph setup and pre-training in motifs-level and graph-level.
graph where each node denotes the motif embedding. With macro graph, the computational burden of GCNs due to the huge graph size can be significantly reduced. Furthermore, some block and module designs in neural architectures can be well captured via motifs sampling and embedding. For better representation learning of neural architectures, we introduce two-stage embedding learning which involves pre-train tasks in motifs-level and graph-level respective.
3.4 MOTIFS-LEVER CONTRASTIVE LEARNING
In motifs embedding, we use GCNs to obtain the motif representations. For accurate representation learning motifs which can be better generalized to OOD motifs, we introduce the motifs-level contrastive learning through the involvement of context graph Ge. We define the context graph of motifs G, as the combined graph of G2 and the k-hop neighbours of G2 in graph G. A toy example sample the motif
of context graph with 1-hop neighbours is shown in Figure (b). For example, with nodes 0 and from the graph, the context graph of this motif includes node and 3 in its 1-hop neighbours. With the motifs and their context graphs, we introduce motifs-level pre-train task in a contrastive manner. Formally, given a motif G, E Gs. we denote the corresponding context graph of G2 as the positive sample G and the rest context graph of motifs G3 as the negative samples GE We denote the two GCN network as F, and F. defined in Eq. the contrastive loss can be formulated as
He)
C, = -log
(4)
ed( (dgH,g.Ho). + Hy)
where Hag F. H. = F.(Gt, H' Fo(G)), and d denotes the similarity measurement which we use cosine distance. With Eq. 4 as the motifs-level pre-training objective, an accurate representation of motifs Hag be derived in the first stage. Note that He and H' only exist in the training phase and are discarded during inference phase.
3.5 GRAPHI-LVVEL PRE-TRAINING STRATEGY
With the optimized motifs embedding H from Eq. a significantly reduced size. Similarly, we use a GCN network F. defined in Eq. 3 to embed G. Hn. F.n. F...Gm)). For the clustering of similar graphs, we propose include contrastive learning and classification as the graph-level pre- pre-train tasks via a low level of granularity, such as model family. An illustration is shown in Figure (c). For example, ResNet-18. ResNet-50, and WidtRessNet33411000 He et al. (2016); Zagoruyko & Komodakis (2016) belong to the ResNet family, while ViT-S, Swin-L., Deit-B B Dosovitskiy et al. (2020): Liu et al. (202T): Touvron et al., (2021) belong to the ViT family. Formally, given macro graph Gm, we denote set of macro graphs which belong to the same model family of G, as the positive samples G with size K and those not as negative samples G,,, with size K-, The graph-level contrastive loss can be formulated as
build the macro graph Gm with
ΣK (..
L. = llgg
(5)
ΣKj cdl H.. Ek H...
where (k) = (9t (k)), H (k) = Fm (k)). Besides the contrastive learning in Eq. 5 also include the macro graph classification as another pre-train task which utilizes model family
5

---

Published as a conference paper at ICLR 2024
as the label. The graph-level pre-traiee vre-training objective can be formulated as
LG Lm (Fmi Hag) Loe (f: Hm.
(6)
where Lee denotes the cross-entropy loss, I denotes the classifier head, and e denote the ground- truth label. With the involvement of contrastive learning and classification in Eq. 6 robust graph representation learning can be achieved where the embedding Hm with similar neural architecture designs are clustered while those different designs are dispersed. The two-stage learning can be formulated as
min Ca(Fmm Heg c),
s.t.
argmin (.,
(7)
Fm.J
FaiT.
In inference phase, the optimized GCNs of embedding macro graph Fm and motifs F, are involved, the network F in Eq. can be reformulated as
F(A) = Fm Fmm)=F((AAss(F(MM (Mss(F.(Mss(A))))
(8)
where Agg denotes the aggregation function which aggregates the motifs embedding to form macro graph, and Mss denotes the motifss sampling strategy.
EXPERIMENTS
In this section, we conduct experiments with both real-world neural architectures and NAS archi- tectures to evaluate our proposed subgraph splitting method and two-phase graph representation learning method. We also transfers models pre-trained with NAS architectures to real-world neural architectures.
4.1 DATASETS
Data Collection: We crawl 12, 517 real-world neural architecture designs from public reposi- tories which have been formulated and configured. These real-world neural architectures cover most deep learning tasks, including image classification, image segmentation, object detection, fill-mask modeling, question-answering, sentence classification, sentence similarity, text summary, text classification, token classification, language translation, and automatic speech recognition. We extract the computational graph generated by the forward propagation of each model. Each node in the graph denotes an atomic op- eration in the network architecture. The data structure of each model includes: the model name, the repository name, the task name, a list of graph edges, the number of FLOPs, and the number of parameters. Besides, we also build dataset with 30, 000 NAS architectures gen- erated by algorithms. The architectures follow the search space of DARTS Liu et al. (2018) and are split into 10 classes based on the graph editing distance.
Fine-grained
Coarse-grained
Fine-groined
+5 gpt
v9g
in
attn
OUT
dense
bert
conv
former
in
uunknown out
resnet
vit
alexnet
former
conv
YPut
inception
Data Pre-Procsssnng: We scan the key phrases and operations from the raw graph edges of the Figure 4: Coarse-grained and fine-grained classes.
model architecture. We identify the nodes in the graph based on the operator name and label each edge as index. Each node is encoded with a one-hot embedding representation. The key hints such as 'former', 'conv' and 'roberta' extracted by regular expressions tools represent fine-grained classification, which are treated as the ground truth label of the neural network architecture Eq, 6 We then map the extracted fine- tine-grained hints to the enn-block, attention- and other block as the coarse-graing labels. Due to the involvement of motifs in neural architectures, we extract the main repeated block cell of each model by the method presented in the Also, we scan these real-world neural architectures and extracted 89 meaningful operators like "Addmm", "NativeLayerNorm" "AvgPool2D" and removed useless operators "Tbackward" and "Accumulate structure of each pre- pro-pssee record consists of model name, repository name, task name, unique operators, edge index, one-hot embeddings representation, coarse -raiined label. We divided the pre- pre-processed data records to train/test splits (0.9/0.1) stratified based on the ane-grained classes for testing the retrieval performance the real-world neural architectures.
The data

---

Published as a conference paper at ICLR 2024
MRR
MAP
NDCG
Dataset
Method
Top-20
Top-50
Top-100
Top-20
Top-50
Top-100
Top-20
Top-50
Top-100
GCN
0.737
0.745
0.774
0.598
0,560
0.510
0.686
0.672
0.628
Real
GAT
0.756
0.776
0.787
0.542
0.541
0.538
0.610
0.598
0.511
Ours
0.825
0.826
0.826
0.593
0.577
0.545
0.705
0.692
0.678
GCN
1.000
1.000
1.000
0.927
0.854
0.858
0.953
0.902
0.906
NAS
GAT
1.000
1,000
1.000
0.941
0.899
0.901
0.961
0.933
0.935
Ours
1,000
1.000
1.000
0,952
0.932
0.935
0.969
0,960
0.958
Table 1: Comparison with baselines on real-world neural architectures and NAS data.
Dataset
Splitting
MRR
MAP
NDCG
Top-20
Top-50
Top-100
Top-20
Top-50
Top-100
Top-20
Top-50
Top-100
Node Num
0.807
0.809
0.809
0.551
0.539
0.537
0.694
0,682
0.667
Motif Num
0.817
0.820
0.823
0.591
0.522
0,518
0.692
0.669
0.661
Real
Random
0.801
0.802
0.804
0,589
0,543
0.536
0.699
0.675
0.668
Ours
0.825
0.826
0.826
0.593
0.577
0.545
0.705
0.692
0.678
Node Nurn
0.999
0.999
0.999
0,941
0.885
0.883
0,962
0,926
0,924
Motif Num
0.998
0.998
0.998
0,931
0.872
0.874
0.956
0.917
0.919
NAS
Random
1.000
1.000
1.000
0,919
0.826
0.824
0.949
0.881
0.883
Ours
1.000
1.000
1.000
0.952
0.936
0.935
0.969
0.957
0.958
Table 2: Evaluation of different graph split methods on real-world and NAS architectures.
4.2 EXPERIMENTAL SETUP
In order to ensure the fairness of the implementation, we set the same hyperparameter training recipe for the baselines, and configure the same input channel, output channel, and number layers for the pre-training models. This encapsulation and modularity ensures that all differences in their retrieval performance come from a few lines of change. In the test stage, each query will get the corresponding similarity rank index and is compared with the ground truth set. We utilize the three most popular rank-aware evaluation metrics: mean reciprocal rank (MRR), mean average precision (MAP), and Normalized Discounted Cumulative Gain (NDCG) evaluate whether the pre-trained embeddings can retrieve the correct answer in the top k returning results. We now demonstrate the use of our pre-training method as a benchmark for neural network search. We first evaluate the ranking performance of the most popular graph embedding pre-training baselines. Afterwards we investigate the performance based on the splitting subgraph methods and graph-level pre-training loss function design. Then we conduct the ablation studies on the loss functions to investigate the influence of each sub-objective and show the cluster figures based on the pre-training class.
4.3 BASELINES
We evaluate the ranking performance of our method by comparing with two mainstream graph embedding baselines, including Graph Convolutions Networks (GCNs) which exploit the spectral structure of graph in convolutional manner Kipf & Welling (0116) and Graph attention networks (GAT) which utilizes masked self-attention layers Veličković al. (217)). For each baseline model, we feed the computational graph edges as inputs. The model self-supervised learning by contrastive learning and classification on the mapped coarse- gaanse-rraiedd label. Each query on the test set gets re- turned similar models list and the performance is evaluated by comparing the top-k candidate models and ground truth of similarity architectures. Table [ lists the rank-aware retrieval scores on the test set. We observed that our pre-training method outperforms baselines by achieving different degrees of improvement. On the dataset, the upper group of Taemontdesssttstees the our pre-training method outperforms the mainstream popular graph embedding methods. The average score of MRR, MAP and NDCG respectively increased by +5.4%, +2.9%, + +33 on the real-world neural architectures search. On the larger nas datasets, model also achieved considerable enhancement of the ranking predicted score with +1.1%, +3.4% on map@20, map@100 and with +8%, +2.9% on NDGC@20 and NDCG@100.
7

---

Published as a conference paper at ICLR 2024
MRR
MAP
NDCG
Dataset
Objective
Top-20
Top-50
Top-100
Top-20
Top-50
Top-100
Top-20
Top-50
Top-100
CE
0.824
0.828
0.829
0.583
0.573
0.539
0.703
0.693
0.692
Real
CL
0.565
0.572
0.573
0.334
0.348
0.373
0.502
0.455
0.451
CF+CI
0.825
0.826
0.826
0.593
0.577
0.545
0.705
0.692
0.678
CE
1.000
1.000
1,000
0.953
0.921
0.923
0,969
0,952
0.950
NAS
CL
0.925
0.925
0,925
0.750
0.658
0.656
0.829
0.762
0.764
CE&CL
1.000
1.000
1.000
0.952
0.932
0.935
0.969
0.955
0.958
Table 3: Ablation study of different loss terms (CE: Cross Entropy; CL: Contrastive Learning).
4.4
SUBGRAPH SPLITTING
We compare our method for splitting subgraphs with three baselines. Firstly, we use two methods uniformly split subgraphs, where the number of nodes in each subgraph (by node number) or the number of subgraphs (by motif number) are specified. If the number of nodes in each subgraph specified, architectures are split into motifs of the same size. Consequently, large networks are split into more motifs, while small ones are split into fewer motifs. If the number of subgraphs is specified, different architectures are split into various sizes of motifs to ensure the total number of motifs is the same. Then, we also use method to randomly split subgraphs, where the sizes of motifs are limited to a given range. We report the results Table As can be seen, our method can consistently outperform the baseline methods on both real-network and NAS architectures. When comparing the baseline methods, we find that for NAS architectures, splitting by node number and by motif number reaches similar performances. It might be because NAS architectures have similar sizes. For real-network architectures, whose sizes vary, random splitting reaches the best NDCG among all baselines, and splitting by motif number reaches the best MRR. Considering MAP, split- ting by motif number achieves the best Top-20 performance, but splitting by node number achieves the best Top- 100 performance. On the other hand, our method can consistently outperform the base- lines. This phenomenon implies that the baselines are not stable under different metrics when the difference in the size of architectures is -egliggige
4.5 OBJECTIVE FUNCTION
Since graph-level pre-training is multi-objectives task, it is necessary to explore the effective- ness of each loss term by removing one of the components. All hyperparameters of the models are tuned using the same training receipt in Table 5 Table provides the experimental records of different loss terms. In terms of the Real dataset, the model trained with both CE and CL. (CF+CL) outperforms the models trained with either CE or CL alone across almost all metries. Specifically, the MRR scores for CE+CL are 0.825, 0.826, and 0.826 for Top-20, Top-50, and Top-100, respec- tively. These scores are marginally better than the CE-only model, which has MRR scores of 0.824, 0,828, and 0.829, and significantly better than the CL-only model, which lags with scores of 0.565, 0.572, and 0.573. Similar trends are observed MAP and NDCG scores, reinforcing the notion that the combined loss term is more effective. For the NAS dataset, the CE+CL model again demon- strates superior performance, achieving perfect MRR scores of 1. 1.000 across rankings. While the CE-only model also achieves perfect MRR scores, it falls short in MAP and NDCG metrics, espe- cially when compared to the combined loss term. The ablation study reveals that a multi-objective approach involving both graph-level contrastive learning and coarse label classification is most ef- fective in enhancing neural architecture retrieval performance. Furthermore, the contrastive loss term, while less effective on its own, plays a crucial role in boosting performance when combined with cross-entropy loss.
4.6 TRANSFER LEARNING
We also monitor whether NAS pre-training benefits the structure similarity prediction of the real- world network. For this, we design the experiment of transferring the pre-trained model from the NAS datasets to initialize the model for pre- pre-training on the real-world neural architectures. The results demonstrated in Table 4 shows the model pre-trained on the real-world neural architectures achieves an improvement on most evaluation metrics, which reveals the embeddings pre-trained by initialized model obtains the prior knowledge and get benefits from the NAS network architecture
8

---

Published as a conference paper at ICLR 2024
MRR
MAP
NDCG
Training Method
Top-20
Top-50
Top-100
Top-20
Top-50
Top-100
Top-20
Top-50
Top-100
Training from Scratch
0.825
0.826
0,593
0.577
0.545
0.705
0.692
0.678
Pre-training with NAS
0.821
0.838
0.839
0.596
0,584
0.573
0.712
0.703
0.706
Table 4: Transfer model pre-trained with NAS architectures real-world neural architectures.
(a) GCN real (b) GAT on real
(c) Ours real (d) GCN on NAS (e) GAT NAS (I) Ours on NAS
Figure 5: Visualization of learnt embeddings. The number of dimensions is reduced by I-SNE. da
b and are embeddings of real-world neural architectures, and d and are embeddings of NAS
architectures)
searching. And with the increment of top-k of rank lists, the model unitized with NAS pre-training yields a higher score compared with the base case, which means enlarging the search space could boost the similarity model structures by using the model that transferred from NAS.
4.7 VISUALIZATION
Besides the quantitative results provided in Table 1 we further provide qualitative results through the visualization of cluttering performance in Figure To illustrate the superiority of our method over other baselines, we include both GCN and GAT for comparison. For visualization, we ap- ply 1-SNE Van der Maaten & Hinton (2008) to visualize the high-dimensional graph embedding through dimensionality reduction techniques. As shown in Figure aa (b.. and ( we visualize the clustering performance of real-world neural architectures on three different categories, including cianttionbbsed blocks (green), CNN-based blocks (red), ang other blocks (blue). Comparing the visualization results on real-world neural architectures, is obvious that both GCN and GAT cannot perform effective clustering of the neural architectures with the blocks from same category. On the contrary, our proposed method can achieve better clustering performance than other baselines. Sim- ilarly, we conduct visualization on the NAS data. We first sample ten diverse neural architectures from the entire NAS space as the center points of ten clusters respectively. Then we evaluate the clustering performance of neural architectures sampled around center points that have similar graph editing distance from these sampled center points. The results are shown in Fig. 5 dd. @e), and ( Consistent with the results on real-world data, we can see that our method can achieve better clus- tering performance on NAS data with clear clusters and margins, which provides strong evidence that our method can achieve accurate graph embedding for neural architectures.
5 CONCLUSION
In this paper, we define a new and challenging problem Neural Architecture Retrieval which aims at recording valuable neural architecture designs as well as achieving efficient and accurate retrieval. Given the limitations of existing GNN-based embedding techniques learning neural architecture representations, we introduce a novel graph representation learning framework that takes into con- sideration the motifs of neural architectures with designed pre-training tasks. Through sufficient evaluation with both real-world neural architectures and NAS architectures, we show the superiority of method over other baselines. Given this success, build dataset with 12k differ- ent collected architectures with their embedding for neural architecture retrieval, which benefits the community of neural architecture designs.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported in part by the Australian Research Council under Projects DP240101848 and FT230100549.
REFERENCES
Bowen Baker, Otkrist Gupta, Nikhil Naik, and Ramesh Raskar. Designing neural network architec- tures using reinforcement learning. arXiv preprint arXiv:1611.0216, 2016.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part 16, pp. 213-229. Springer, 2020.
Wei-Cheng Chang, Felix X Yu, Yin-Wer Wen Chang, Yiming Yang, and Sanjiv Kumar. Pre-training tasks for embedding- large-scale retrieval. arXiv preprint arXiv:2002.03932, 2020.
Xin Chen, Lingxi Xie, Jun Wu, and Qi Tian. Progressive differentiable architecture search: Bridging the depth gap between search and evaluation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1294-1390 2019.
Yaofo Chen, Yong Guo, Qi Chen, Minli Li, Wei Zeng, Yaowei Wang, and Mingkui Tan. Contrastive neural architecture search with neural architecture comparators. In Proceedings the conference on computer vision and pattern recognition, pp. 9502- 1002-911,, 2021.
the IEEE/CVF
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: arXiv:1810.04805. 2018.
Minjing Dong, Yanxi Li, Yunhe Wang, and Chang Xu. Adversarially robust neural architectures.
arXiv preprint arXiv:2009.00000 2020.
Minjing Dong, Yunhe Wang, Xinghao Chen, and Chang Xu. Handling tailed feature distribu- tion in addernets. Advances in Neural Information P Processing Systems, 34: 14:17992-1791,, 2021.
Minjing Dong, Xinghao Chen, Yunhe Wang, and Chang Xu. Improving lightweight addernet via distillation from & to fo-nrmm IEEE Transactions on Image Processing. 2023.
Xuanyi Dong and Yang. Searching for a robust neural architecture in four gpu hours. Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1761-1770, 2019.
Xuanyi Dong and Yang. Nas-bench-200 Extending the scope of reproducible neural architecture search. arXiv preprint 0tti::20010002200 2020.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. arXiv preprint 0rXiv:2010.11929 2020.
Yong Guo, Yaofo Chen, Yin Zheng, Peilin Zhao, Jian Chen, Junzhou Huang, and Mingkui Tan. Breaking the curse of space explosion: Towards efficient nas with curriculum search. In Interna- tional Conference on Machine Learning, pp. 3822 1831- PMLR, 2020.
Yong Guo, Yin Zheng, Mingkui Tan, Qi Chen, Zhipeng Li, Jian Chen, Peilin Zhao and Junzhou Huang. Towards accurate and compact architectures via neural architecture transformer. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10):6501-6516 2021.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs.
Advances in neural information processing systems, 30, 2017.
Kai Han, Yunhe Wang, Qi Tian, Jianyuan Guo, Chunjing Xu, and Chang Xu. Ghostnet: More features from cheap operations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1580-1589, 2020.
10

---

Published as a conference paper at ICLR 2024
Kai Han, Yunhe Wang, Chang Xu, Jianyuan Guo, Chunjing Xu, Enhua Wu, and Qi Tian. Ghostnets on heterogeneous devices via cheap operations. International Journal of Computer Vision, 130 (4):1990-106, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andrectto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: (70.vv/70440486., 2017.
Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132-7141, 2018.
Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. arXiv preprint arXiv: arXiv:1905. 1905. 12265, 2019.
Gao Huang. Zhuang Liu, Laurens Van Der Maaten, and Kilian Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
Jui-Ting Huang, Ashish Sharma, Shuying Sun, Li Xia, David Zhang, Philip Pronin, Janani Padman- abhan, Giuseppe Ottaviano, and Linjun Yang. saahedding-bassed retrieval in facebook search. In Proceedings the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 2553-2561, 2020.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. arXiv preprint 16Xvv1660.00207, 2016.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In F. Pereira, C.J. Burges, L. Bottou, and K.Q. Wein- berger (eds.), Advances Neural Information Processing Systems, volume 25, Curran Asso- ciates, Inc., 2012. URL https: http:://ccoeeeeee/s.ssoe https:/:///poeeeeeeeeeeee 23382222266336663333322222222222
neurips wttp:/pss..pppppp//////////////. pdf
Yanxi Li, Xinghao Chen, Minjing Dong, Yehui Tang, Yunhe Wang, and Chang Xu. Spatial-channel token distillation for vision mlps. In International Conference on Machine Learning, pp. 12685- 12695. PMLR, 2022a.
Yanxi Li, Minjing Dong, Yunhe Wang, and Chang Xu. Neural architecture search via proxy valida- tion. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(6):7595-7611 2022b.
Yujia Li, Daniel Tarlow, Mare Brockschmidt, and Richard Zemel. Gated graph sequence neural networks. arXiv preprint
2015.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv
preprint arXiv: (a0o::0000..0555. 2018.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings the JEEE/CVF international conference on computer vision, pp. 10012-10022, 2021.
Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yong Guo, Peilin Zhao, Junzhou Huang, and Mingkui Tan. Disturbance-immun immune weight sharing for neural architecture search. Neural Networks, 144: 553-564, 2021.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Le. Regularized evolution for image classifier architecture search. Proceedings of the aaai conference on artificial intelligence, pp- 4780-4789, 2019.
11

---

Published as a conference paper at ICLR 2024
Xiu Su, Shan You, Jiyang Xie, Mingkai Zheng. Fei Wang, Chen Qian, Changshui Zhang, Xiaogang Wang, and Chang Xu. Vitas: Vision transformer architecture search. In European Conference on Computer Vision, pp. 139-157. Springer Nature Switzerland Cham, 2022.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-9, 2015.
Mingxing Tan, Ruoming Pang, and Quoe V Le. Efficientdet: Scalable and efficient object detection. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp-
1078 110790, 2020.
Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Feos: Fully convolutional one-stage object detection. In Proceedings of the IEEE/CVF international conference computer vision, pp- 9627-9636, 2019.
Hugo Touvron, Matthicu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pp. 10347-10357. PMLR, 2021.
Arash Vahdat, Arun Mallya, Ming-Yo Yu Liu, and Jan Kautz. Unas: Differentiable architecture search meets reinforcement learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11266-11271 2020.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint 1700 2017.
Petar Velickovic, William Fedus, William Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. Deep graph infomax. ICLR (Poster), 2(3):4, 2019.
Yunhe Wang, Chang Xu, Chunjing Xu, Chao Xu, and Dacheng Tao. Learning versatile filters for efficient convolutional neural networks, Advances in Neural Information Processing Systems, 31, 2018.
Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter. Nas- bench-101: Towards reproducible neural architecture search. In International Conference on Machine Learning, pp. 7105-7114. PMLR, 2019.
Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. Advances in neural information processing systems, 33: 5812-5823, 2020.
Sergey Zagoruyko and Nikos Komodakis. 16X1:16605.07166. 2016.
Wide residual networks.
arXiv preprint
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 6848-6856, 2018.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoe V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8697-8710, 2018.
12

---

Published as a conference paper at ICLR 2024
A
APPENDIX
A.I
APPLICATION DEMOS
A,
Embeskings
What my derigned trchiesttrna could schims on the given taak
What toski this design could petentidly applied?
Ae EE
Eneader
H
f
Query Embedding
Foss
A $
(Simiority Search)
Figure 6: depiction the potential downstream applications for NAR.
Naural Netwerk Search Engine
Naural Network Search Engine
Architectures Retrieval for the specific task.
(h) Architectures Retrieval for various applications.
Figure 7: Cases of potential downstream tasks based on the NAR.
A.2 ALGORITHM
Algorithm NAR Pre-training
Require: set G of computational graph
Ensure: Ft and Fe
1: for G do
b motifs-level CL, Fig. motifs sampling, Fig.
2: Get G. from G
3: for Ge E Gg do
4:
Get G G based G.
5:
Calculate L, with Eq. and update F,, Fo
6: end for
7:
end for
8:
for G E 1 do
graph-level CL, Fig.
9:
Get G. from G
motifs sampling, Fig.
Build Gm with Fz
build macro graph, Fig.
11:
Calculate La with Eq. 6 and update Fm
12: end for
A.3
DETAILS
A.4 NEURAL ARCHITECTURE GENERATION
To generate diverse neural architectures, we follow the search space design for neural architecture search (NAS) in DARTS Liu et al. (2018), which considers neural architectures directed acyclic graphs (DAGs). A difference is that DARTS treats operations as edge attributes, while we insert an additional node representing operation each edge with operation for consistency. Our space
13

---

Published as a conference paper at ICLR 2024
EPs
Layers
L.R
Emb
Drop
Motifs CL
5
256
3
le-2
512
Graph CL
15
512
1e-3
512
0.1
Baselines
15
512
3
1e-3
512
0.1
Table Pre-training Recipes. EPs: Epochs; BS: Batch size; LR: Learning rate.
(a) The neural architecture generation space;
(b) Example
(c) Example
Figure 8: The neural architecture generation space and two examples.
for architecture generation is as shown in Fig.
g a
We also provide two examples in Fig. and
neural architecture is used to build a cell, and cells are repeated to form a neural network. Each cell c. with k O,..., K takes inputs from two previous cells Ck-2 and Cx-1- The beginning of network is stem layer with an convolutional layer. The first cell co is connected to the stem layer, and the second cell с₁ connected both en and the stem layer. In the other word, c_ and c_ both refer to the stem layer. Finally, the last cell ск is connected to global average pooling and fully connected layer to generate the network output.
In each cell, there are "ADD" nodes A with 0, 1,3,3 The node ADD (+) can be connected to Ck-2, Ck-1, or (DD with S < practice, the number connections is limited to 2. For each connection, we insert node represent an operation. Operations are chosen from 7 candidates, including skip connection tekip_comecth pool_oo3333 3 or 5 x 5 separable convolution (sep_conv_3) _33 or sep_conv cop_conv_5x5) _5x5,, and 3 x 3 or 5 x 5 dilated convolution (dil_cony 3x3 or dil_conv coloc-x5x5) Finally, "Concat" node is used concatenate the 4 "ADD" nodes as the cell output Cb.
3 max pooling amax_pool_3s
3 x 3 average pooling
14


Reviews:
[{"review_id": "HtxEu7b6Kc", "reviewer": "Reviewer_NBZG", "summary": "This paper defines a new question called Neural Architecture Retrieval (NAR) which returns a set of similar neural architectures given a query neural architecture. To address NAR problem, this paper proposes to split the graph into several motifs and rebuild the graph through treating motifs as nodes in a macro graph. Then this paper uses two-level pretrain task to train the architecture representation for retrieval.", "questions": "One small question is that I can't find how the ground truth topk set is constructed, can the author describe it in detail?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This work proposes a large new NAS dataset of 12k real-world neural architectures rather than pre-define search space. In my opinion, this thing makes a lot of sense for the NAS field which can explore the diversity of search space species architectures.\n\n2. The idea uses motifs to encode architecture and reduce the graph size is novel and reasonable to encode architecture and capture the connection between structures in one architecture.\n\n3. The two-level pretrain task to train the architecture's embedding is reasonable and effective.", "weaknesses": "1. In my opinion, the NAR problem to find similar architectures for the query architecture doesn't seem particularly significant for real-world usage, Can the author point out the need for this problem and more application scenarios?\n\n2. Some retrieval-based papers[1-3] for giving a query dataset to search architectures should be discussed.\n\n3. Another question is that I want to know how much performance can be achieved by retrieving similar models only based on the architecture corresponding to the text description or the code by a language mode like Chatgpt or other language-based retrieval model.\n\nRefs: \n\n[1] Task-Adaptive Neural Network Search with Meta-Contrastive Learning. NeurIPS 2021.\n\n[2] MetaGL: Evaluation-Free Selection of Graph Learning Models via Meta-Learning. ICLR 2023\n\n[3] Retrieving GNN Architecture for Collaborative Filtering. CIKM 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BHx5lyiFsS", "reviewer": "Reviewer_wPum", "summary": "The paper presents a new problem of Neural Architecture Retrieval, which aims to find similar neural architectures from a large collection of existing and potential ones. The paper introduces a novel graph representation learning framework that leverages the motifs of neural architectures and contrastive learning to achieve accurate and efficient retrieval. The authors also construct a new dataset of 12k real-world neural architectures with their embeddings and evaluates the proposed method on both real-world and NAS architectures, showing its superiority over baselines. However, the paper has some issues that need to be addressed. My detailed comments are as follows.", "questions": "1. In Page 2 line 8, “exact” should be “exactly”.\n\n2. In Page 2 line 11, “through” should be “by”.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe paper presents a new problem called Neural Architecture Retrieval (NAR). This problem is to find similar neural architectures quickly and easily from a big pool of existing and possible designs. It’s a smart way to make the search for neural architectures simpler and more efficient.\n\n2.\tThe authors propose a novel and easy-to-understand graph representation learning framework that addresses the computational graph in neural architectures. This framework adopts motifs of neural architectures and multi-level contrastive learning for accurate graph representation learning.\n\n3.\tThe paper contributes the community by creating a new dataset with 12k real-world network architectures and their embeddings. This dataset is specifically designed for neural architecture retrieval and demonstrates the effectiveness of the retrieval algorithm. It’s a helpful benchmark for everyone working in this field.\n\n4.\tThe experimental results on both human-designed and synthesized neural architectures benchmarks verify the effectiveness of the proposed method.", "weaknesses": "1.\tThe proposed NAR appears to focus solely on the topological similarity of architectures. However, it is important to note that the similarity between architectures can vary across different tasks or datasets. For instance, certain architectures might yield comparable results in image classification but diverge significantly in performance when applied to other tasks. Could the authors provide additional insights and elaborations on this matter?\n\n2.\tThe authors introduce a motif-level contrastive learning approach, wherein the corresponding context graph is treated as the positive sample. However, this raises a concern as there may be other context graphs that also encompass the same motifs, yet they are deemed as negative samples, which seems unreasonable. Could the authors provide further clarification and justification for this aspect of their methodology?\n\n3.\tIt is unclear how the authors collect/annotate the ground-truth label of the architecture dataset. Without such labels, it is infeasible to calculate the correlation between the different networks. Please provide more details.\n\n4.\tIn the \"Related Work\" section, it would enhance the manuscript's thoroughness if the authors could offer a more comprehensive discussion on mainstream NAS methods. This should include an exploration of reinforcement learning-based NAS methods, as referenced in [A-G].\n\n[A] Designing Neural Network Architectures using Reinforcement Learning. ICLR 2017.\n\n[B] Learning Transferable Architectures for Scalable Image Recognition. CVPR 2018.\n\n[C] UNAS: Differentiable Architecture Search Meets Reinforcement Learning. CVPR 2020.\n\n[D] Breaking the Curse of Space Explosion: Towards Efficient NAS with Curriculum Search. ICML 2020.\n\n[E] Contrastive Neural Architecture Search with Neural Architecture Comparators.  CVPR 2021.\n\n[F] Disturbance-immune Weight Sharing for Neural Architecture Search. Neural Networks 2021.\n\n[G] Towards Accurate and Compact Architectures via Neural Architecture Transformer. TPAMI 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZCkRmKUSOA", "reviewer": "Reviewer_BEfk", "summary": "This is a very interesting paper, considering the burgeoning field of neural architecture designs which has made it increasingly challenging for researchers to position their contributions or establish relationships between their designs and existing ones. The paper presents a novel problem of neural architecture retrieval, aiming to automate the discovery of similar neural architectures. The authors identify the limitations of existing graph pre-training strategies, which are unable to handle the computational graphs in neural architectures due to the graph size and motifs. They propose a creative solution by dividing the graph into motifs, which are then used to reconstruct the macro graph, addressing the identified issues. Moreover, the introduction of multi-level contrastive learning is put forth to attain precise graph representation learning. The paper boasts extensive evaluations of both human-designed and synthesized neural architectures, showcasing the superiority of their proposed algorithm. The authors also construct a valuable dataset comprising 12k real-world network architectures along with their embeddings, laying a solid foundation for neural architecture retrieval. This endeavor may address a pressing issue in the domain with pre-training an embedding database for finding and comparing neural architectures. The paper potentially opens up a new domain within the neural architecture community, paving the way for more organized and efficient exploration in this field by building an embedding database, and the video demo shows promising applications.", "questions": "Page 5 Section 3.4 Eq 4: What role does the context graph $G_s$ play?\n\nPage 6, Section 4.1: How were the real-world repositories collected, and how were the models obtained? Will the 12k real-world computational graphs, along with corresponding labels, be made available for follow-up work?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "* The motivation behind the paper is well-articulated and resonates with the ongoing challenges faced by researchers in situating their contributions amidst a plethora of existing neural architectures. The introduction of Neural Architecture Retrieval as a solution to automate the discovery of similar neural architectures is timely and could significantly alleviate the existing bottleneck.\n* The methods used in the paper sound and well-justified. The logic behind each step of the solution is sound and reasonable, showcasing a thorough understanding of the challenges at hand. For instance, the approach to addressing the repeated design of blocks is practical and applicable, demonstrating a commendable level of methodological rigor and a pragmatic stance.\n* The evaluation metrics, particularly the information retrieval scores, are promising, especially when applied to the nas dataset. This suggests that the proposed methods are effective and could potentially set a new standard in evaluating neural architectures. \n* Contribute a 12k real-world computational graph and its corresponding embedding database. The availability of such a dataset could spur further research and development in the domain of neural architecture retrieval and related areas.", "weaknesses": "* Page 5 Section 3.4 Eq 4: The objective of the first stage may have a better way. The encoder encodes the architecture into motifs, and then concatenates the embeddings.  Directly sampling the highest-frequency motifs $H$ to represent the main design for large models may more reasonable, especially considering that the concatenation of motif embeddings cannot backpropagate the gradients without two stages.\n\n* Page 7 Section 4.3: The details of the evaluation part are lacking. Although it covers the mainstream information retrieval scores, the parameters of the computational score are still unclear. For example, when testing the NDCG, is graded relevance or non-graded relevance used? It would be better to provide a formula here.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)