NAR-Former V2: Rethinking Transformer for Universal Neural Network Representation Learning
Authors: Yun Yi
Haokui Zhang
Rong Xiao
Nannan Wang
Xiaoyu Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: Transformer
Graph Neural Network
neural network encoding
representation learning
neural architecture search
neural network deployment

Abstract: As more deep learning models are being applied in real-world applications, there is a growing need for modeling and learning the representations of neural networks themselves. An effective representation can be used to predict target attributes of networks without the need for actual training and deployment procedures, facilitating efficient network design and deployment. Recently, inspired by the success of Transformer, some Transformer-based representation learning frameworks have been proposed and achieved promising performance in handling cell-structured models. However, graph neural network (GNN) based approaches still dominate the field of learning representation for the entire network. In this paper, we revisit the Transformer and compare it with GNN to analyze their different architectural characteristics. We then propose a modified Transformer-based universal neural network representation learning model NAR-Former V2. It can learn efficient representations from both cell-structured networks and entire networks. Specifically, we first take the network as a graph and design a straightforward tokenizer to encode the network into a sequence. Then, we incorporate the inductive representation learning capability of GNN into Transformer, enabling Transformer to generalize better when encountering unseen architecture. Additionally, we introduce a series of simple yet effective modifications to enhance the ability of the Transformer in learning representation from graph structures. In encoding entire networks and then predicting the latency, our proposed method surpasses the GNN-based method NNLP by a significant margin on the NNLQP dataset. Furthermore, regarding accuracy prediction on the cell-structured NASBench101 and NASBench201 datasets, our method achieves highly comparable performance to other state-of-the-art methods. The code is available at https://github.com/yuny220/NAR-Former-V2.

NAR-Former V2: Rethinking Transformer for Universal Neural Network Representation Learning
Yun Vil.3.t
Haokui Zhang 2.*+
Rong Xiao
Nannan Wang
Xiaoyu Wang IIntellifusion
Xidian University Northwesternn Polytechnical University
yuny2200163. com
hkzhang1 99111999911m1.1m11
ataaam..mmpppa....m..e.mmaeeeeea wwwu. edu cn
riao.rongdintellif.. com
nnwangōxidian edu.co fanghuaxue@gaii..com
Abstract
As more deep learning models are being applied in real-world applications, there is a growing need for modeling and learning the representations neural net- works themselves. An effective representation can be used to predict target at- tributes of networks without the need for actual training and deployment proce- dures, facilitating efficient network design and deployment. Recently, inspired by the success Transformer, some Transformer-based representation learning frameworks have been proposed and achieved promising performance in handling cell-structured models. However, graph neural network (GNN) based approaches still dominate the field of learning representation for the entire network. In this paper, we revisit the Transformer and compare it with GNN to analyze their different architectural characteristics. We then propose a modified Transformer- based universal neural network representation learning model -AR-Former V2. It can learn efficient representations from both cell-structured networks and en- tire networks. Specifically, we first take the network as graph and design a straightforward tokenizer encode the network into a sequence. Then, we incorporate the inductive representation learning capability of GNN into Trans- former, enabling Transformer to generalize better when encountering unseen ar- chitecture. Additionally, we introduce series of simple yet effective modifica- tions to enhance the ability of the Transformer in learning representation from graph structures. In encoding entire networks and then predicting the latency, our proposed method surpasses the GNN-based method NNLP by a significant mar- gin on the NNLQP dataset. Furthermore, regarding accuracy prediction cell-structured NASBench101 and NASBench201 datasets, our method achieves highly comparable performance other state-of-the methods. The code is available
the
Former- V2
Introduction
With the maturity of deep learning technology, increasing number of deep neural network mod- els various sizes and structures are being proposed and implemented academic research and industrial applications. In this process, the rapid deployment of networks and the design of new net- works that meet task requirements are significant. To address this issue, researchers propose using machine learning models solve the deployment and design problems of the models themselves. One popular strategy is encoding the input neural network and utilizing the resulting neural network representation to predict specific target attribute directly without actually executing the evaluation program. In recent years, we have witnessed success in accelerating model deployment and design
This work was done while Yun Yi was intern at Intellifusion. *Corresponding authors.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

processes with the help of neural network representations P 55 12. DJ, significant time cost and expertise efforts can be saved by not having to carry out the time-consuming process of compilation, deployment, inference, and latency evaluation when engineers choose networks for application. Through the use of accuracy predictors [47,, 2R, H.. B. D am! 20], researchers can avoid the resource-intensive pro- cess of network training and instead perform a forward inference process to evaluate the accuracy of multitude of networks. This measure dramatically reduces the time cost associated with network design.
211. R.
An,
44 12 Taking the 26
advantages of latency predictors [9:Z. 2T. I7
Although the vanilla Transformer is designed for natural language processing, Transformer archi- tecture has found widespread adoption across diverse fields owing to its strengths in global mod- eling and parallelizable computation [ 22. D3, IZL AI R]. Very recently, several researchers have attempted to learn appropriate representations for neural networks via Transformer [477, [6]]- These methods have indeed achieved leading performance on relevant tasks. Nevertheless, they are mainly designed for encoding the architecture of cells (basic micro units of repeatable neural net- works) in cell-structured networks. As shown in the latency prediction experiment in -AR-Former (Transformer- neural architecture representation learning framework) [] poor generalization performance occurs when the depth of the input architecture reaches hundreds of layers. In the de- velopment process of neural network representation learning, Graph neural network (GNN) [TG. 35] is also a promising technique for learning neural network representations [21 BM Tq, 33 [2] They model the input neural architecture as a directed acyclic graph (DAG) and operate on the graph- structured data, which comprises the node information matrix and adjacency matrix. Recently, the NNLP [2T] introduced a dedicated lateney prediction model based on GNNs, which is capable of en- coding the complete neural network having hundreds of layers and achieves cutting-edge advance.
In fact, both cell-structured architectures and complete neural networks are widely used in various applications. Cell-structured models offer good scalability, allowing for easy scaling by adding or removing cells. This adaptability makes them suitable for addressing problems of different com- plexities and data sizes, while also facilitating incremental model development and deployment. Complete neural networks provide better flexibility in connectivity and can achieve higher accuracy in certain cases. Furthermore, in some cases, such as in latency estimation, encoding the complete network is necessary. To handle various network architectures in different tasks, both GNN- and Transformer-base models are necessary. However, this issue of utilizing multiple architectures can introduce constraints that may not be conducive to practical applications. For instance, when designed network requires specific attributes, having similar model structures and high-accuracy predictions for different attributes can reduce code redundancy and improve work efficiency.
In this paper, we build upon the research conducted in NAR-Former [+7]] and present a novel frame- work called NAR-Former V2 for universal neural network representation learning. Our framework can handle cell-structured networks as well learn representations for the entire network. To ac- complish this, we incorporate graph-specific properties into the vanilla Transformer and introduce graph-aided attention-based Transformer block. This approach combines the strengths of both Transformer and graph neural networks (GNNs). Extensive experiments are conducted to evaluate our proposed framework. Results show that:(1) our method can be applied to predict different at- tributes, can outperform the state-of-the-ar method in latency prediction on the NNLQP dataset [2]]], and can achieve promising results in accuracy prediction on the NAS-Bench-IM and NAS-Bench- 201 datasets [48, I0;; (2) our method has good scalability, which is capable of encoding network having only a few operations or complete neural networks that have hundreds of operations.
2 Related work
2.1 Representation and attribute prediction of neural networks
Neural network representation learning is the base for evaluating the attributes of different networks via machine learning models. Early methods [7. 20] construct representation models for learning neural network representation based on LSTM and MLP. Peephole [M]] inputs the embedding of each layer to LSTM to predict accuracy, which neglects the topological structure and limited to han- dling only sequential architectures. Later, in order to better capture the structural information of the network, an accuracy predictor [+:]] uses a binary path encoding with a length equal to the number of possible paths from input to output given in terms of operations, where the element at the corre-
2

---

sponding position of the path presenting in the input network is set to 1. When the neural network is regarded as a directed acyelic graph, the adjacency matrix describes the connection between nodes, so it is naturally used to encode the topological structure of the neural network. [88] proposed to encode the given neural network as concatenated vector of flat adjacency matrix and list of node labels. Many other methods [2I1, I7, B. IT2 S3 [9] realize accuracy and latency pre- diction by directly inputting the original two-dimensional adjacency matrix together with the node information matrix to GNN, which can realize the explicit encoding of the input network topology. There also are some works exploring other ways of obtaining representations of neural networks, such as by mimicking actual data processing [+ or message exchange [S0]] of neural networks, and by deriving a number of persistent topology measures for DNNs [S]] Recently, other methods have focused enhancing the original GNN [ introducing transformers [26, [7] to obtain more meaningful neural network representations.
101
2.2 Transformer
Transformer |
is self-attention--assee neural network architecture that has revolutionized natu- ral language processing [28] 8. [3 and has been adopted in many other fields [54, 24,, 22, 2X PW
26. II [T]. Because of its excellent modeling capability and wide range of applications, there is a growing interest in the direction of further enhancing its performance [5.. II, 39 K]], improving
computing efficiency [23 EIT. G combining with other backbone networks 1|S. IS 6X. 29 [
and even taking inspiration from it to optimize other types of network structures, such as ConvNets B5. PA 522 Transformer has recently been successfully introduced into neural network rep- resentation learning [26, AZ]]. TNASP [26] inputs the sum of the operation type embedding matrix and Laplacian matrix into the standard Transformer. NAR-Former [47]], the other hand, encodes each operation and connection information of this operation into a token and inputs all tokens into a proposed multi-stage fusion transformer. Excellent attribute prediction results have been achieved on cell-based datasets by using these methods. However, the strong long-range modeling ability of the self-attention mechanism may also result in subtle local variation affecting the representation of all tokens. Due to the potential impact of this feature on the generalization ability. although NAR-Former [+7]] has made attempts to encode complete neural networks, the results are still unsat- isfactory.
2.3 Graph neural network
data, which is a fundamental representation for many real-world problems such as social network analysis and recommendation systems [99, B2] Given that neural networks can be viewed as graphs, GNN-based models have emerged as a prominent and widely adopted approach for neural network representation learning [211. 17. R. 122 EK [%]] GNNs show generalization ability through a simple mechanism of aggregating information from neighbors. For instance, the recently proposed GNN-based model [21]] can obtaín representations of neural networks with hundreds of layers and achieves new state-of-the-art results in latency prediction, even if the input network structure has not been seen during training. Nevertheless, the simple structural characteristics of GNNs, which contribute to their strong generalization ability, also lead to the need for further improvement in the performance of methods based on original GNN in cellular structure and complete neural network representation learning. Therefore, it is promising approach for neural network representation learning to combine the Transformer and GNN to leverage the strengths of both models.
GNNs are designed handle
3 Method
Our final framework diagram for realizing neural architecture representation learning and attribute
prediction shown in Fig. II consisting three main phases: neural network encoding, backbone- based representation learning, and attribute predicting using prediction heads. We will introduce the
motivation and details of the backbone network design in Sec. 3.] and Sec. 6.2, and the details of the entire model in Sec.
3

---

Enation Information network represomation
GraphAln-basedd Transformer Block
Predicting
cell//
Takenizer
Neural Network Encoding
Representation Learning
Atributes Predcting
Figure 1: Overview of attribute prediction model.
3.1 Motivation
As mentioned in the Sec. Transformer-base based models have demonstrated remarkable performance in encoding and learning representations of neural networks when the input is in the form of cells. However, when dealing with complete deep neural networks (DNNs) consisting of hundreds of lay- ers, or when the depth of the input data is unseen during training, they may sometimes exhibit poorer performance compared GNN-based methods. Additionally, highlighted [2T]] real-world ap- plications often show significant differences in the topologies and depths between training and test samples. Consequently, representation learning models must possess strong generalization abilities for handling unseen data. In this regard, GNN-based models appear to achieve better performance.
This observation has prompted to reconsider the two types of inputs, namely cells and complete DNNs, as well as the two representation learning models, the Transformer and GNN. Through a detailed comparative analysis of the structures of the Transformer and GNN, we speculate that the insufficient generalization capability of Transformer-based methods may be attributed to its struc- ture and computation characteristics. As know, the self-attention structure in transformers is a crucial design that allows for the effective extraction of global features in data-driven manner. However, this structure becomes a double-edged sword when learning network representations. For input neural networks with depths of hundreds of layers, the Transformer's 's impressive capability to capture global information can sometimes lead excessive sensitivity. This stems from the fact that the Transformer models interactions between all tokens using its self-attention mechanism, treating the entire sequence as fully connected graph. This dense attention mechanism can give rise particular issue: even a subtle variation, such as reducing the kernel size in layer from 5 5 to 3 x 3, can affect the representation of other layers, ultimately leading to significant differences in final representation. As result of this issue, the trained model may be biased toward fitting the train- ing data. Consequently, when the model is employed for inferring architectures outside the training data distribution, it yields inferior results and demonstrates poorer generalization performance. The corresponding experiments are presented in Sec.
Transformer grafted with GNN
Fig. A shows the vanilla transformer block, GNN block, and our proposed graph-aided attention Transformer block. As shown in Fig.1 (a), the vanilla Transformer block has two major parts:
A' =SCEAtn(LL((E--)) H'-1 +
(1)
H' = =F=FF(LL((")) + A
where H' is the feature for the layer 1. At' is an intermediate result. SelfAttn, FFN, and LN refer to self-attention, feed-forward network, and layer normalization, respectively. GNN block just has one major part, where the representation is updated following:
H' (raphAggre((H11 (AA) W
(3)
H' L2
(4)
where GraphAggre(He)) ( A) W (Normi(A ()) A E N is the adjacency matrix, and T.2 denotes the 12-normalization function. W with different superscripts and subscripts represents different learnable transformation matrices.

---

A
Abenti mulipiictionn
Elemens--siae milllpicatoon
FFN
OFFN
SayerSoo
Add
Aodk 2hkmm
Lineer
Grapveded
Adtitionn
Livar
Norm
Linerr
Linser
Linerr
Lavenfom
Figure 2: Diagrams of three modules. (a) The vanilla Transformer block B22). (b) The GNN layer
with mean aggregator [T6]. (c) The proposed graph-aided attention Transformer block. Comparing formulas (3) and (4) with formulas (1) and (2), we can observe two major differences
between the Transformer block and GNN block:
The Transformer utilizes self atffetttenion to fuse information from a global perspective, while the GNN uses graph aggregation fuse neighbor information based on the adjacency ma- trix.
The Transformer block includes additional FFN (Feed-Forward Network) component, which enhances information interaction between channels.
The advantage of self-attention lies in its data-driven structure, allowing for flexible adjustment of information fusion weights based on the input data. On the other hand, the advantage of GNN is that graph aggregation focuses the topological structure. These two advantages to each other. Consequently, we have naturally come up with an idea to combine the strengths of self-attention and graph aggregation. This approach inherits the flexibility of self-attention while benefiting from the good generalization capability of graph aggregation.
contradictory
To implement this idea, we consider the neural network encoded as graph, with the operations or layers in the network treated as nodes. Assuming the graph has nodes, the transformer layer we have designed for universal neural network representation learning (Fig. D (c)) is calculated as follows:
H' = TAEnhance ,D), it (GraphhAtt (7' A) W
(5)
(6)
H' GFFN(LN((')) At'
(7)
where D E Nx1 is a vector that records the number of nodes directly connected each node. The Graph-aided Attention (GraphAttn) module is responsible for performing attention caleulations using the properties of the graph structure to adjust global self-attention. The Type-aware Enhance- ment module (TAEnhance) is utilized to further enhance the representation. We introduce Grouped Feed-Forward Network (GFFN) by introducing group linear transformation into the original FFN. In the following sections, will provide detailed introduction to each component.
Graph-aided attention In proposed graph-aided attention, we employ the adjacency matrix to govern the attention calculation range. Moreover, the adjacency matrix characterizes the inter- layer connection relationships within the neural network, enabling the model to acquire topology knowledge. Hence, we define this module the Graph-aided Attention module:
5

---

X Sigmoid H + bbg,, S = (x'x") /Vd) の A, z (N ( 1/g
(8)
(9)
(10)
The Norm(-) means that for each node, the attention weights between it and other nodes trans- formed to (0, 1) by dividing it by the sum. The character with different superscripts and subscripts represents different learnable biases. The d refers the feature dimension of X'. Note that simply using the adjacency matrix to control the attention map in self-attention is insufficient. To make this approach work, have discarded the original softmax operation in self-attention and replaced it with a linear attention mechanism. This is because the softmax operation tends to focus excessively on the current node while neglecting neighboring nodes. Consequently, we have inserted sigmoid activation function before the linear attention to ensure that all values in the attention map are posi- tive. For further comparisons with the original self-attention, please refer the supplementary.
Type-Aware enhancement module The connection between layer and other layers is related to the type of that layer. Therefore, the number of connected layers in each layer can be used to assist the model in learning the type of layer. By fully utilizing the internal characteristics of this graph- structured data, it is beneficial to improve the learned representations. The enhanced representation obtained by:
TAEnhance(H' D) = Sig H'
(11)
3.3 Universal representation learning for neural network
In this subsection, we will present the construction a comprehensive framework for neural network encoding and representation learning based on the proposed enhanced Transformer block. We will also explain how this framework can utilized to predict network attributes. The overall system, as depicted in Fig. MI is composed of three consecutive stages: neural network encoding, representation learning, and attribute prediction.
Neural network encoding We have taken inspiration from the tokenizer used in NAR-Former [ and made certain modifications to encode the input neural network. For given network consisting of layers or operations, whether is a cell architecture or complete DNN, we represent as sequence feature comprising vectors corresponding to each layer: T (2, (N) E R In NAR-Former, for each vector of the current node, encapsulates information of operation, self- position, and the position of its source nodes: (t,",, ttcll , ..pour)
E C
Following the encoding scheme proposed in NAR-Former [. we use the position encoding for- mula [AII, [0] to transform the single real-valued numbers (e.g. operation labels and node position indices) of relevant information into a higher-dimensional space. We denote this mapping scheme as (-)
For the operation encoding L.P, there slight differences in the specific encoding content for input networks of different scales. If the input is in the form of a cell architecture [48, IOJ. there are usually more than ten different options for each architecture operation. In this case, we directly assign category labels to all candidate operations, and then use the function (-) to encode the label of the operation obtain t, However, when a complete DNN is used as input [2I]], more abundant operational information can be extracted and encoded. In this case, we first use one-hot vectors, which ensure the same distance between different categories, convolution, batch normalization, ReLU, concatenation). Then use the function fre(-) to encode the properties (e.g. kernel size, number of groups) of the operation, which then concatenated with the one-hot type vector as t.p NAR-Former [47]] uses the concatenation of ghelf and tsour to encode information flow. Since our improved transformer can obtain the topology information the network with the help of the adjacency matrix, these two parts of the encoding are not necessarily needed in our method. Details about the encoding in the different experiments are provided in our released code and supplementary material.
encode the type of operation (e.g.

---

Representation learning The model for learning neural network representations H* is con- structed by stacking multiple instances of our proposed enhanced Transformer blocks. These im- provements, specifically tailored for neural network data, allow the model to learn representations that are more meaningful and exhibit enhanced generalization capabilities.
Attributes predicting Taking the representation H* as input, the target attribute can be predicted by using the predicting head:
g =
(12)
Currently, among the various attributes of the network, accuracy and latency the two main types of predicted objects. Because they have extremely high acquisition costs and are the primary mani- festation of network performance and efficiency.
For latency prediction, due to the strong correlation between batch size, memory access, parameter quantity, and FLOPs with network latency, the encoding corresponding to these characteristics and the representation H* are input into the predicting head together. To train the whole latency pre- diction model, the mean square error (MSE) function is adopted to measure the difference between predicted results and the ground truths.
For accuracy prediction, in addition to the MSE loss function, architecture consistency loss (AC_loss) and sequence ranking related loss (SR loss) proposed by Following NAR-Former, we employed hierarchical fusion strategy in accuracy prediction exper- iments. We use a simplified approach, which computes the weighted sum of the outputs of each transformer layer with adaptive weights.
[[7]] are also used.
4 Experiments
In this section, we conduct experiments on NNLQP [2I]]. NAS-Bench-109 [88], and NAS-Bench- 201 [[TO] to evaluate the performance of our NAR-Former V2. A series of ablation experiments were performed to corroborate the effectiveness of our design details. More experiments and more details about implementation will be provided in the supplementary materials.
4.1 Implementation details
Model details For latency experiments, the number of GraphAttn-base based Transformer blocks is set to 2, which is the same the baseline [2T]. As for accuracy predicting, fix the number of Transformer blocks to to align with the standard Transformer used in the baseline [П7]].
Training details All experiments were trained using the Adam optimizer. We used linear learn- ing rate decay strategy with a warm-up, in which the learning rate uniformly increased to 0.001 (latency prediction) or 0.0001 (accuracy prediction) during the first 10% of the training steps and then gradually decayed to O. The batch size was fixed at 16. Our models are trained on a machine with GeForee RTX 3090 GPU. To avoid randomness, each model was trained 12 times and the two experiments with the best and worst indicators were discarded.
4.2 Latency prediction
We conduct latency prediction on the recently released NNLQP dataset [2.1], which comprises 20000 complete deep learning networks and their corresponding latencies on the target hardware. This dataset has 10 different types of networks (referring to the first column of Tab. M. with 2000 net- works per type. Following NNLP [2]]], we use Mean Absolute Percentage Error (MAPE) and Error Bound Accuracy (Acc(o)) to measure the deviations between latency predictions and ground truths. The lower the MAPE, the higher the prediction accuracy, while the opposite is true for Acc(o).
Here, we considered two different scenarios. In the first scenario, the training and testing sets are from the same distribution. We constructed the training set with the first 1800 samples from each of the ten network types, and the remaining 2000 networks were used as the testing set. The detailed results are shown in Tab. MI When testing with all test samples, the average MAPE of our method 0.4% lower than that of NNLP [2]]], and the average Acc(10%) is 1.16% higher than that of NNLP.

---

Table 1: Latency prediction on NNLQP 2T Training and test sets have the same distribution.
MAPEL
10c
Test Model
NAR-
NNLP [2T]
Ours
NAR-
NNLP [20]]
Ours
Former [97
avg best
avg / best
Former [+
avg / best
avg / hest
All
22.37%
3.47% / 3.44% 3.07% 3.00%
35.00%
95.25% 995.50% 96.41% 96.00%
Netx.Net
26.25%
1.37%/6.2115 6. 6.18%/5.0 5.97%
27.00%
81.75% / 84.50% 81.90% / 84.00%
EfficientNet
13.91%
3.04% / 2.82% 2.34%
2.22%
45.50%
98.00% / 97. 97.00% 98.50% 100.0%
GoogleNet
16.00%
4.18% 44.12% 3.63% /3.46%
39.00%
93.70% / 99..50% 95.95% 95.50%
MnasNet
15.76%
2.60% 2.46% 1.80%
1.70%
33.00%
97.70% / 99..00% 99.70% / 100.0%
Mob
15.19%
2.47% / 2.37% 1.83% / 1.72%
39.00%
99.30% / 19.550% 99.90% 100.0%
MobileNetV3
16.88%
3,50% / 3.43% 3.12% / 2.98%
36.00%
95.35% / 96.00% 96.75% 98.008
NasBench201
43.53%
1.46% / 1.31% 1.82% / 1.18%
55.50%
100.0% / 100.0% 100.0% 100.0%
SqueezeNet
24.33%
4.03% / 3.97% 3.54% 3.34%
23.00%
93.25% / 93.00% 95.95% 96.50%
VGG
23.64%
3.73%
/3.63% 3.51% 3.29%
26.50%
95.25% / 96.50% 95. 96.00%
ResNet
28.18%
3.34% / 3.25% 3.11% 2.89%
25.50%
98.40% / 98.50% 98.55% 99.00%
Table 2: Latency prediction on NNLQP [2T]] "Test Model = AlexNet" means that only AlexNet models are used for testing, and the data from the other 9 model families are used for training. The best results refer to the lowest MAPE and corresponding ACC (10%) in 10 independent experiments.
* obtained based on the released code without using its fine-tuning step.
FLOPs +MAC
nn-Meter [53]
TPU
BRP.
NAR-
NNLP [4]
Ours
Metric
Test Model
FLOPss
[17]]
NAS [T2]
Forner [x
(avg / best)
(avg/ / best)
AlexNet
44.65%
15.45%
7.20%
10.55%
31.68%
46.28%
10.64% 79.71%
24.28% / 18.29%
EfficientNet
58.36%
53.96%
18.93%
16.74%
51.97%
29.34%
21.46%
18.72%
13.20% / 11.37%
GoogleNet
30.76%
32.54%
11.71%
8.10%
25.48%
24.71%
13.28% 10.90%
6.61% / 6.15%
MnasNet
40.31%
96%
10.69%
11.61%
17.26%
26.70%
12.07%
10.86%
7.16% / 5.93%.
Ne
37.42%
35.27%
6.43%
12.68%
20.42%
25.74%
8.87% 7.34%%
6.73% / 5.65%
MAPE]
Net33
64.64%
57.13%
35.27%
9.97%
58.13%
33.99%
14.57% / 13.17%
9.06% / 8.72%
NasBench201
80.41%
33.52%
9.57%
58.94%
13.28%
105.71%
9.60% 8.1%
9.21% / 7.89%
ResNet
21.18%
18.91
15.58'5
20.05%
15.84%
40.37%
7,54%
7.12/
6.80% / 6.44%
SqueezeNet
29.89%
23.19%
18.69%
24.60%
42.55%
74.59%
9.84% / 9.52%
7.08% / 6.56%
VGG
69.34%
66.63%
19.47%
38.73%
30.95%
44.26%
7.60% / 7.17%
15.40% / 14.26%
Average
47.70%
37.26%
15.35%
21.20%
30.76%
45.17%
11.55%
10.27%
10.55% 9316
AlexNet
6.55%
40.50%
75.45%
57.10%
15.20%
7.60%%
59.07%
64, 40%
24.65% 28.60%
EfficientNet
0.05%
0.05%
23.40%
17.00%
0. 1.00%%
15.15%
25.37%
28. 28%
44.01% 50.20%
GoogleNet
12.75%
9.80%
47.40%
69.00%
12.55%
24.35%
36.30%
48.75%
80.10% / 83.35%
MnasNet
6,20%
9.80%
60.95%
44.65%
34.30%
20.90%
55.89%
61.25%
73 73.46/ / 81.60%
Nob
6.90%
8.05%
80.75%
33.95%
29.05%
20.70%
63.03%
45.4%% 72.50% 83.80%
Acc(10%))
MobileNetV3
0.05%
0.05%
23.45%
64.25%
13.85%
16.05%
43.26%
49.66%
68.43% / 70.50%
NasBench2011
0.009
10.55%
60.65%
2.50%
43.45%
0.005
70.649% 60.70%
63.13% / 71.70%
ResNet
26.50%
29. 99.80%
39.45%
27.30%
39.80%
13.25%
72.88%
70%
77 .7..24% 79.70%
SqueezeNet
16.10%
21.35%
36.20%
25.65%
11.85%
11.40%
58.69%
75.01% / 79.25% 40%
VGG
4.80%
2.10%
26.50%
2.60%
13.20%
11.45%
71.04%
73.75%
45 211% 445.30%
Average
7.99%
13.20%
47.4%%
34.40%
21.34%
14.09%
54.62% 60.65%
62.70% 76.40%
When tested on various types of network data separately, except for the NASBench201 family, our method consistently outperforms NNLP. This indicates that our improved transformer, which utilizes the structural characteristics of the graph, has learned more reasonable representations than the original GNN. Comparison with NAR- NAR-Former's results also proves that our NAR-Forma V2 is more effective than the original transformer-basee model in learning representations of complete DNNs.
The second scenario has more practical application significance, that is, the network type needed to be inferred is not seen during the training process. There are ten sets of experiments in this part, with each set taking one type of network as the test set, while all samples from the other nine types of networks are used as the training set. As shown in Tab. E1 it can be seen that using only FLOPs and memory access information the accumulation of kernel delays and the actual latency, kernel-based methods (TPU[IZ) and nn- Meter([5S]) perform worse than the GNNed model NNLP that directly encodes and predicts the entire network. Despite encoding the entire network directly, because of the sensitive nature introduced by the global modeling computational properties of the transformer, it may result in the transformer-based NAR-Former being less able to generalize to networks that have not been seen at the training stage. Benefiting from considering the entire input network and grafting GNN into the transformer, our method achieves the best MAPE and Acc(10%) on the average indicators of 10
predict latency is not enough. Suffering from the gap between
8

---

experimental groups. Compared with the second best method NNLP, the average Acc(10%) of our method has a marked increase 8.08%.
4.3
Accuracy prediction
4.3.1
Experiments on NAS-Bench-101
Table Accuracy prediction on NAS-Bench- 101
NAS-Bench-101 provides 423624 differ-
+
B "SE" denotes the self-evolution strategy pro-
posed by TNASP [26]].
ent cell architectures and the accuracies of the complete neural network constructed based on each cell on different datasets. Following [26]), 0.1% and 1% of the whole data are used as the training set, and another 200 samples are used for validation. We use Kendall's Tau [BGi] evaluate the correlation between the predicted sequence and the real sequence, and a higher value indicates better results.
Training Samples
1%
Backbone
Method
(424)
(424)
(4236)
Test Samples
100
all
CNN
ReNAS [H6]
0.634
0.657
0.816
NAO
0.704
0.666
0.775
LSTM
NAO+SE
0.752
0.680
0.787
NP НКЛ
0.710
0.679
0.769
GNN
NP + SF:
0.713
0.684
0.773
CTNAS
0.751
Kendall's Tau is calculated on the whole dataset or 100 testing samples. We report the average results of our predictor in 10 repeated experi- ments. Results are shown in Tab. G When only 424 samples were available for training, our method achieved the highest Kendall's Tau. We achieve 0.773 when tested using the whole testing set, which is 0.8% and 8.9% higher than the transformer- based model [[7]] and GNN-based model [[3]] respectively. This proves that the modifications we made to the transformer based on inspiration from GNN are effective.
TNASP (26]
0.752
0.705
0.820
Transformer
TNASP SE
0.754
0.722
0.820
NAR- AorForrer e8
0.801
0.765
0.871
NAR- FAR-Former V2
0.802
0.773
0.861
4.3.2 Experiments on NAS-Bench-201
NAS-Bench-201 [10]] is another cell-based dataset, which contains 15625 cell-accuracy pairs. Following [2G]], 5% and 10% of the
Table 4: Accuracy prediction on NAS-Bench-200
[[C]] "SE" denotes the self-evolution strategy pro-
posed by TNASP [26]]
whole data is used the training set and
Training Samples
other 200 samples are used for validation.
Backbone
Model
(781)
(1563)
5%
10%
We use Kendall's Tau [6 computed on the whole dataset as the evaluation metric in this part. The average results of our predictor of 10 runs are reported. Results are shown in Tab. HL The conclusion of this experiment is similar to Sec. I When compared with the second- best method, substantial improvement (2.5%) of Kendall's Tau can be seen in the setting of training with 781 samples. A more comprehensive comparison with the baseline tional metrics can be found in the supplementary materials.
NAO L
0.522
0,526
NAO + SE
0.529
0,528
NP [E
0.634
0.646
GNN
NP
0.652
TNASP [26]
0.689
0.724
Transformer
TNASP SE
0.690
0.726
NAR NAR-Former [+]
0.849
0.901
NAR-Foree
0.874
using addi-
Regarding the average Kendall's Tau under different training settings, our method is par with NAR- WAR-Former with 0.812 on nas-bench- and slightly higher than NAR-former by 0.6% on NAS- Bench-201. Therefore, compared to
V2 achieves comparable accuracy prediction performance. In latency prediction experiments, our proposed model exhibits a clear advantage NNLP and outperforms porating the strengths of GNN, the universal representation learning frumework NAR-Former V2 significantly enhanced. NAR-Former V2 addresses the shortcomings of NAR-Former, which was overly sensitive when handling complete network structures, while still retaining the outstanding when handling structrree networks.
by a significant margin. In summary. by incor-
performance of
4.4 Ablation studies
In this section, we conducted a series of ablation experiments the NNLQP dataset investigate the impact of various modifications. The results from Rows (2) and (3) in Table 5 indicate that for type encoding without numerical relationships, using one-hot vectors with equidistant proper-

---

Table 5: Ablation studies on NNLQP [.]] "PE" denotes position encoding.
Structure
Type
Op
Attibutes
Op
Graph- Atm
GFPN
TA- Enhance
MAPEL
Acx(10%))
Acc(5
((Baseline)
One-hot
Real Num
3.48
95.26
77.80
GNN
PE
PE
3.43( 0.05)
95.11(-0.15)
79.58(+1.78)
GNN
One-hot
PE
3.33(-0.15)
95.57(=0.31)
80.19(+..39)
Transforner
Onc-hot
PE
3.201-0.28)
96.00(
81.86(-4.000
Transformer
One-hot
PE
4
3.20(-0.28)
96.06(-0.80)
81.76(+3.96)
Transformes
Oue-hot
PE
/
3.070-041)
96.41(-1.15)
82.71(+-491)
ties across different categories is more suitable. Comparing Row (3) in Table G with Row (4), observe that introducing GNN characteristies into the Transformer improves the model's ability to learn effective representations and achieve more accurate predictions compared to using the original GNN. When replacing the FFN with the GFFN module with eight groups (Row (5)), the number of model parameters reduces to approximately one-eighth of that in Row (4), without a significant de- crease in prediction accuracy. Compared to Row (5), Row (6) demonstrates an increase of 0.35% in ACC(10%) and 0.95% ACC(5%) This confirms the role of the type-aware enhancement module in further refining and enbancing the rationality of the representations.
To verify our hypothesis regarding the gener- alization ability of the network and the effec- tiveness of the proposed graph-aided attention, we conducted comparative experiments in sce- narios where the training and testing data have
Table 6: The influence of using different atten- tions. Test on EfficientNet family.
Attention
MAPE]
ACC(10%)
Global
16.88%
36.32%
Local
13.20%
44.01
different distributions. The results of these ex- periments are presented in Table B. In order to perform the experiment on global attention, we excluded the step of multiplying the adjacency matrix A in Equation E. and instead replaced S' with X X /vd. Results in Table B demonstrate that incorporating the adjacency matrix to restrict the scope of attention calculation indeed beneficial for latency prediction on unseen data. The model utilizing graph-aided attention exhibited significant improvement of 7.68% in ACC(10%) compared to the model using global attention.
Conclusion
In this paper, combine the strengths of Transformer and GNN to develop universal neural network representation learning model. This model is capable of effectively processing models of varying scales, ranging from several layers to hundreds of layers. Our proposed model addresses the limitations of previous Transformer-based methods, which exhibited excessive sensitivity when dealing with complete network structures. However, it still maintains exceptional performance when handling cell-structured networks. In future work, we will focus on optimizing the design of the representation learning framework and applying it broader range of practical applications. Such as using the proposed model to search for the best mixed precision model inference strategies.
Acknowledgement
This work was supported in part by the National Key Research and Development Program of China under Grant 2018AAA0103202; in part by the National Natural Science Foundation of China un- der Grants U22A2096 and 62036007: in part by the Technology Innovation Leading Program of Shaanxi under Grant 2022QFY01-15: in part by Open Research Projects of Zhejiang Lab under Grant 2021 KGOAB01 and in part by the Fundamental Research Funds for the Central Universities under Grant QTZX23042.
10

---

References
[1] Jie-Neng Chen, Shuyang Sun, Ju He, Philip HS Torr, Alan Yuille, and Song Bai. Transmix: Attend to mix for vision transformers. Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pages 12135-12144, 2022.
[2] Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Haichen Shen, Meghan Cowan, Leyuan Wang, Yuwei Hu, Luis Ceze, al. (TVM): An automated (End-to-End) optimizing compiler for deep learning. 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI
18). pages 578-594, 2018.
[3] Yaofo Chen, Yong Guo, Qi Chen, Minli Li, Wei Zeng. Yaowei Wang, and Mingkui Tan. Contrastive neural architecture search with neural architecture comparators. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 9502-9511, 2021.
[4] Ziye Chen, Yibing Zhan, Baosheng Yu, Mingming Gong, and Bo Du. Not all operations contribute equally: Hierarchical operation-adaptive predictor for neural architecture search. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 10508-10517, 2021.
[5] Ciprian Corneanu, Sergio Escalera, and Alcix M Martinez. Computing the testing error without a testing set. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2677-2685, 2020.
[6] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoe Le, and Ruslan Salakhutdi- nov. Transformer-xt xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:191.002800 2019.
Boyang Deng, Junjie Yan, and Dahua Lin. Peephole: Predicting network performance before training. arXiv preprint arXiv: 1712.0331,, 2017.
[8] Jacob Devlin, Ming-Wai Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre- Preitraining of deep bidirec- tional transformers for language understanding. arXiv preprint 1rx.vv1810004805, 2018
[9] Xiaohan Ding, Xiangyu Zhang. Jungong Han, and Guiguang Ding. Scaling up your kemels to 31x31: Revisiting large kernel design in cnns. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 11963-111 11975, 2022.
[10] Xuanyi Dong and Yi Yang. Nas-bench-201: Extending the scope of reproducible neural architecture search. arXiv preprint 20X1:20001.00326 2020.
[11] Alexey Dasovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner. Mostafa Dehghani. Matthias Minderer, Georg Heigold. Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recognition scale. arXiv preprint arXiv:2010.1 11929,, 2020.
[12] Lukasz Dudziak, Thomas Chau, Mohamed Abdelfattah, Royson Lee, Hyeji Kim, and Nicholas Lane. Brp- nas: Prediction-haseee nas using gens. Advances in Neural Information Processing Systems, 33:10480- 10490, 2020.
[13] Stéphane dAscoli, Hugo Touvron, Matthew I. Leavitt, Ari S Morcos, Giulio Biroli, and Levent Sagun. Convit: Improving vision transformers with soft convolutional inductive biases. In International Confer- ence on Machine Learning, pages 2286-2296. PMLR, 2021.
[14] Anmol Gulati, James Qin, Chung -hheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang. Zhengdong Zhang, Yonghui Wu, et al. Conformer: Convolution-angmentte transformer for speech recog- nition. arXiv preprint 2rXiv:2005.08100 2020.
[15] Jianyuan Guo, Kai Han, Han Wu, Yehui Tang, Xinghao Chen, Yunhe Wang, and Chang Xu. Cmt: Con- volutional neural networks meet vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12175-12185, 2022.
[16] Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Ad- vances in neural information processing systems, 30, 2017.
[17] Sam Kaufman, Phitchaya Phothilimthana, Yanqi Zhou, Charith Mendis, Sudip Roy, Amit Sabne, and Mike Burrows. A learned performance model for tensor processing units. Proceedings of Machine Learning and Systems, 3:387-400, 2021.
[18] Jiashi Li, Xin Xia, Wei Li, Huixia Li, Xing Wang, Xuefeng Xiao, Rui Wang Min Zheng, and Xin Pan. Next-vit: Next generation vision transformer for efficient deployment in realistic industrial scenarios. arXiv preprint arXiv:2207.05501 2022.
11

---

[19] Wei Li, Shaogang Gong, and Xiatian Zhu. Neural graph embedding for neural architecture search. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pages 4707-4714, 2020.
[20] Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In Proceedings of the Euro- pean conference on computer vision (ECCV), pages 19-34, 2018.
[21] Liang Liu, Mingzhu Shen, Ruihao Gong, Fengwei Yu, and Hailong Yang. Nnlqp: A multi-phan pulti-platform neural network latency query and prediction system with an evolving database. In 51 International Conference on Parallel Processing ICPP, ICPP '22. Association for Computing Machinery, 2022.
[22] 7e Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao. Zheng Zhang, Li Dong, et al. Swin transformer v2: Scaling up capacity and resolution. In Proceedings of the IEFF/CVF conference on compuder vision and paltern recognition, pages I 12009-12019, 2022.
[23] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang. Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012-10022, 2021.
[24] Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer. In Proceedings of the IEEE/CVE conference on computer vision and puitern recognition, pages 3202- 3211, 2022.
[25] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. convnet for the 2020s. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 1 11966-1 1986, 2022.
[26] Shun Lu, Jixiang Li, Jianchao Tan, Sen Yang. and Ji Liu. Tnasp: A transformer-based nas predictor with self-evolution framework. Advances in Neural Information Processing Systems, 34:15125 15137, 2021.
[27] Renqian Luo, Fei Tian, Tao Qin, Enhong Chen, and Tie-Yan Liu. Neural architecture optimization. Ad- vances in neural information processing systems, 31, 2018.
[28] Sachin Mehta, Marjan Ghazvininejad, Srinivasan Iyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. De- light: Deep and light- weight transformer. arXiv preprint arXiv:2008. 00623, 2020.
[29] Sachin Mehta and Mohammad Rastegari. Mobilevit: light-weight, general-puroose, and mobile-friendly vision transformer. arXiv preprint arXiv:211000111 2021.
[30] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. Communications of the ACM, 65(1):99- 10099-0022220 2021.
[31] Xuefei Ning, Yin Zheng, Tianchen Zhao, Yu Wang, and Huazhong Yang. A generic graph-based neural architecture encoding scheme for predictor-based nas. In European Conference Computer Vision, pages 189-204. Springer, 2020.
[32] Jiezhong Qiu, Jian Tang, Hao Ma. Yuxiao Dong, Kuansan Wang, and Jie Tang. Deepinf: Social influence prediction with deep learning. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pages 21 10-2119, 2018
[33] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485 5551, 2020.
[34] Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsich. Dynamievit: Effi- cient vision transformers with dynamic token sparsification. Advances in neural information processing systems, 34:13977-13999 2021.
[35] Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini, The graph neural network model. IEEE transactions on neural networks, 20(1):61 180, 2008.
[36] Pranab Kumar Sen. Estimates of the regression coefficient based on kendall's tau. Journal of the American statistical association, 137922)) 1968.
[37] Han Shi, Renjie Pi, Hang Xu, Zhenguo Li, James Kwok, and Tong Zhang. Bridging the gap between sample-based and shot neural architecture search with bonas. Advances in Neural Information Pro- cessing Systems, 33:1808 1819, 2020.
12

---

[38] Aravind Srinivas, Tsung-Y Yi Lin, Niki Parmar, Jonathon Shiens, Pieter Abbeel, and Ashish Vaswani. Bot- tleneck transformers for visual recognition. In Proceedings of the IEEE/CVF conference on computer vision and puttern recognition, pages 16519-16529, 2021.
[39] Hugo Touvron, Matthieu Cord, Alexandre Sablaynolles, Gahriel Synnaeve, and Hervé Jégou. Going deeper with image transformers, In Proceedings of the IEEF/CVF international conference on computer vision, pages 32-42, 2021.
[40] Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Maxvit: Multi-axis vision transformer. In European conference on computer vision, pages 459-479 479. Springer, 2022.
[41] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[42] Qiang Wang. Bei Li. Tong Xiao, Jingbo Zhu, Changliang Li, Derek Wong, and Lidia Chao. Learning deep transformer models for machine translation. arXiv preprint
2019.
[43] Wei Wen, Hanxiao Liu, Yiran Chen, Hai Li, Gabriel Bender, and Pieter- Kindermans. Neural predictor for neural architecture search. In European Conference on Computer Vision, pages 660-676. Springer, 2020.
[44] Colin White, Willie Neiswanger, and Yash Savani. Bananas: Bayesian optimization with neural archi- tectures for neural architecture search. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 10293 10001, 2021.
[45] Ruihan Xu, Haokui Zhang, Wenze Hu, Shiliang Zhang, and Xiaoyu Wang. Parcnetv2: Oversized kernel with enhanced attention. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1752-5762. 2023.
[46] Yixing Xu, Yunhe Wang, Kai Han, Ychui Tang, Shangling Jui, Chunjing Xu, and Chang Xu. Renas: Relativistic evaluation of neural architecture search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 4411-4420, 2021.
[47] Yun Yi, Haokui Zhang, Wenze Hu, Nannan Wang, and Xiaoyu Wang. Nar-former: Neural architecture representation learning towards holistic attributes prediction. arXiv preprint arXiv:221 1.0004,, 2022.
[48] Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter. Nas-bench- 101: Towards reproducible neural architecture search. In International Conference Machine Learning, pages 7105-7114. PMLR, 2019.
[49] Rex Ying, Ruining He. Kaifeng Chen, Pong Eksombatchai, William L Hamilton, and Jure Leskovec. Graph convolutional neural networks for web-scale recommender systems. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pages 974-983, 2018.
[50] Jiaxuan You, Jure Leskovee, Kaiming He, and Saining Xie. Graph structure of neural networks. In International Conference on Machine Learning, pages 10881-10891. PMLR, 2020.
[51] Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. Metaformer is actually what you need for vision. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages
2022.
[52] Haokui Zhang, Wenze Hu, and Xiaoyu Wang, Pare-net: Position aware circular convolution with merits from convnets and transformer. In European Conference on Computer Vision, pages 613-630. Springer, 2022.
[53] Haokui Zhang, Wenze Hu, and Xiaoyu Wang- Feaformer: Forward cross attention in hybrid vision transformer. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 6060- 6069, 2023.
[54] Haokui Zhang, Buzhou Tang, Wenze Hu, and Xiaoyu Wang. Connecting compression spaces with trans- former for approximate nearest neighbor search. Furopean Conference on Computer Vision, pages 515-530. Springer, 2022.
[55] Li Lyna Zhang, Shihao Han, Jianyu Wei, Ningxin Zheng, Ting Cao. Yuqing Yang, and Yunxin Liu. nn- meter: towards accurate latency prediction of deep- learning model inference on diverse edge devices. In Proceedings of the 19th Annual International Conference on Mobile Systems, Applications, and Services. pages 81-93, 2021.
13


Reviews:
[{"review_id": "fjXg5dYwMr", "reviewer": "Reviewer_9nHL", "summary": "The paper proposes a transformer based neural network representation learning model that incorporates the inductive representation capabilities of graph neural networks into the transformer architecture to improve it's generalizability on unseen architectures. The NAR-Former framework, which the proposed model builds on, has poor generalization performance when the depth of the input architecture gets large(eg: in the hundreds). The authors argue that for a neural network representation model to process neural architectures in varying tasks, a combination of both the Transformer and GNN architectures is necessary. The resulting model, NAR-Former V2, is a combination of the self-attention mechanism of transformers which discovers pair-wise correlations between its inputs and the graph aggregation mechanism of graph neural networks which is capable of discovering topological structures in its inputs.", "questions": "- Lines[176-178] How does the Type-aware Enhancement module actually 'enhance' the representation? What does enhance mean here? Was there a comparison with the representations of NAR-Former's representations? The usage of the term seems a bit loose and i'm not sure how to intepret this. If this is by some metric? Then it is best to provide this to the reader.\n\n- In Tables 3 and 4 where NAR-Former is compared with NAR-FormerV2, there seems to be no difference between the two models when averaged over all the test samples. Specifically, averaging the results in Table 3 gives 0.82 for both NAR-Former and the proposed NAR-FormerV2. Similarly for Table 4, taking an average over all testing settings gives 0.875 and 0.881 for NAR-Former and NAR-FormerV2 respectively. Since the proposed method is supposed to be a direct enhancement of NAR-Former, do these results still justify the modifications made to the original model?\n\n- Since the proposed model is a direct improvement on NAR-Former, why isn't there a comparison with NAR-Former in Tables 1 and 2?\n\n- What is the parameter count of NAR-Former V2 compared to NAR-Former? While the latency measurements on an RTX 3090 GPU is provided in Table 2, the actual parameter count could be useful. \n\n- In the ablation study (Table 5), what happens when:\n    - Only GFFN is incorporated in the Transformer.\n    - Only TA-Enhance is incorporated in the Transformer.\n    - A combination of only GraphAtnn and TA-Enhance only.\n\n  Table 5 suggests than combining all 3 leads to the best performance but the ablation ignores the combinations listed above. For instance it might be the case that combining either TA-Enhance or GFFN with the regular transformer based attention performs better than the combinations with the GraphAttn.\n\n- In the Appendix(Lines 16-18), again claims that the inadequacy of Eq. 5 are directed at the usage of the softmax function with a description of why this is the case together with how Eq. 2 restricts attention to connected nodes only. Could the attention matrix and adjacency matrix be visualized to actually confirm if the models behave as the authors say? This would be a much better quantitative illustration of the benefits of the proposed model over regular Transformer based attention.\n\n- [Minor] Typo on line 59.", "limitations": "No obvious limitations.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The proposed method can handle cell-structured networks and can learn representations of entire networks extending the capabilites of NAR-Former. \n\n- The method is extensively benchmarked on various tasks such as attribute and latency prediction as well as on varaious NAS benchmarks.\n\n- The proposed method is scalable and can encode networks with hundreds of operations. \n\n- The paper is easy to read and follow.", "weaknesses": "- There are many claims made in the paper which, perhaps made by intution, are not supported quantitatively (See questions).\n\n- Comparison with NAR-Former is missing for some crucial results used to validate the proposed method which is a direct modification to NAR-Former.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the valuable questions and suggestions you've given, and here are our responses.\n\n> 1. More explanation for the proposed Type-aware Enhancement module\n\nThe ``Enhancement\" in the Type-Aware Enhancement module refers to the use of intrinsic information of the input data to modify the hidden representations to make them more sensible and relevant to the structure of the input architecture. Since it is observed that the number of connections for each type of operation in a neural network contains certain patterns, e.g. the convolutional layer has only one input, while the addition operation consists of more than one input. Therefore, for each node, we take as input the number of nodes it is connected to, and obtain a modulation parameter to adjust the representation, in order to make them more relevant to the corresponding type of operation. The ablation experiments verified the rationality and validity of our design. \n\n> 2. The advantages in Tables 3 and 4 are not significant\n\nThe main contribution of this paper is to address the poor generalization performance of NAR-Former on encoding complete deep neural networks while maintaining its state-of-the-art results on encoding cell structures. The results in Tables 3 and 4 indicate that our method can indeed maintain the outstanding performance of accuracy prediction on cell structures. However, more importantly, as shown in Table 3 of the appendix, we get a significant boost in encoding complete DNNs compared to NAR-Former and exceed the SOTA (NNLP) in latency prediction. Thus, overall, the method in this paper is an improved version of NAR-Former, which can reach the SOTA in two vastly different scenarios, that is, encoding cell and encoding complete DNNs.\n\n> 3. The direct comparison between Nar-Former v1 and Nar-Former v2\n\nWe apologize for any confusion caused by placing the direct comparative experiments in the supplementary material due to time and space constraints. The reviewer rightfully pointed out the absence of these direct comparison experiments. In response, we acknowledge this oversight and are committed to rectifying it in the revised version by incorporating the direct comparative experiments into the main body of the paper.\n\nTables 1 and 3 in the supplementary material provide a direct comparison of the performance of Nar-Former v1 and Nar-Former v2 in terms of network architecture search and latency prediction. The experimental results consistently demonstrate that Nar-Former v2 outperforms Nar-Former v1 in these aspects.\n\n> 4. Comparison of parameter counts\n\nA more comprehensive comparison with NAR-Former on NAS-bench-101 is as follows:\n\n|               | Params(M) | Latency(ms) |\n|---------------|-----------|-------------|\n| NAR-Former    | 4.7850    | 17.4876     |\n| NAR-Former V2 | 8.5825    | 5.2276      |\n\nResulting from the fact that (1) our network has a larger width (512) than NAR-Former (192), and (2) in contrast to NAR-Former's transformer-based multi-stage fusion mechanism, we use the a simplified but  parameter-heavy fully-connected layer-based fusion scheme, our model in this setting has more parameters than NAR-Former. Even with more parameters, the simplicity of the computation of the fully-connected layer allows us to still achieve faster inference.\n\n> 5. Additional ablation experiments on 1) combining GFFN, TA-Enhance with Transformer 2) combining Graph Atnn and TA-Enhance. \n\nThe results of the ablation experiments for the several scenarios you are concerned about (Row (1), (2), and (3)) are shown in the table below:\n\n| Row           | Structure   | GraphAttn | GFFN     | TA-Enhance | MAPE$\\downarrow$  | ACC(10%)$\\uparrow$ | ACC(5%)$\\uparrow$ |\n|---------------|-------------|-----------|----------|------------|-------|----------|---------|\n| (a1)          | Transformer | &#10004;  | &#10004; | -          | 3.20  | 96.00    | 81.86   |\n| (1)           | Transformer | -         | &#10004; | -          | 10.24 | 60.62    | 36.91   |\n| (2)           | Transformer | -         | -        | &#10004;   | 9.16  | 64.96    | 40.05   |\n| (3)           | Transformer | &#10004;  | -        | &#10004;   | 3.09  | 96.15    | 82.76   |\n| NAR-Former V2 | Transformer | &#10004;  | &#10004; | &#10004;   | 3.07  | 96.41    | 82.71   |\n\n As we can see from the Rows (a1) and (1), combining GFFN with the regular transformer (with global attention) performs much worse than the combinations with the GraphAttn.  Comparing the results in Rows (2) and (3) reveals that a similar conclusion holds for TA-Enhance. These again validates what we described in the paper that global attention may cause excessive sensitivity when encoding complete DNNs, learning too many non-general features from the training set data and thus showing poor generalization ability on the test set.\n\n> 6. Visualizing attention maps to validate our propositions \n\nFor the visualization of attention map and adjacency matrix, please refer to figure of the PDF. Subplot (a) of Fig. 1 is the raw attention graph that has not yet been normalized. We analyze the 15th node as an example (line 15 marked by the red box). We can draw conclusions from the Fig. 1: (1) The feature of softmax makes it overly focused, with most of the other nodes, which include neighboring nodes, being ignored and given very similar weights. This can be seen from the color distribution of the heat map in the red box in subplot (c). (2)  This neglect and equal weighting leads to  having similar attention to different neighbors in the softmax GraphAttn, which will likely cause the model to degenerate into the GNN aggregation rule. The linear GraphAttn avoids this problem. This can be seen in subfigures (d) and (f)."}, {"review_id": "V2T8EBrdtL", "reviewer": "Reviewer_L4c1", "summary": "This submission builds on previous work and proposes NAR-Former V2, an updated transformer architecture to learn representations of neural network architectures. NAR-Former V2 targets NAS tasks, predicting model properties like performance or latency, to improve NAS runtime and cost. NAR-Former V2 makes use of a proposed graph-aided attention block, which uses the network's adjacency matrix to condition the attention matrix. The proposed NAR-Former V2 is empirically evaluated on NASBench101 and NASBench201 where it achieves comparable performance to state-of-the art models.\n\n", "questions": "Question 1: Have you encountered any kind of ‘receptive field’ effect? Conditioning on the adjacency matrix should lead to several hops being necessary to aggregate information globally, unless I'm mistaken? What is your intuition into why that doesn’t seem to be an issue.\n\nQuestion 2: Have you investigated other ways of encoding graph connectivity in transformers, e.g. like Kim et al, 2022: Pure Transformers are Powerful Graph Learners?\n\n", "limitations": "Not discussed explicitly.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The submission is generally well written. It identifies a specific problem,latency and accuracy prediction for NAS, for which separate approaches have so far been suitable. The proposed method bridges that gap as a unified property predictor that covers both aspects.\n\nThe idea of using network connectivity by conditioning on the adjacency matrix of the network's graph is well motivated and convincing.\nThe empirical evaluation follows previous work and is thorough. The method achieves its goal of closing the performance gap between NNLP and NAR Former on latency and accuracy prediction.\n\n", "weaknesses": "While I find the submission generally of high quality, there are a few weaknesses.\n\nWhile it may not have been the goal for this work, there are only small improvements over NAR-Formver v1, particularly on NASbench family. On these results (Table 3 and 4) there are also only average performance values reported. I'd ask the authors to add standard deviations or similar metrics, since the performance differences are very small.\nFurther, the submission shows different baselines for different experiments. While some numbers are mentioned in the text, I would find it more convincing to have the two main baselines the authors identified (NNLP and NAR-Former) in all results.\n\nAlso, the authors propose their method for universal neural network representation learning. However, they only show results on latency prediction and accuracy, which are important but relatively coarse labels. NasBench201 at least contains more detailed information on each model, which may be used to assess the quality of the representation on a more fine-grained level.\n\nLastly, there is an entire field of work that attempts to learn representations of neural networks, for example to predict model properties. While many of them use the actual weights, some use graph-based features. E.g.,\n- You et al, 2020: Graph Structure of Neural Networks\n- Corneanu et al, 2020: Computing the Testing Error Without a Testing Set.\nThey may merit discussion in the related work and may be used as different baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the many detailed and insightful questions you have raised. Below are our responses.\n\n> 1. add standard deviations or other metrics in Table 3 and Table 4\n\nThanks for your suggestion. Indeed, this is a good way to evaluate two methods. Compared with Nar-Former, Nar-Former V2 exhibits greater stability. For the accuracy prediction experiments on NAS-Bench-201, when employing a more stringent error bound, the advantages of Nar-Former v2 can also be demonstrated. We report more results using the 10% error bound accuracy metric (ACC(10%)) for further comparison with NAR-Former. The results are as follow:\n\n| Train/test    | 781/all | 1563/all |\n|---------------|---------|----------|\n| NAR-Former    | 96.77   | 99.32    |\n| NAR-Former V2 | 99.45   | 99.50    |\n\nThese results will be included in the revised version.\n\n> 2. Utilize Nar-Former and NNLP as baseline models across all experiments.\n\nNar-Former has already been validated in the accuracy prediction experiments and latency prediction experiments (as presented in the supplementary material). Here, we provide additional experiments regarding NNLP's performance in accuracy prediction. The experimental results are as follows:\n| Dataset    | 101     | 101     | 101      | 201     | 201     |\n|------------|---------|---------|----------|---------|---------|\n| Trian/Test | 424/100 | 424/all | 4236/all | 781/all | 781/all |\n| KT of NNLP | 0.497   | 0.484   | 0.563    | 0.675   | 0.703   |\n\nThe table ablove shows the accuracy prediction results of NNLP on the NAS-Bench-101 and NAS-Bench-201, which are significantly worse than the performance of NAR-Former.  As described in our introduction, Transformer models currently hold an advantage in accuracy prediction tasks.  Therefore, the NAR-Former is utilized as the baseline for accuracy prediction. For latency prediction, the results in Table 1, Table 2 of the main text and Table 3 of supplementary material demonstrate that NNLP is the SOTA model, which is then adopted as the baseline for this task.\n\n>3. Predicting more attributes with the proposed NarFormer-V2 on NasBench201\n\nIn this paper, we show the results of applying our method to predict accuracy and latency, two of the most frequently considered attributes in network design and deployment. Actually, our approach can also be easily used for other attribute predictions of neural networks, simply by changing labels and adjusting hyper parameters as needed. We use our NAR-Former V2 with the same hyper parameters as the experiments on NASBench family to predict the testing loss. The results are shown in the table below:\n\n| Metrics       | Kendall's Tau$\\uparrow$ | ACC(10%)$\\uparrow$ | ACC(5%)$\\uparrow$ | MAPE$\\downarrow$ |\n|---------------|---------------|----------|---------|------|\n| NAR-Former V2 | 0.851         | 96.01    | 77.70   | 3.48 |\n\nThe Kendall's Tau is 0.851, indicating that our model is capable of predicting the relative ordering of the test loss for the entire dataset (15625 samples) with high correlation. The value of the other three metrics MAPE, ACC(10\\%), and ACC(5\\%) demonstrate the high accuracy of our method in predicting the actual values of loss for all networks in the dataset. In the revised version, we will incorporate additional experimental results.\n\n> 4. Some references \n\nThanks for the reminders. We provide an introduction to some common and classical neural network representation learning methods in the first part of the related work section. Regrettably, maybe it is not comprehensive enough. We will add some discussions later according to your suggestion.\n\n> 5.The receptive field effect\n\nYes, we have encountered some receptive field effect. Each layer of our graph-aided attention can only observe neighboring nodes. As the graph-aided attention-based transformer stack up, the receptive field gradually expands. Consistent with existing experience, when containing very few attention layers, e.g., only 1 layer, the receptive field is too small, leading to poor performance. However, in the task of this paper, there are some differences from the common sense that the larger the receptive field the better the result usually is. When the receptive field is too large, such as when using the vanilla global attention-based transformer model, the encoding of neural networks with depths up to hundreds of layers tends to unsatisfactory. \n\n> 6. Other ways of encoding graph connectivity in transformers\n\nWhile there are indeed some other works to encode graphical connectivity in the Transformer [1, 2, 3], we are different from them in terms of what each focus on, motivation, the details of design, etc. \n\nFor example, the TokenGT [1] mainly proposes to build graph learners directly using pure transformers by treating all nodes and edges as tokens in order to exploit the advantages of transformer in conjunction with large-scale training. But instead, we focus on modifying the transformer using GNN's inductive bias to improve its generalization ability and performance on representation acquisition and attribute prediction of very deep neural networks. In order to learn the representation of graph-structured data, some methods also allow nodes to attends only to their local node neighbors in the attention computation of Transformers, but our motivation and goals are different from theirs [2]. \n\nIn addition, we are the first attempt to analyze the impact of graph connectivity properties and global connectivity properties in the transformer on the task of neural network encoding and representation learning. It can be observed from the extensive experiments in the main text, supplementary material and rebuttal PDF that our analysis is sound and our modifications are effective.\n\n**Reference**\n\n[1] Kim et al, 2022, Pure Transformers are Powerful Graph Learners.\n\n[2] Dwivedi et al, 2021, A Generalization of Transformer Networks to Graphs.\n\n[3] Ying et al, 2021, Do Transformers Really Perform Bad for Graph Representation?"}, {"review_id": "GGPXcaMwPQ", "reviewer": "Reviewer_kdkx", "summary": "This paper proposes to design graph-aided attention Transformer blocks that can be used to model and learn the representation of neural networks. Specifically, one can leverage this kind of representation learning framework to provide more accurate latency and accuracy predictions of given deep neural networks.\n\nThe proposed method surpasses the GNN-based NNLP baseline in latency estimation on the NNLQP dataset, and improves accuracy prediction on the NASBench101 and NASBench201 datasets.", "questions": "See weaknesses.", "limitations": "N/A. Not provided in the main context.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* Leveraging graph topology to help the attention networks seems new to me. And the target problem of how to predict latency or accuracy is of great significance to the NAS and representation learning community.\n\n* Results indeed show better predictions for latency and accuracy as compared to baselines.", "weaknesses": "* The author does not provide a context for what \"NAR\" refers to. Also, the writer assumes all the audience is familiar with NAR-Fromer, which is not the case. Most audiences are still new to this field so better to make it more accessible to a general audience.\n\n* The paper organization could be better, it starts with quite some details telling us what is Graph-aided attentions. But why should we care about it? I would suggest moving the Fig. 2 to the front, so the readers know what is going on and what module exactly are we talking about. Without this big picture, one can easily get lost and lose interest in this work.\n\n* How is the proposed representation learning methods surpass other network selection algorithms, like general NAS algorithms as they are also designed with the philosophy of predicting accuracy given networks? In terms of both prediction accuracy and computational costs.\n\n* I would like to know the efficiency of the proposed techniques as compared to pure attention-based or pure graph-based variants.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for your review and detailed sugession. Next, we will endeavor to address your concerns point by point.\n\n> 1. Provide a context for what \"NAR\" refers to.\n\nThank you very much for your suggestions; this was an oversight on our part. We should consider more for the audiences who are new to this field.  In the introduction part, the definition of the task and a brief introduction has been added in the revised version.\n\n> 2. Reorganizing the structure of the article\n\nThank you again for your suggestions for the details of our article organization. We have given this issue serious consideration. We agree with you that placing Figure 2 up front might give the reader a quicker understanding and a clearer idea of the role of the modules we are describing (graph-aided attention) in the overall model that corresponds to the task. New origination has been adopted in the revised version.  \n\n>3. Comparison with NAS algorithms.\n\nTable 1 in the supplementary material presents a comparison of NAS experiments. The experimental results demonstrate that the proposed Nar-Former v2 assists NAS algorithms in discovering superior architectures, characterized by higher accuracy and fewer parameters compared to architectures identified by other NAS algorithms.\n\n>4.  The efficiency of the proposed techniques as compared to pure attention-based or pure graph-based variants.\n\nWe complement the latency information for different backbone-based methods on NAS-Bench-101 in the table below:\n\n|               |                 |             |  424/100      |  424/all     |  4236/all      |\n|---------------|-----------------|-------------|---------------|---------------|---------------|\n|               | Structure       | Latency(ms) | Kendall's Tau | Kendall's Tau | Kendall's Tau |\n| NNLP          | graph-based     | 1.4625            | 0.497         | 0.484         | 0.563         |\n| TNASP         | attention-based | 4.6623      | 0.754         | 0.722         | 0.820         |\n| NAR-Former    | attention-based | 17.4864     | 0.801         | 0.765         | 0.871         |\n| NAR-Former V2 | GraphAttn-based | 5.2276      | 0.802         | 0.773         | 0.861         |\n\nThe \"412/100\" indicates that the number of training and test samples are 424 and 100 respectively. As can be seen from the table, the pure graph-based approach NNLP has the fastest inference speed due to the simplicity of the structure and computation. However, it suffers from the very poor performance of accuracy prediction. Although NAR-Former can achieve kendall's tau results that are as advanced as our method, its quite longer inference time makes it less efficient. Compared to the pure attention-based TNASP model, we have the same level of latency (only 0.6ms more), but achieve a significant improvement in prediction accuracy (4.7% on average).\n\n\n"}, {"review_id": "dbcURFna3v", "reviewer": "Reviewer_sJnp", "summary": "This paper studies how to model and learn the representations of neural networks themselves. Inspired by Transformer and GNN, NAR-Former V2 is proposed which is a modified Transformer-based universal neural network representation learning model. Extensive experiments demonstrate its effectiveness.", "questions": "Please see weaknesses.", "limitations": "The authors have adequately addressed the limitations", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper is well-motivated. It is reasonable to incorporate graph-specific properties into the vanilla Transformer. A new graph-aided attention-based Transformer block is proposed. The novelty of this paper is good.\n- Compared to the baseline NAR-Former, NAR-Former V2 improves latency prediction performance significantly. And the authors conduct experiments on the recently released benchmark. Hence, NAR-Former V2 can be considered as a new baseline in this field.\n- The whole paper enjoys well writing and good organization.", "weaknesses": "- The latency of the proposed NAR-Former V2 is not considered. It is better to make a comparison of the model latency.\n- In lines 331-338, please report the specific number when comparing NAR-Former V2 with NAR-Former.\n- In Tab. 2, NAR-Former V2 suffers from poor performance on AlexNet and VGG. Both of them have no residual connections. Why does NAR-Former V2 perform badly on these networks?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive comments. The responses to your questions is as follows.\n\n>1. The latency of the proposed NAR-Former V2.\n\nThe latency of the proposed NAR-Former V2 is presented in Table 3 of the supplementary.  More latency comparison experiments are provided as follows:\n|               |                 |             |  424/100      |  424/all     |  4236/all      |\n|---------------|-----------------|-------------|---------------|---------------|---------------|\n|               | Structure       | Latency(ms) | Kendall's Tau | Kendall's Tau | Kendall's Tau |\n| NNLP          | graph-based     |   1.4625      | 0.497         | 0.484         | 0.563         |\n| TNASP         | attention-based | 4.6623      | 0.754         | 0.722         | 0.820         |\n| NAR-Former    | attention-based | 17.4864     | 0.801         | 0.765         | 0.871         |\n| NAR-Former V2 | GraphAttn-based | 5.2276      | 0.802         | 0.773         | 0.861         |\n\nThe \"412/100\" notation signifies that there are 424 training samples and 100 test samples. As indicated in the table, the NNLP model, based solely on graphs, demonstrates the fastest inference speed due to its streamlined structure and computations. However, it faces challenges in accurately predicting outcomes. While the NAR-Former achieves comparable Kendall's tau results to our approach, its notably longer inference time hampers its efficiency. In contrast, in comparison to the exclusively attention-based TNASP model, our method maintains a similar latency level  while significantly enhancing prediction accuracy.\n\n>2. Report the specific number in lines 331-338\n\nThank you for your suggestion. We have revised lines 331-338 as follows based on your suggestion: Regarding the average Kendall's Tau under different training settings, our method is on par with NAR-Former with 0.812 on nas-bench-101 and slightly higher than NAR-former by 0.6\\% on NAS-Bench-201. Therefore, compared to NAR-Former, NAR-Former V2 achieves comparable accuracy prediction performance. In latency prediction experiments, our proposed model exhibits a clear advantage over NNLP, and outperforms NAR-Former by a significant margin. In summary, by incorporating the strengths of GNN, the universal representation learning framework NAR-Former V2 is significantly enhanced. NAR-Former V2 addresses the shortcomings of NAR-Former, which was overly sensitive when handling complete network structures, while still retaining the outstanding performance of NAR-Former when handling cell-structured networks.\n\n>3.  Poor performance on AlexNet and VGG. \n\nWe conjecture that this is mainly due to the fact that, compared to the other networks, the gap in data distribution between the training and test set is larger when using AlexNet and VGG as test sets, making prediction more difficult. This large difference in data distribution is reflected in two main areas. One aspect lies in the structure of the neural network. With the exception of AlexNet and VGG, all other networks are constructed by stacking cells or modules. This leads to the problem that the structure of the network to be encoded at the testing phase will be far from the vast majority of networks seen in the training stage. On the other hand, the gap between the depth of these two networks and the average network depth during training is the largest, and this difference in network size will also further enhance the difficulty of encoding and prediction during testing."}]

Author rebuttal:
We thank all reviewers for their valuable comments. Reviewers 9nHL,L4c1 and sJnp comment this paper is well written and easy to read. Reviewers L4c1, kdkx and sJnp agree with our motivation and comment the idea of using network connectivity by conditioning on the adjacency matrix of the network's graph is novel (new) or convincing. 

Next, we will address the reviewers' questions point by point.

Meta-review: This paper introduces NAR-Former V2, a universal neural network representation learning method capable of encoding any neural network architecture. It addresses the limitations of NAR-Former, particularly its poor generalization with larger networks. Specifically, the authors combine the self-attention mechanism of Transformers with the graph aggregation mechanism found in graph neural networks (GNNs). This fusion enables the Transformer to incorporate the inductive learning capabilities of GNNs, allowing it to generalize effectively to previously unseen architectures. Experimental validation of the proposed method on multiple benchmark datasets for attribute and latency prediction tasks demonstrates its effectiveness.

Initially, the paper received borderline ratings, with reviewers leaning slightly toward rejection. Reviewers generally appreciated the paper's well-written nature, clear motivations, and the novel idea of integrating graph representation learning into Transformers. The extensive experimental validation also left them convinced that the proposed method could be practical.

However, concerns were raised about several unsubstantiated claims lacking quantitative results. Reviewers also expressed worry about the modest improvements of the proposed method over NAR-Former and the absence of a comparison between the two in some experimental settings. Additionally, the method's subpar performance on certain architectures (e.g., AlexNet and VGG) raised concerns. There was also a lack of discussion on related works and comparisons with them.

During the discussion period, many of these concerns were addressed through clarifications and additional experimental results provided by the authors. This led to a consensus among the reviewers that the paper's strengths outweigh its weaknesses, despite it still remaining on the borderline.
Decision: Accept (poster)