SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Currently, among the various attributes of the network, accuracy and latency are the two main types of predicted objects. Because they have extremely high acquisition costs and are the primary manifestation of network performance and efficiency.
For latency prediction, due to the strong correlation between batch size, memory access, parameter quantity, and FLOPs with network latency, the encoding corresponding to these characteristics and the representation  $H^{K}$  are input into the predicting head together. To train the whole latency prediction model, the mean square error (MSE) function is adopted to measure the difference between predicted results and the ground truths.
For accuracy prediction, in addition to the MSE loss function, architecture consistency loss (AC_loss) and sequence ranking related loss (SR_loss) proposed by NAR-Former [47] are also used. Following NAR-Former, we employed a hierarchical fusion strategy in accuracy prediction experiments. We use a simplified approach, which computes the weighted sum of the outputs of each transformer layer with adaptive weights.
In this section, we conduct experiments on NNLQP [21], NAS-Bench-101 [48], and NAS-Bench-201 [10] to evaluate the performance of our NAR-Former V2. A series of ablation experiments were performed to corroborate the effectiveness of our design details. More experiments and more details about implementation will be provided in the supplementary materials.
Model details For latency experiments, the number of GraphAttn-based Transformer blocks is set to 2, which is the same as the baseline [21]. As for accuracy predicting, we fix the number of Transformer blocks to 6 to align with the standard Transformer used in the baseline [47].
Training details All experiments were trained using the Adam optimizer. We used a linear learning rate decay strategy with a warm-up, in which the learning rate uniformly increased to 0.001 (latency prediction) or 0.0001 (accuracy prediction) during the first  $10\%$  of the training steps and then gradually decayed to 0. The batch size was fixed at 16. Our models are trained on a machine with a GeForce RTX 3090 GPU. To avoid randomness, each model was trained 12 times and the two experiments with the best and worst indicators were discarded.
4.2 Latency prediction
We conduct latency prediction on the recently released NNLQP dataset [21], which comprises 20000 complete deep learning networks and their corresponding latencies on the target hardware. This dataset has 10 different types of networks (referring to the first column of Tab. 1), with 2000 networks per type. Following NNLP [21], we use Mean Absolute Percentage Error (MAPE) and Error Bound Accuracy  $(\mathrm{Acc}(\delta))$  to measure the deviations between latency predictions and ground truths. The lower the MAPE, the higher the prediction accuracy, while the opposite is true for  $\mathrm{Acc}(\delta)$ .
Here, we considered two different scenarios. In the first scenario, the training and testing sets are from the same distribution. We constructed the training set with the first 1800 samples from each of the ten network types, and the remaining 2000 networks were used as the testing set. The detailed results are shown in Tab. 1. When testing with all test samples, the average MAPE of our method is  $0.4\%$  lower than that of NNLP [21], and the average Acc  $(10\%)$  is  $1.16\%$  higher than that of NNLP.
Table 1: Latency prediction on NNLQP [21]. Training and test sets have the same distribution.
Test Model	MAPE↓	Acc(10%)↑
NAR-Former [47]	NNLP [21]avg / best	Oursavg / best	NAR-Former [47]	NNLP [21]avg / best	Oursavg / best
All	22.37%	3.47% / 3.44%	3.07% / 3.00%	35.00%	95.25% / 95.50%	96.41% / 96.30%
AlexNet	26.25%	6.37% / 6.21%	6.18% / 5.97%	27.00%	81.75% / 84.50%	81.90% / 84.00%
EfficientNet	13.91%	3.04% / 2.82%	2.34% / 2.22%	45.50%	98.00% / 97.00%	98.50% / 100.0%
GoogleNet	16.00%	4.18% / 4.12%	3.63% / 3.46%	39.00%	93.70% / 93.50%	95.95% / 95.50%
MnasNet	15.76%	2.60% / 2.46%	1.80% / 1.70%	33.00%	97.70% / 98.50%	99.70% / 100.0%
MobileNetV2	15.19%	2.47% / 2.37%	1.83% / 1.72%	39.00%	99.30% / 99.50%	99.90% / 100.0%
MobileNetV3	16.88%	3.50% / 3.43%	3.12% / 2.98%	36.00%	95.35% / 96.00%	96.75% / 98.00%
NasBench201	43.53%	1.46% / 1.31%	1.82% / 1.18%	55.50%	100.0% / 100.0%	100.0% / 100.0%
SqueezeNet	24.33%	4.03% / 3.97%	3.54% / 3.34%	23.00%	93.25% / 93.00%	95.95% / 96.50%
VGG	23.64%	3.73% / 3.63%	3.51% / 3.29%	26.50%	95.25% / 96.50%	95.85% / 96.00%
ResNet	28.18%	3.34% / 3.25%	3.11% / 2.89%	25.50%	98.40% / 98.50%	98.55% / 99.00%
Table 2: Latency prediction on NNLQP [21]. "Test Model = AlexNet" means that only AlexNet models are used for testing, and the data from the other 9 model families are used for training. The best results refer to the lowest MAPE and corresponding ACC (10%) in 10 independent experiments. *: obtained based on the released code without using its fine-tuning step.
Metric	Test Model	FLOPs	FLOPs +MAC	nn-Meter [55]	TPU [17]	BRP-NAS [12]	NAR-Former [47]*	NNLP [21] (avg / best)	Ours (avg / best)
MAPE↓	AlexNet	44.65%	15.45%	7.20%	10.55%	31.68%	46.28%	10.64% / 9.71%	24.28% / 18.29%
EfficientNet	58.36%	53.96%	18.93%	16.74%	51.97%	29.34%	21.46% / 18.72%	13.20% / 11.37%
GoogleNet	30.76%	32.54%	11.71%	8.10%	25.48%	24.71%	13.28% / 10.90%	6.61% / 6.15%
MnasNet	40.31%	35.96%	10.69%	11.61%	17.26%	26.70%	12.07% / 10.86%	7.16% / 5.93%
MobileNetV2	37.42%	35.27%	6.43%	12.68%	20.42%	25.74%	8.87% / 7.34%	6.73% / 5.65%
MobileNetV3	64.64%	57.13%	35.27%	9.97%	58.13%	33.99%	14.57% / 13.17%	9.06% / 8.72%
NasBench201	80.41%	33.52%	9.57%	58.94%	13.28%	105.71%	9.60% / 8.19%	9.21% / 7.89%
ResNet	21.18%	18.91%	15.58%	20.05%	15.84%	40.37%	7.54% / 7.12%	6.80% / 6.44%
SqueezeNet	29.89%	23.19%	18.69%	24.60%	42.55%	74.59%	9.84% / 9.52%	7.08% / 6.56%
VGG	69.34%	66.63%	19.47%	38.73%	30.95%	44.26%	7.60% / 7.17%	15.40% / 14.26%
Average	47.70%	37.26%	15.35%	21.20%	30.76%	45.17%	11.55% / 10.27%	10.55% / 9.13%
Acc(10%)↑	AlexNet	6.55%	40.50%	75.45%	57.10%	15.20%	7.60%	59.07% / 64.40%	24.65% / 28.60%
EfficientNet	0.05%	0.05%	23.40%	17.00%	0.10%	15.15%	25.37% / 28.80%	44.01% / 50.20%
GoogleNet	12.75%	9.80%	47.40%	69.00%	12.55%	24.35%	36.30% / 48.75%	80.10% / 83.35%
MnasNet	6.20%	9.80%	60.95%	44.65%	34.30%	20.90%	55.89% / 61.25%	73.46% / 81.60%
MobileNetV2	6.90%	8.05%	80.75%	33.95%	29.05%	20.70%	63.03% / 72.50%	78.45% / 83.80%
MobileNetV3	0.05%	0.05%	23.45%	64.25%	13.85%	16.05%	43.26% / 49.65%	68.43% / 70.50%
NasBench201	0.00%	10.55%	60.65%	2.50%	43.45%	0.00%	60.70% / 70.60%	63.13% / 71.70%
ResNet	26.50%	29.80%	39.45%	27.30%	39.80%	13.25%	72.88% / 76.40%	77.24% / 79.70%
SqueezeNet	16.10%	21.35%	36.20%	25.65%	11.85%	11.40%	58.69% / 60.40%	75.01% / 79.25%
VGG	4.80%	2.10%	26.50%	2.60%	13.20%	11.45%	71.04% / 73.75%	45.21% / 45.30%
Average	7.99%	13.20%	47.42%	34.40%	21.34%	14.09%	54.62% / 60.65%	62.70% / 67.40%
When tested on various types of network data separately, except for the NASBench201 family, our method consistently outperforms NNLP. This indicates that our improved transformer, which utilizes the structural characteristics of the graph, has learned more reasonable representations than the original GNN. Comparison with NAR-Former's results also proves that our NAR-Former V2 is more effective than the original transformer-based model in learning representations of complete DNNs.
The second scenario has more practical application significance, that is, the network type needed to be inferred is not seen during the training process. There are ten sets of experiments in this part, with each set taking one type of network as the test set, while all samples from the other nine types of networks are used as the training set. As shown in Tab. 2, it can be seen that using only FLOPs and memory access information to predict latency is not enough. Suffering from the gap between the accumulation of kernel delays and the actual latency, kernel-based methods (TPU[17] and nn-Meter[55]) perform worse than the GNN-based model NNLP that directly encodes and predicts the entire network. Despite encoding the entire network directly, because of the sensitive nature introduced by the global modeling computational properties of the transformer, it may result in the transformer-based NAR-Former being less able to generalize to networks that have not been seen at the training stage. Benefiting from considering the entire input network and grafting GNN into the transformer, our method achieves the best MAPE and Acc(10%) on the average indicators of 10
experimental groups. Compared with the second-best method NNLP, the average  $\mathrm{Acc}(10\%)$  of our method has a marked increase of  $8.08\%$ .
4.3 Accuracy prediction
4.3.1 Experiments on NAS-Bench-101
NAS-Bench-101 [48] provides 423624 different cell architectures and the accuracies of the complete neural network constructed based on each cell on different datasets. Following [26],  $0.1\%$  and  $1\%$  of the whole data are used as the training set, and another 200 samples are used for validation. We use Kendall's Tau [36] to evaluate the correlation between the predicted sequence and the real sequence, and a higher value indicates better results.
Kendall's Tau is calculated on the whole dataset or 100 testing samples. We report the average results of our predictor in 10 repeated experiments. Results are shown in Tab. 3. When only
Table 3: Accuracy prediction on NAS-Bench-101 [48]. "SE" denotes the self-evolution strategy proposed by TNASP [26].
Backbone	Method	Training Samples
0.1% (424)	0.1% (424)	1% (4236)
Test Samples
100	all	all
CNN	ReNAS [46]	0.634	0.657	0.816
LSTM	NAO [27]	0.704	0.666	0.775
NAO+SE	0.732	0.680	0.787
GNN	NP [43]	0.710	0.679	0.769
NP + SE	0.713	0.684	0.773
CTNAS [3]	0.751	-	-
Transformer	TNASP [26]	0.752	0.705	0.820
TNASP + SE	0.754	0.722	0.820
NAR-Former [47]	0.801	0.765	0.871
NAR-Former V2	0.802	0.773	0.861
424 samples were available for training, our method achieved the highest Kendall's Tau. We achieve 0.773 when tested using the whole testing set, which is  $0.8\%$  and  $8.9\%$  higher than the transformer-based model [47] and GNN-based model [43], respectively. This proves that the modifications we made to the transformer based on inspiration from GNN are effective.
4.3.2 Experiments on NAS-Bench-201
NAS-Bench-201 [10] is another cell-based dataset, which contains 15625 cell-accuracy pairs. Following [26],  $5\%$  and  $10\%$  of the whole data is used as the training set and another 200 samples are used for validation.
We use Kendall's Tau [36] computed on the whole dataset as the evaluation metric in this part. The average results of our predictor of 10 runs are reported. Results are shown in Tab. 4. The conclusion of this experiment is similar to Sec. 4.3.1. When compared with the second-best method, a substantial improvement  $(2.5\%)$  of Kendall's Tau can be seen in the setting of
Table 4: Accuracy prediction on NAS-Bench-201 [10]. "SE" denotes the self-evolution strategy proposed by TNASP [26].
Backbone	Model	Training Samples
(781) 5%	(1563) 10%
LSTM	NAO [27]	0.522	0.526
NAO + SE	0.529	0.528
GNN	NP [43]	0.634	0.646
NP + SE	0.652	0.649
Transformer	TNASP [26]	0.689	0.724
TNASP + SE	0.690	0.726
NAR-Former [47]	0.849	0.901
NAR-Former V2	0.874	0.888
training with 781 samples. A more comprehensive comparison with the baseline [47] using additional metrics can be found in the supplementary materials.
Regarding the average Kendall's Tau under different training settings, our method is on par with NAR-Former with 0.812 on nas-bench-101 and slightly higher than NAR-former by  $0.6\%$  on NAS-Bench-201. Therefore, compared to NAR-Former, NAR-Former V2 achieves comparable accuracy prediction performance. In latency prediction experiments, our proposed model exhibits a clear advantage over NNLP and outperforms NAR-Former by a significant margin. In summary, by incorporating the strengths of GNN, the universal representation learning framework NAR-Former V2 is significantly enhanced. NAR-Former V2 addresses the shortcomings of NAR-Former, which was overly sensitive when handling complete network structures, while still retaining the outstanding performance of NAR-Former when handling cell-structured networks.
In this section, we conducted a series of ablation experiments on the NNLQP dataset to investigate the impact of various modifications. The results from Rows (2) and (3) in Table 5 indicate that for type encoding without numerical relationships, using one-hot vectors with equidistant proper
Table 5: Ablation studies on NNLQP [21]. "PE" denotes position encoding.
Row	Structure	Op Type	Op Attributes	Graph-Attn	GFFN	TA-Enhance	MAPE↓	Acc(10%)↑	Acc(5%)↑
1(Baseline)	GNN	One-hot	Real Num	-	-	-	3.48	95.26	77.80
2	GNN	PE	PE	-	-	-	3.43(-0.05)	95.11(-0.15)	79.58(+1.78)
3	GNN	One-hot	PE	-	-	-	3.33(-0.15)	95.57(+0.31)	80.19(+2.39)
4	Transformer	One-hot	PE	✓	-	-	3.20(-0.28)	96.00(+0.74)	81.86(+4.06)
5	Transformer	One-hot	PE	✓	✓	-	3.20(-0.28)	96.06(+0.80)	81.76(+3.96)
6	Transformer	One-hot	PE	✓	✓	✓	3.07(-0.41)	96.41(+1.15)	82.71(+4.91)
ties across different categories is more suitable. Comparing Row (3) in Table 5 with Row (4), we observe that introducing GNN characteristics into the Transformer improves the model's ability to learn effective representations and achieve more accurate predictions compared to using the original GNN. When replacing the FFN with the GFFN module with eight groups (Row (5)), the number of model parameters reduces to approximately one-eighth of that in Row (4), without a significant decrease in prediction accuracy. Compared to Row (5), Row (6) demonstrates an increase of  $0.35\%$  in  $\mathrm{ACC}(10\%)$  and  $0.95\%$  in  $\mathrm{ACC}(5\%)$ . This confirms the role of the type-aware enhancement module in further refining and enhancing the rationality of the representations.
To verify our hypothesis regarding the generalization ability of the network and the effectiveness of the proposed graph-aided attention, we conducted comparative experiments in scenarios where the training and testing data have different distributions. The results of these ex
Table 6: The influence of using different attentions. Test on EfficientNet family.
Attention	MAPE↓	ACC(10%)↑
Global	16.88%	36.32%
Local	13.20%	44.01%
periments are presented in Table 6. In order to perform the experiment on global attention, we excluded the step of multiplying the adjacency matrix  $A$  in Equation 9, and instead replaced  $S^l$  with  $X^l X^{lT} / \sqrt{d}$ . Results in Table 6 demonstrate that incorporating the adjacency matrix to restrict the scope of attention calculation is indeed beneficial for latency prediction on unseen data. The model utilizing graph-aided attention exhibited a significant improvement of  $7.68\%$  in ACC(10%) compared to the model using global attention.
In this paper, we combine the strengths of Transformer and GNN to develop a universal neural network representation learning model. This model is capable of effectively processing models of varying scales, ranging from several layers to hundreds of layers. Our proposed model addresses the limitations of previous Transformer-based methods, which exhibited excessive sensitivity when dealing with complete network structures. However, it still maintains exceptional performance when handling cell-structured networks. In future work, we will focus on optimizing the design of the representation learning framework and applying it to a broader range of practical applications. Such as using the proposed model to search for the best mixed precision model inference strategies.
This work was supported in part by the National Key Research and Development Program of China under Grant 2018AAA0103202; in part by the National Natural Science Foundation of China under Grants U22A2096 and 62036007; in part by the Technology Innovation Leading Program of Shaanxi under Grant 2022QFY01-15; in part by Open Research Projects of Zhejiang Lab under Grant 2021KG0AB01 and in part by the Fundamental Research Funds for the Central Universities under Grant QTZX23042.