TNASP: A Transformer-based NAS Predictor with a Self-evolution Framework
Authors: Shun Lu
Jixiang Li
Jianchao Tan
Sen Yang
Ji Liu
Venue: NeurIPS 2021 Poster (2021)
Keywords: Neural Architecture Search
Predictor
Transformer
Self-evolution

Abstract: Predictor-based Neural Architecture Search (NAS) continues to be an important topic because it aims to mitigate the time-consuming search procedure of traditional NAS methods. A promising performance predictor determines the quality of final searched models in predictor-based NAS methods. Most existing predictor-based methodologies train model-based predictors under a proxy dataset setting, which may suffer from the accuracy decline and the generalization problem, mainly due to their poor abilities to represent spatial topology information of the graph structure data. Besides the poor encoding for spatial topology information, these works did not take advantage of the temporal information such as historical evaluations during training. Thus, we propose a Transformer-based NAS performance predictor, associated with a Laplacian matrix based positional encoding strategy, which better represents topology information and achieves better performance than previous state-of-the-art methods on NAS-Bench-101, NAS-Bench-201, and DARTS search space. Furthermore, we also propose a self-evolution framework that can fully utilize temporal information as guidance. This framework iteratively involves the evaluations of previously predicted results as constraints into current optimization iteration, thus further improving the performance of our predictor. Such framework is model-agnostic, thus can enhance performance on various backbone structures for the prediction task. Our proposed method helped us rank 2nd among all teams in CVPR 2021 NAS Competition Track 2: Performance Prediction Track.

TNASP: A Transformer-based NAS Predictor with a Self-evolution Framework
Shun Lu Jixiang Li Jianchao Tan', Sen Yang', Liu' Research Center for Intelligent Computing Systems, State Key Laboratory of Computer Architecture,
Institute Computing Technology. Chinese Academy of Sciences
University of Chinese Academy of Sciences
Kuaishou Technology lushun19s@ict. ac. cn, {lijixiang, Janchaatta,, senyang, jiliu)/kkaishou.com
Abstract
Predictor-based Neural Architecture Search (NAS) continues to be an important topic because aims to mitigate the consuming search procedure of traditional NAS methods. A promising performance predictor determines the quality of final searched models predictor-based NAS methods. Most existing predictor-bast based methodologies train model-basee predictors under proxy dataset setting, which may suffer from the accuracy decline and the generalization problem, mainly due their poor abilities represent spatial topology information of the graph structure data. Besides the poor encoding for spatial topology information, these works did not take advantage of the temporal information such as historical evaluations NAS performance predictor,
during training. Thus, we propose a Transformer-basee associated with Laplacian matrix based positional encoding strategy, which better represents topology information and achieves better performance than previous state-of-the the-art methods NAS-Bench- 101, NAS Bench-201, and DARTS search space. Furthermore, we also propose a self-evolution framework that can fully utilize temporal information guidance. This framework iteratively involves previous evaluation information as constraints into current optimization iteration, thus further improving the performance of our predictor. Such framework is model- agnostic, thus can enhance performance on various backbone structures for the prediction task. Our proposed method helped rank 2nd among all teams CVPR 2021 NAS Competition Track 2: Performance Prediction Track.
Introduction
Neural Architecture Search (NAS) aims to automatically find out superb architectures in pre- defined search space. The NAS models have outperformed human-designed models in many domains [37, 18, 13, 3, 15]. However, traditional NAS methods like Reinforcement learning and Evolutionary learning require enormous computation resources, i.c., hundreds or thousands of GPU hours, to train sub-models to obtain their performance estimation. The intensive computation prohibits NAS models from deployments to real applications. Although differentiable NAS methods [26] have less search time than traditional methods, they usually suffer from performance collapse due to several problems such as optimization gap [6], discretization discrepancy and the unfair advantage [9], trapping into sharp minimum and the dominant eigenvalue [49, 5].
To reduce the search cost in NAS, predictor- based NAS methods [30, 23, 43, 29, 32, 7, 46, 4, 1] use performance predictors to predict the accuracy of architectures quickly instead of training all architectures get the accuracy. Simple training-free predictors have been shown promising in some applications, however, those performances are usually not good enough in practice. Many
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

works focus on how to design effective training-basee predictors [43, 29, 32, 7, 46]. A training-based predictor usually consists of an encoder module and regressor module, and only needs learn from few architecture-accurae sampled pairs, thus leading to fast learning procedure. After the performance predictor learns the relationship between the architecture and its corresponding accuracy, it is able to predict the performance of other unseen architectures in the search space, which greatly accelerates the search process of NAS.
One of the key components of a performance predictor is to encode discrete architectures into continuous feature representations. Neural predictor[43] and CTNAS[7] applied GCN [20] to capture the feature representation of input model structures. Both SemiNAS [29] and GATES [32] learned an embedding matrix for the candidate operations in the search space, and represented architectures as a combination of different embeddings. ReNAS[46] calculated the types matrix, flops matrix, and parameters matrix, and then concatenated them together to form a feature tensor to represent a specific architecture. Unlike previous methods, we propose a Transformer-base Predictor (TNASP) and use the linear transformation output over the Laplacian matrix of the model structure graph to be the positional encoding.
NAS performance
There are several advantages of the Transformer that can be used to train a good performance predictor. First, the self-attention module can help explore better feature representations from the graph structure data. Second, the multi-hear mechanism can further help encode the different subspace information at different positions from the graph structure data, as also claimed by the original Transformer paper. Third, the Laplacian matrix based positional encoding method also fits well find topology position information on the graph. In summary, we demonstrate that Transformer is an effective method to extract feature representation from discrete architecture graphs, and also has superb generalization abilities for processing unseen data, as shown in our experiments. Since the predictors are trained on a small size proxy dataset but the test dataset is much larger, training-based NAS predictors usually have poor generalization abilities. The powerful Transformer encoder can mitigate this problem to certain degree, due to its good ability to encode topology information.
To further improve our predictor, we introduce a self-evolution framework, which makes full use of temporal information to guide the training. The framework iteratively involves each evaluation score of the previously predicted results on validation dataset into gradient based optimization iteration as constraints, to push the predictions close to ground truths gradually. We demonstrate the framework can make our predictor have better generalization, and also achieve better performance than previous predictors.
Our proposed framework is fruitful and practical in several scenarios, for example, all of the machine learning competitions. In competition, the test set that the host provided users is exactly the validation role in our setting. And the temporal information, such as the feedback of each submission, able to gradually improve original training iteratively with our framework. Generally speaking, our contributions can be summarized as follows:
We propose a Transformer-based NAS performance predictor (TNASP) to better encode the spatial topology information, utilizing the multi-head self-attention mechanism to map the discrete architectures to a meaningful feature representation and applying the lincar transformation of the Laplacian matrix as the positional encoding.
By leveraging each evaluation score information in history as constraints in training, and applying a gradient based optimization method to iteratively solve the constrained opti- mization problem, we introduce a generic self-evolution framework to further improve the performance the proposed predictor, making full use of temporal information.
Our proposed method surpasses the previous state-of-the-art methods under the same proxy training dataset and achieves state-of-the-arr results NAS-Bench-101 [48], NAS-Bench- 201[14]. and DARTS [26] search space.
2
Related Work
Due to the enormous search cost of traditional NAS methods such as reinforcement learning based methods [52, 2] and evolutionary methods [34, 27], network performance predictor based NAS methods have become pretty active topic recently. Most works only require a few architecture- accuracy data pairs to train predictor and estimate the performance of unseen architectures, which

---

can be categorized as training-bass based network performance predictors. Other works propose to calculate some kinds of metrics over network structure represent network performance without training, which are denoted as training-free network performance predictors.
Training-basee based network performance predictors Training- Training-based predictors follow unified paradigm learn the correlation from network architectures and their corresponding accuracy. As it's hard to learn useful features directly from the discrete network architectures, various methods have been explored to map the discrete representation into a continuous latent space and can be roughly divided into sequence-based schemes and graph-based methods. Sequence-basee based schemes denote each architecture by a discrete sequence with fixed length and use MLP [25, 44, 46], embedding matrix [11, 32), Auto-Encoder [30, 51, 29] GBDT [28] to convert the sequence into continuous representation. However, graph-based methods treat the architecture as a graph and use the graph- format data i.e. adjacency matrix and node features as inputs. Diverse graph processing methods such as GHN [50], GCN [22, 43, 36, 71. GIN [47] and WL-Kernel [35] have been tried. Similar to these methods, our Transformer-bast -based predictor also belongs to the training-based network performance predictors category. Differently, we propose a novel encoding scheme by utilizing the Laplacian matrix as the positional encoding and leveraging the multi-head self-attention mechanism from the Transformer [41] ] to encode network architectures to excavate more representative features.
Training-fre free network performance predictors Recently, several works proposed to predict the network performance by designing some metries, without training model to perform prediction. [31] scored networks at initialization by computing correspondence between binary codes in the whole mini-batch. TE-NAS [4] analyzed trainability and expressivity evaluate each network performance. Zero-Cost NAS [I] compared six proxies to predict network performance, such as grasp [42]. fisher [39, 40], synflow [38], and on, but failed rank architectures NAS-Bench-100 [48]. Zen-NAS [24] measured the expressivity of a network by expected Gaussian complexity to represent its performance. Although these training-free network performance predictors are efficient and have achieved satisfying results on several datasets, their robustness can not be guaranteed and the performance fluctuated dramatically among different tasks. Compared to these methods, our training- brasedng-based predictor is more tine-consuming, but its performance is dramatically better.
Methods
In this section, we first briefly review the common paradigm for the training-based NAS predictors as preliminary. Following the same paradigm, we then introduce the details of our Transformer-base based predictor. Furthermore, we introduce general framework to utilize the information in evaluation histories over a validation dataset as constraints in gradient based optimization to advance the predictor by a self-evolution framework.
3.1 Training-basee based network performance predictors
Previous works [30, 43, 46, 7] proposed to first apply an encoder fø to encode the discrete architec- tures into continuous feature representations, which can be formulated as:
fE
where A E RNNNN denotes the adjacency matrix and stands for the directed acyclic connections between nodes, N denotes the number of the nodes. k E NxF stands for the feature matrix and represents the characteristics of the nodes, F denotes the output dimension of the embedding extractor. For NAS predictors, the adjacency matrix A shows the topology information of an architecture and usually indicates the representation of operations for nodes or edges. Among previous works, the encoder JE can be GCN or LSTM or simply an embedding matrix, and the embedding vector can be interpreted as a latent representation of a specific architecture.
After encoding the discrete architecture into a continuous representation using an encoder, it is easier and more accurate to estimate the network accuracy by a simple regressor fR once the embedding vector e meaningfully represents the architecture in the latent space.
fR(E)
(2)
3

---

Operationll Encodrg
Embedding
Extractor
24 2.
Operation Indox
Operaional Raprosertation
Transfomerr Encoder
Postoona Erooding
Festure
Precicioo
0
MLP
Posiiionl
Encoder
Regressor
Figure I: Our Transformer-basee NAS predictor mainly consists of an encoder and a regressor. We first encode the information of operations and connections into continuous representation, followed by 3 Transformer encoder layers, and the regressor uses the output feature of Transformer encoder layers derive the final prediction. Color grey in features denotes integral form and other colors denote float form. N, F and M denote the number the nodes, the number of candidate operations for each node and the latent dimension of the continuous feature, respectively.
Generally, we apply the MSE loss to supervise the training of the encoder and the regressor, and we use 9 to denote the parameters of both encoder fE and regressor fn- The optimal parameters 6" can be optimized by stochastic gradient descent method:
0 arg min Σ Wt y1)
where represents the total number of the training data, yi denotes the ground truth of the i-th sample.
3.2 Transformer-based predictor
Recently, the Transformer architecture [41] has attracted much attention and outperformed many state-of-the-arr models due to its powerful representation ability, which motivates us to borrow the decent encoder structure to be our predictor backbone.
As shown in 1, we first get the operation feature er E RNN by transforming the operation vector & with an embedding matrix E E Fx
el = E())
(4)
To characterize the positional information, we have tried various encoding methods, and find that the Laplacian matrix is an effective scheme and has the potential to better represent topology information of graph structure data as it contains both the connectivity of graph and the centrality of each node. Thus we propose to encode the positional information by utilizing the Laplacian matrix (L E R NNx. computed from the adjacency matrix (A E E NxN and the degree matrix (D N
L= D
(5)
We did not choose the normalized Laplacian matrix, because we find it does not perform well in our experiments.
Then we use linear layer to map the the Laplacian matrix (L) to a continuous feature vector ez whose dimension same as the embedding vector e1:
c MLP(L)
Finally, we get the the continuous representation from the Transformer encoder with multi-head self-attention module:
Transformer (E1 e2)

---

Optimization Process
Optimization Status
training at fiention
git
ytrue
evaluate ituration
ovaluate at Iteration
fark)
(5)
crue
(x)
y
Model (a()
trinn loss
fv( (v)
evaluare iteration
volld
training at ineration 2+]
uptimion madid parmeccc
tool loss
oycumine validation
lobel perameters
consuaini
comamaan
conataire
g
g
evaluste si itenation
g(0+1)
ytrue
evalunte
itenatin
Modal farin
training Doraaion 1
Figure 2: The left sub-figure illustrates the optimization process of self-evolution framework, the blue arrow denotes the input of the validation dataset and the yellow arrow stands for the input the training dataset. The right sub-figure shows the status transition of the validation predictions and the trainable validation label parameters, both are approaching ground truth gradually. All the symbols can be found in equations in Section 3.3.
As the continuous representation contains richer information about the architecture (including the operations and the connections) in the latent space. it's easier to learn the relationship between this representation and its corresponding accuracy. Therefore, we only choose simple regressor, specifically Multi-Layer Perceptions (MLP), to estimate the final accuracy. The experiments demonstrate the promising performance of whole designs.
3.3
Self-evolution framework
Besides our predictor design that has good encoding ability for spatial topology information, we also propose framework that further improve performance by fully utilizing temporal information during training. This framework iteratively involves the evaluation information in the previous iteration over validation dataset into the current iteration of optimization constraints, shown in Fig. 2. Our training strategy is useful and practical in several scenarios, for example, all of the machine learning competitions. In competition, the test set that the host provided to users is exactly the validation role in our framework. And the system feedback of each submission can gradually improve the original training iteratively by adding more and more historical evaluations constraints. Specifically, our objective function can be formulated as below:
[o(())) vill + Σ [15((1)) -0312
min
==1
(8)
s.t. M 1105 - =: 1.,2,,3....
where, is training data with size n. And we call as help data validation data with size V in this paper. We denote number evaluation on validation dataset o with And a is the weight to balance train loss and validation loss. We choose different value for in different experiments. Y; is an auxiliary variable that serves as the proxy of ground truth accuracy in validation dataset. fo(v;) current forward pass inference results of predictor fe(x)) over validation data Uy, which is used in
5

---

back-propagation step to update model parameters 0 at current iteration, (00) is the corresponding previous predictions at iteration t = 1, 3........ T.
For e it should be computed the system back-end, for example, in the competition situation, users can only get the final evaluation score instead of the ground truth label of every validation sample. For simplicity, we choose MSE as the evaluation metric between the historical prediction results 1/5) and the user-unknown ground truth vr In practical situation, the metric can be more complicated. urrue is user-unknown, which has diverse meanings in different situations. On the one hand, for the Machine Learning competitions, teh submission, submit our predictions o//" to the system, and the system will compute the e(1) at the back-end privately and return only e(*) to us, and thus we call uffue is user-unknown. Our self-evolution system can make use of only these e(()) fort 2, 3, .... incrementally improve our predictor. On the other hand, for offline benchmark experiments conducted in this paper, we use evaluation error on validation set V in each previous iteration as e!!) to guide the current iteration training using our SE framework. We did know the ground truth this validation set V, but we did not make use of this ground truth directly in the framework. We just mimic the aforementioned system of the Machine Learning competitions to compute e(t) as in equation 9, which is used our evolving procedure. Thus the ground truth here is also called user-unknown.
V
(1)
Σ
10
10 - yj
2
1 = 1,2,,,
(9)
J=1
We formulate above Eq. 8 as Minimax optimization problem using Lagrangian Multiplier:
L(O, 9. A) min max 0,0
[
yill? + Σ [1 1sl1
(10)
V
100
(0) "0b112 (1))
At each training step, we adopt gradient based optimization method to update the variables. And the whole framework is summarized in algorithm 1.
0L g A²)
g++1 0²
(11)
00
@ D. X*)
t+ -it²
17g
(12)
d (8h ,²,
h+1 A 7 Ox
(13)
In summary, the self-evolution framework can make full use of any available information (either the competition system historical submission feedback historical evaluation information validation dataset during model training) to guide the predictor training to avoid over-fitting, thus generalized well on the test dataset. Moreover, this framework directly treats each historical validation evaluation information as each hard constraint during training and reformulates the whole constrained training problem minimax optimization problem, solved by pradient-based" optimization method efficiently and effectively.
Experiments
We employ our TNASP on three different search spaces, specifically NAS-Bench-10111 NAS- Bench-201 and DARTS[26). Moreover, we put several experimental results performed on ImageNet [21]; a comprehensive comparison with GCN, SemiNAS [29], BONAS [36]; the imple- mentation details; searched architecture visualizations in supplementary materials.

---

Algorithm/ Self-evolution Optimization Algorithm
Input: Input training data z; input validation data e, input training target y, neural network f. Output: Network parameters 8, estimated target y.
1: Optimize the network parameters 0 until convergence using normal training performed on the training dataset only.
2: for t = 1 to I do
3: Compute e(t) according Eq. (9) using predictor's prediction results on validation dataset.
4: Add a new constraint: + Σ f= [0) - ø = e(l) into Eq. (8)
5: while not converged do
6: Update according to the Eq. (11)
7: Update according to the Eq. (12)
8: Update according to the Eq. (13)
9: end while
end for
11:
return Network parameters 9 and estimated targets y
4.1 Experiments NAS-Bench-101
There are 423,624 different architectures in NAS- HAS-Bench-101[48) Each architec- ture stacked by repeated cells. The maximum number of nodes and edges for each cell are and 9, respectively. Nodes represent different candidate operations and edges show the connection between nodes.
Bench-101[48] 101[48] provides the validation accuracy and the test accuracy for three different runs. Following Neural Predictor [43], we utilize the validation accuracy from a single run as the training target and apply the mean test accuracy over three runs as the ground truth accuracy for evaluating the performance of our predictions. We train all models for 300 epochs with batch size 10 using Adam optimizer and the learning rate is le-4 with a cosine decay strategy.
Comparison with SOTA methods Following the settings in [43, 7, 46], we choose 0.02%, 0.04%, 0.1% and 1 1% of the whole data test set to calculate Kendall's Tau to evaluate the performance of different predictors. The results are shown in 1. We can see that when the training data is extremely deficient (only 0.02% and 0.04%), our predictor achieves obviously higher Kendall's Tau than Neural Predictor [43] and NAO [30], which illustrates the stronger representation capability of method in few-shot training scenarios. When using 0. % of the whole data as the training set, our method still outperforms theirs. When the training data size becomes larger(1 %),, the performance of all predictors has been improved obviously, due to more information gained. However, our method still beats them.
training set to train our predictor. And we use all the data
Although our TNASP has got the best results in all kinds of data splits, we demonstrate that the performance can be further improved with our proposed self-evvlution((S)) framework. We use another 200 data as the validation set and only apply the MSE loss over these validation data the constraints to guide the training optimization with our SE framework, TNASP can get higher Kendall's Tau under a variety of different data splits. Furthermore, our framework is generic and easy to combine with other methods. When applied to Neural Predictor [43] and NAO [30], both methods achieve higher Kendall's Tau shown in Tab. fully demonstrating the effectiveness of our proposed self-evolution framework.
our predictor. As shown in Tab. when applied
4.2 Experiments NAS-Bench-201
NAS-Bench-201 Architectures in WAS-Bench-201 [14] are constructed with repeated cells. All cells are composed of 4 nodes and 6 edges, and every cell inside an architecture shares the same structure, Each edge represents an operation, and there are candidate operations in total, resulting 15,625 different cell candidates. NAS-Bench-200 [14] provides three different results of each architecture on three different datasets and we choose C1FAR-10 results as our targets.
Comparison with SOTA methods Similar to the experimental setup on we directly train predictors ranging from 0.05% to 10% of the whole dataset and also evaluate the performance the whole dataset. As shown in Tab.2, TNASP consistently achieves the highest
[48],

---

Training Samples
100 (0.02%)
172 (0.04%)
424 (0.1%)
424 (0.1%)
4236 (1%)
Validation Samples
200
200
200
200
200
Test Samples
all
all
100
all
all
Neural Predictorf [43]
0.391
0.545
0.710
0.679
0.769
SPOS [16]
0.196*
FairNAS [8]
-0.232*
NAO* [30]
0.501
0.566
0.704
0.666
0.775
ReNAS [46]
0.634*
0.657
0.816
RegressionNAS
0.430*
CTNAS [7]
0.751*
TNASP
0.600
0.669
0.752
0.705
0.820
Neural Predictor SE
0.458
0.577
0.713
0.684
0.773
NAO + SE
0.564
0.624
0.732
0.680
0.787
TNASP + SE
0.613
0.671
0.754
0.722
0.820
Table 1: Comparison with other methods on MASS-Bnnch-101 We calculate the Kendall's s Tau by predicting accuracy of all architectures in NAS-Bench-101 101. t; re-implemented by ourselves. implemented based on their released model. *: reported by CTNAS[7].
Training Samples
78(0.05%)
156(1 %)
469(3%)
781(5%)
1563(10%)
Validation Samples
200 all
200
200
200
200
Test Samples
all
all
Neural Predictor [43]
0.343
0.413
0.584
0.634
0.646
NAO* [16]
0.467
0.493
0.470
0.522
0.526
TNASP
0.539
0.589
0.640
0.689
0.724
Neural Predictor + SE
0.377
0,433
0.602
0.652
0,649
NAO SE
0.511
0.51
0.514
0.529
0.528
TNASP SE
0.565
0.594
0.642
0.690
0.726
Table 2: Comparison with other methods on predicting the accuracy all architectures in TAS-Bench-201 and comparing them with ground truths. i: re-implemented by ourselves. 8; implemented based their released model.
We calculate the Kendall's Tau by
Kendall's Tau and has large improvement compared to other models regardless of the amount of training data. When applied with the self-evolution framework, TNASP+SE obtains further higher Kendall's Tau at each proportion of the data. Neural Predictor [43] and NAO [30] have the maximum Kendall's Tau improvement of 0.034 and 0.044, respectively, indicating the self-evolution framework effective for various predictors and different search spaces.
From the above observations, we can conclude that when the number of validation samples is greater than training samples, the predictor can get a larger performance improvement by using the SE- framework. On the contrary, the predictor only got little performance gains when the number of training samples is much larger than validation samples. This phenomenon happens not only on our Transformer-based predictor but also on Neural Predictor and NAO.
4.3 Experiments on DARTS
DARTS Architectures DARTS search space are built by normal cells and reductions cells. Each cell consists of 7 nodes and 14 edges. Every edge is a selection from the 7 candidate operations (we omit the Zero operation our experiments). Following CTNAS [7], we first train supernet on CIFAR-10 10 with a uniform sampling strategy in [16] for 120 epochs, to get proxy labels for architectures. After the supernet training, we randomly sample 1000 architectures in this search space and query their test accuracy by inheriting the supernet weights to construct the architecture-acuuee pair. Finally, we utilize the generated data pairs to train our TNASP model and apply in an evolutionary algorithm [10] to search for good architectures in the search space.

---

Architecture
Test Accuracy(%)
#Params.(M)
Search Cost(G-D)
DenseNet-BC[19) [19]
96.54
25.6
PyramidNet-BC [17]
96.69
26.0
Random search baseline
96.71 + 0.15
3.2
NASNet-A [53] + cutout
97.35
3.3
1,800
NASNet-B [53] cutout
96.27
2.6
1,800
NASNet-C [53] + cutout
96,41
3.1
1,800
AmoebaNet-A [34] cutout
96.66 0.06
3.2
3,150
SNAS [45] ENAS [33] + cutout
97.02
2.9
1.5
97.11
4.6
0.5
DARTS [26] + cutout
97.24 + 0.09
3.4
4
NAONet [30]
97.02
28.6
200
PNAS [25] + cutout
97.17 0.07
3.2
GHN [50] cutout
97.16 + 0.07
5.7
0.8
D-VAE [51]
94,80
NGE [22] + cutout
97.40
0.1
BONAS-A [36] + cutout
97.31
3.45
2.5
CTNAS [7] cutout
97.41 = 0.04
3.6
0.3
TNASP cutout(avg)
97.43 0.04
3.6 + 0.1
0.3
TNASP cutout(best)
97.48
3.7
0.3
Table 3:
Comparison with other methods in DARTS [26] search space on CIFAR-10. "cutout"; evaluate the searched cells using cutout [12] data augmentation. "G-D": GPU days.
Comparison with SOTA methods To avoid randomness, select the top-3 searched architectures to re-train and report the average metrics in the Tab.3. We can notice that our searched cells achieve higher average test accuracy than all the other methods, and the best searched cells get the highest test accuracy 97.48, which implies that our predictor learns the effective correspondence between architectures and accuracy. We visualize the best searched architecture in Fig.3 and append the other searched cells in our supplementary material. Noticeably, several convolution operations appear in the reduction cell and the normal cell contains mar __l__poo_____ 3 3. which rarely appears other methods searched normal cells. Our searched cells have a novel structure and achieve higher test accuracy, showing that our method obtains better local minimum than other methods. It's interesting to explore the essential differences between the cells of different methods in the future.
A cunnect
(_(k-)
sp come
al Loaty,
(k2))
sp, corv: 3x3
sons 3x5
pyp_poo_333
pool 367
k_0k3
come 5n5
0.3k)
R (b-3)
sep, coev_3x1
(k-1
pool
up come 3s3
(a) Normal Cell
(b) Reduction Cell
Figure 3: Our best searched normal cell and reduction cell.
4.4 Ablation studies
Different positional encoding strategies We compare different positional encoding schemes in Tab.4 on NAS-Bench-LO1 [48], Adjacency option gets the highest score when training data size is 100, Laplacian option obtains highest Kendall's Tau in all other cases. Thus we choose the Laplacian matrix as the positional encoding in our experiments. Interestingly, the Normalized Laplacian option's ' performance is worse than the Laplacian option.

---

Training Samples
100 (0.02%)
172 (0.04%)
424 (0.1%)
424 (0.1%)
4236 (1%)
Test Samples
all
all
100
all
all
Random
0.374
0.366
0.469
0.362
0.365
WL-PE
0,439
0,483
0.509
0.519
0.628
Normalized Adjacency
0.549
0.645
0.685
0.671
0.786
Adjacency
0.620
0.643
0.725
0.693
0,802
Normalized Laplacian
0,592
0.642
0.740
0.697
0,813
Laplacian Eigen-vector
0.414
0.421
0.614
0.524
0.656
Laplacian
0.600
0.669
0.752
0.705
0.820
Table 4: Comparison with different positional encoding strategies. calculate the Kendall's Tau by predicting accuracy of all architectures in NAS-Bench- 101ch-101 and comparing them with ground truths.
Different evaluation numbers We investigate the rela- tionship between the number of evaluations performed on the validation dataset and the final Kendall's s Tau. As shown in Fig.4, when the number of evaluations increases from to 10, we can see noticeable increase of Kendall's s Tau. However, when the number of evaluations contin- ues to grow, the increase of Kendall's Tau seems to be trivial but the training time increases lot. Hence, we choose to evaluate on validation dataset 10 times in our self-evolution framework for all experiments.
0.81
Kandelfs Tou
0.805
10
20
30
40
50
Evalustion numbers
Figure 4: Kendall's Tau versus the ber of validation evaluations.
Ranking results over only good architectures Although there are a large number of architectures the search space, we usually only care about the top-performing ones. Therefore, we compared our [48] using Kendall's s Tau as the metric, over
method with the other two methods on NAS-Bench-ID different top portions of good architectures. Specifically, we only evaluate the good architectures whose ground truth accuracies rank in the top 10%, 20% and 30%. The results are shown in Fig.5. When ranking over the top 10% architectures, NAO [30] and our TNASP get similar results and are better than Neural Predictor [43]. When ranking over more architectures i.e. top 20% and top 30%, TNASP shows steady and better performance compared with NAO [30] and Neural Predictor [43].
0.8
TNABP
0.6
TNASPP
0.6
TNASP NAD
0.5
NAO
NAO
Naure Predictrr
Neumi Predictor
Kendalte Tal
Neuri Predicer
Kendalfe
0.4
Kandall's
0.A
0.4
0--
0.
0.
0.
100
Training
Training
Training
(a) Top 10% architectures
(b) Top 20% architectures
(c) Top 30% architectures
Figure 5; Ranking results over different top portions of good architectures.
Conclusion
In this paper, we propose Transformer-based NAS performance predictor and utilize the linear transformation of the Laplacian matrix as the positional encoding. Our predictor has better encoding ability for spatial topology information, leading state-of-the-an performance several benchmarks. Moreover, we devise a general enough self-evolution framework to further improve our NAS predictor by fully utilizing the temporal information like historical evaluations during training. Unfortunately, we didn't it explore how to involve complicated metrics, for example, non-differentiable metrics, as constraints in our framework. In the future, we will further explore how choose more reasonable and effective constraints to improve the NAS predictors stably and efficiently.
10

---

Acknowledgements
This work supported in part by the National Key R&D Program of China under Grant No. 2018AAA0102701, and in part by the National Natural Science Foundation of China under Grant No. 62176250. Moreover, we would like thank Yu Hu for careful guidance; Longxing Yang, Peng Yao, Yuhang Jiao, and Wentao Zhu for insightful discussions.
References
[1] Mohamed S Abdelfattah, Abhinav Mehrotra, Lukasz Dudziak, and Nicholas D Lane. Zero-cost proxies for lightweight nas. In ICLR, 2021.
[2] Bowen Baker, Otkrist Gupta, Nikhil Naik, and Ramesh Raskar. Designing neural network architectures using reinforcement learning. In ICLR, 2017.
[3] Wuyang Chen, Xinyu Gong, Xianming Liu, Qian Zhang, Yuan Li, and Zhangyang Wang. Fasterseg: Searching for faster real-time semantic segmentation. ICLR, 2020.
[4] Wuyang Chen, Xinyu Gong, and Zhangyang Wang. Neural architecture search on imagenet in four gpu hours: A theoretically inspired perspective. In ICLR, 2021.
[5] Xiangning Chen al. Stabilizing differentiable architecture search via perturbation-basee regularization. ICML, 2020.
[6] Xin Chen, Lingxi Xie, Jun Wu, and Qi Tian. Progressive differentiable architecture search: Bridging the depth gap between search and evaluation. In ICCV, 2019.
[7] Yaofo Chen, Yong Guo, Qi Chen, Minli Li, Yaowei Wang, Wei Zeng, and Mingkui Tan. Contrastive neural architecture search with neural architecture comparators. In CVPR, 2021.
[8] Xiangxiang Chu, Bo Zhang, Ruijun Xu, and Jixiang Li. Fairnas: Rethinking evaluation faimesss of weight sharing neural architecture search. In ICCV, 2021.
[9] Xiangxiang Chu, Tianbao Zhou, Bo Zhang, and Jixiang Li. Fair DARTS: eliminating unfair advantages in differentiable architecture search. In ECCV, 2020.
[10] Kalyanmoy Deb, Amrit Pratap. Sameer Agarwal, and TAMT Meyarivan. A fast and elitist multiobjective genetic algorithm: Nsga-ii. IEEE transactions on evolutionary computation, 6(2):182-197, 2002.
[11] Boyang Deng. Junjie Yan, and Dahua Lin. Peephole: Predicting network performance before training. arXiv preprint arXiv: 17 12.003351, 2017.
[12] Terrance DeVries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint 10XV:11708.04552, 2017.
[13] Shaojin Ding, Tianlong Chen, Xinyu Gong, Weiwei Zha, and Zhangyang Wang. Autospecch: Neural architecture search for speaker recognition. In InterSpeech, 2020.
[14] Xuanyi Dong and Yi Yang. Nas-bench-201: Extending the scope of reproducible neural architecture search. In ICLR, 2020.
[15] Jiemin Fang, Yuzhu Sun, Kangjian Peng. Qian Zhang, Yuan Li, Wenyu Liu, and Xinggang Wang. Fast neural network adaptation via parameter remapping and architecture search. In ICLR, 2020.
[16] Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng. Zechun Liu, Yichen Wei, and Jian Sun. Single path shot neural architecture search with uniform sampling. In ECCV, 2020.
[17] Dongyoon Han et al. Deep pyramidal residual networks. In ICCV, 2017.
[18] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chiel Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, al. Searching for mobilenety3. In ICCV, 2019.
11

---

[19] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR, 2017.
[20] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In ICLR, 2017.
[21] Alex Krizhevsky et al. Imagenet classification with deep convolutional neural networks. Com- munications of the ACM, 60(6):84 90, 2017.
[22] Wei Li et al. Neural graph embedding for neural architecture search. In AAAI, 2021.
[23] Zhihang Li, Teng Xi, Jiankang Deng, Gang Zhang, Shengzhao Wen, and Ran He. Gp-nas: Gaussian process based neural architecture search. In CVPR, 2020.
[24]
Ming Lin, Pichao Wang, Zhenhong Sun, Hesen Chen, Xiuyu Sun, Qi Qian, Hao Li, and Rong Jin. Zen-nas: A zero-shot nas for perh-rrrformeece deep image recognition. In ICCV, 2021.
[25] Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In ECCV, 2018
[26] Hanxiao Liu al. DARTS: Differentiable architecture search. In ICLR, 2019.
[27] Zhichao Lu, Ian Whalen, Vishnu Boddeti, Yashesh Dhebar, Kalyanmoy Deb, Erik Goodman, and Wolfgang Banzhaf. Nsga-net: neural architecture search using multi-objective genetic algorithm. In GECCO, 2019.
[28] Renqian Luo, Xu Tan, Rui Wang. Tao Qin, Enhong Chen, and Tie-Yan Liu. Neural architecture search with gbdt, arXiv preprini wwXivv2007.047880 2020.
[29] Renqian Luo, Xu Tan, Rui Wang. Tao Qin, Enhong Chen, and Tie- Yan Liu. Semi-supervised neural architecture search. In NeurlPS, 2020.
[30] Renqian Luo. Fei Tian, Tao Qin, Enhong Chen, and Tie- Yan Liu. Neural architecture optimiza- tion. In NeurlPS, 2018.
[31] Joseph Mellor, Jack Turner, Amos J. Storkey, and Elliot J. Crowley. Neural architecture search without training. In ICMI., 2021.
[32] Xuefei Ning. Yin Zheng, Tianchen Zhao, Yu Wang, and Huazhong Yang. A generic graph-based neural architecture encoding scheme for predictor- based nas. In ECCV, 2020.
[33] Hieu Pham, Melody Y. Guan, Barret Zoph. Quoc V. Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. In /CML, 2018.
[34] Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe Le. Regularized evolution for image classifier architecture search. In AAAI, 2019.
[35] Binxin Ru, Xingchen Wan, Xiaowen Dong, and Michael Oshorne. Interpretable neural architec- ture search via bayesian optimisation with weisfeiler-lehman kernels, In ICLR, 2021.
[36] Han Shi, Renjie Pi, Hang Xu, Zhenguo Li, James T Kwok. and Tong Zhang. Bridging the gap between sasple-based and one-shot neural architecture search with bonas. In NeurIPS, 2020.
[37] Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In JCML, 2019.
[38] Hidenori Tanaka, Daniel Kunin, Daniel LK Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. In NeurlPS, 2020.
[39] Lucas Theis, Iryna Korshunova, Alykhan Tejani, and Ferenc Huszár. Faster gaze prediction with dense networks and fisher pruning. arXiv preprint arXiv: 005787 2018
[40] Jack Turner, Elliot J Crowley, Michael Boyle, Amos Storkey, and Gavin Gray. Blockswap: Fisher-guided block substitution for network compression budget. In ICLR, 2020.
12

---

[41] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.
[42] Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. In ICLR, 2020.
[43] Wei Wen, Hanxiao Liu, Yiran Chen, Hai Helen Li, Gabriel Bender, and Pieter-Jan Kindermans. Neural predictor for neural architecture search. In ECCV, 2020.
[44] Colin White et al. Bananas: Bayesian optimization with neural architectures for neural architec- ture search. In AAAI, 2021.
[45] Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. SNAS: Stochastic neural architecture search. In ICLR, 2019.
[46] Yixing Xu, Yunhe Wang, Kai Han, Shangling Jui, Chunjing Xu, Qi Tian, and Chang Xu. Renas: Relativistic evaluation of neural architecture search. CVPR, 2021.
[47] Shen Yan, Yu Zheng, Wei Ao, Xiao Zeng, and Mi Zhang. Does unsupervised architecture representation learning help neural architecture search? In NeurlPS, 2020.
[48] Chris Ying, Aaron Klein, Eric Christiansen, Estehan Real, Kevin Murphy, and Frank Hutter. Nas-bench-101: Towards reproducible neural architecture search. In ICML, 2019.
[49] Arber Zela, Thomas Elsken, Tonmoy Saikia, Yassine Marrakehi, Thomas Brox, and Frank Hutter. Understanding and robustifying differentiable architecture search. In ICLR, 2020.
[50] Chris Zhang al. Graph hypernetworks for neural architecture search. In ICLR, 2019.
[51] Muhan Zhang, Shali Jiang, Zhicheng Cui, Roman Garnett, and Yixin Chen. D-vae: A variational autoencoder for directed acyclic graphs. In NeurlPS, 2019.
[52] Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. In ICLR, 2017.
[53] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoe V Le. Learning transferable architectures for scalable image recognition, In CVPR, 2018.
13


Reviews:
[{"review_id": "rphwQGc4UWV", "reviewer": "Reviewer_cAuS", "summary": "This work leverages a transformer-based predictor network to approximate the accuracies of unseen architectures from a search space, for the purpose of NAS. The authors further proposed a ADMM-based method to accelerate the convergence of the predictor training. Experiments are conducted on NAS benchmarks and DARTS spaces.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. I think the idea or graph or transformer-based predictor is reasonable. However, I do not think it is novel enough, given there are already many graph-based predictor works for NAS. I further named a few below that are not cited and compared by the authors.\n2. The description from Eq. 8 to Eq. 13 is clear to me, but the flow char in Fig 2 is a bit confusing and less informative.\n3. The self-evolution (SE) method further introduces extra hyper-parameters, the three $\\eta$s in Eq.11~13. From Table 1 it seems the gain from SE is quite marginal. I am concerned about the cost of extra hyper-param tuning vs. this performance gain.\n\n\n[1] Graph HyperNetworks for Neural Architecture Search\n\n[2] D-VAE: A Variational Autoencoder for Directed Acyclic Graphs\n\n[3] Neural Graph Embedding for Neural Architecture Search\n\n[4] Fitting the Search Space of Weight-sharing NAS with Graph Convolutional Networks", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "My main concern is the limited novelty of transformer-based NAS predictor, and the performance gain from self-evolution given the extra cost of hyper-parameter tuning.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1 hour"}, "extra_text": {}}, {"review_id": "rdKTJFRnUpR", "reviewer": "Reviewer_bjoz", "summary": "This paper proposed a transformer-based NAS performance predictor, accompanying with a Laplacian matrix based positional encoding to represent spatial topology information and a self-evolution mechanism to take advantage of the temporal information such as historical evaluations.", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is well-organized with solid experiments in both the main paper and supplementary material.\n\nStrength:\n+ Using Laplacian matrix  to model spatial topology information is somewhat novel \n+ Good performance on NAS-Bench 101, 201\n\nWeakness: \nThere are some issues needed to be addressed before accepted.\n- The claimed contributions (positional embedding and self-evolution) are closely related to BONAS from NIPS 2020. What is the difference and advantage over BONAS? \n- Is transformer necessary? How about replacing transformer with GCN or MLP?\n- In the ImageNet experiment, the performance gap between TNASP-A,B,C is big while the flops difference is small. What is the reason?  What is the variance between different runs?\n- Potential double blind violation: \"Our proposed method ranked 2nd among all teams in one recent international NAS challenge.\" This is a near miss. I will warn the author not to play this trick next time, if not deck rejected this time.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Please justify the weakness to improve the paper. \n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "mPg6D9-7UWb", "reviewer": "Reviewer_fP1R", "summary": "This paper proposes a transformer-based NAS performance predictor to better encoding the spatial topology information and introduces a self-evolution framework to further improve the performance of the predictor. The proposed method achieves SOTA results on three benchmarks. ", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n(1)\tA transformer-based predictor and a self-evolution framework are proposed, and the experiments on different benchmarks verify the effectiveness of the proposed method. \n\n(2)\tThe self-evolution framework makes full use of the dataset by leveraging the validation loss. When the size of the training dataset is small, it improves the generalization of the model.\n\nCons:\n\n(1)\tThough the author said that the self-evolution framework is practical in machine learning competitions, the test dataset in competitions is usually not provided to users to store the predictions for validation loss, which limits the usage of the proposed framework. \n\n(2)\tThe detailed experiment setting of the transformer-based predictor is not reported. \n\n(3)\tThe author said the core factor of the proposed predictor is the better encoding ability for spatial topology information. However, it is not convincing that Laplacian matrix-based positional encoding does represent topology information. As a comparison, what is the performance of transformer-based predictor associated with random initialized positional embedding? Similarly, the Laplacian matrix is also used by GCN-based predictor which strictly limits the direction of information aggregation by Laplacian matrix. It would be better to provide the experiment results of GCN-based methods.\n\n(4)\tThe main idea of self-evolution framework is similar to the pseudo label technique that SemiNAS adopts. What is the performance of SemiNAS on these benchmarks? \n\n(5)\tWhy is lambda in Eq. (10) must be greater than or equal to 0?\n\n(6)\tHow does the proportion of training set and test set affect the performance of self-evolution framework? \n\n(7)   There are some recent/concurrent works on NAS missing in the related works, such as \"CATE: Computation-aware Neural Architecture Encoding with Transformers\", \"Vision Transformer Architecture Search\" and \"AutoFormer: Searching Transformers for Visual Recognition\". It is suggested to add them into comparison and discussion, especially for CATE. Recent papers on NeurIPS'20 are also not included in the comparison.\n\nTypos:\n(1)\tIn line 127, acrylic may be acyclic.\n(2)\tIn line 128, N may denote the number of nodes, not the connections (edges).\n(3)\tIn line 4 of Algorithm 1, the new constraint may be added into Eq. (8), not Eq. (10)\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}, {"review_id": "Qubdt9dsUj", "reviewer": "Reviewer_8hPB", "summary": "This paper proposed a predictor based on the Transformer architecture to predict the performance of neural architectures in NAS.\nIt proposed to use Laplacian matrix based positional encoding strategy to better represent the topology information to model the neural architectures.\nIt also proposed a model-agnostic self-evolution framework that can fully utilize temporal information as guidance.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Predictor plays an important role in NAS and predictor-based NAS is straight and simple for deploying in practice.\nBesides previous LSTM/GNN based predictors, the authors proposed to use the advanced Transformer architecture for modeling the neural network architectures to predict the performance (e.g., accuracy).\nGiven the experiment results and the announced \"... 2nd among all teams in one recent international NAS challenge\", the methods demonstrate its effectiveness.\n\nStrengths:\n1. The proposed positional encoding seems to bring the benefit, compared to [1] where the authors showed that directly applying Transformer to the task gets bad performance. Does the gain come from the positional encoding?\n\nWeaknesses and questions:\n1. There lacks an introduction of Laplacian matrix but directly uses it. A paper should be self-contained.\n2. The motivation is not strong. The authors stated that \"... the transformer architecture ... outperformed many SOTA models ... motivates us\". This sounds like \"A tool is powerful, then I try the tool on my task, and it works well! Then I publish a paper\". However, it lacks of analysis why Transformer works well on this task, which would bring more insights to the community.\n3. Section 3.3 needs to be polished more on writing.\\\n1 In Eqn. (8), The e^{(t)} is proposed without further explanation. What is it? Why is it needed? What is the motivation of proposing it?\\\n2 In Eqn. (8), are f_\\theta(v_j) and \\hat{y}_j^{(t)} the same thing since they both are the predictions of v_j by the predictor.\\\n3 In line 177-178, if the y_j^{true} is \"user-unkonwn\", then how do you compute e^{(t)} in Enq. (8)?\\\n4 Why this SE framework can help to improve, how does it help? Similar to 2, please DO NOT just show me what you have done and achieved, but also show me why and how you manage to do these. \n\nI would consider increasing the rating based on the authors' response.\n\nReference:\n[1] Luo, et al. \"Neural architecture search with gbdt.\" arXiv preprint arXiv:2007.04785 (2020). https://arxiv.org/abs/2007.04785", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors apply transformer with Laplacian matrix positional encoding as a predictor in predictor based neural architecture search. They also introduce a self-evolution framework to incorporate new data points to improve the performance. They test the model on standard NAS and DARTS benchmarks (collected set of architectures with their performances) and achieve some of the best results. 

Overall the paper is sound. The main issue from the reviewers was if the paper is sufficiently novel to be accepted to neurips and also partially the small size of the NAS and DARTS benchmarks as an evaluation for the neural architecture search. There were also a lot of question to relations to other works and reference that were missing in the paper, however the authors have clarified a lot of the differences from the previous works in the rebuttal.
Decision: Accept (Poster)