TA-GATES: An Encoding Scheme for Neural Network Architectures
Authors: Xuefei Ning
Zixuan Zhou
Junbo Zhao
Tianchen Zhao
Yiping Deng
Changcheng Tang
Shuang Liang
Huazhong Yang
Yu Wang
Venue: NeurIPS 2022 Accept (2022)
Keywords: Architecture Encoding Scheme
Neural architecture search (NAS)
Predictor-based NAS
AutoML

Abstract: Neural architecture search tries to shift the manual design of neural network (NN) architectures to algorithmic design. In these cases, the NN architecture itself can be viewed as data and needs to be modeled. A better modeling could help explore novel architectures automatically and open the black box of automated architecture design. To this end, this work proposes a new encoding scheme for neural architectures, the Training-Analogous Graph-based ArchiTecture Encoding Scheme (TA-GATES). TA-GATES encodes an NN architecture in a way that is analogous to its training. Extensive experiments demonstrate that the flexibility and discriminative power of TA-GATES lead to better modeling of NN architectures. We expect our methodology of explicitly modeling the NN training process to benefit broader automated deep learning systems. The code is available at https://github.com/walkerning/aw_nas.

TA-GATES: An Encoding Scheme for Neural Network Architectures
Xuefei Ning 1 12
Zixuan Zhou
Junbo Zhao'
Tianchen Zhao'
Yiping Deng
Changeheng Tang
Shuang Liang?
Huazhong Yang'
Yu Wang
Department of Electronic Engineering, Tsinghua University
TCS Lab, Huawei?
Novauto Technology Co. Ltd.3
Abstract
Neural architecture search tries to shift the manual design of neural network (NN) architectures to algorithmic design. In these cases, the NN architecture itself can be viewed data and needs to be modeled. A better modeling could help explore novel architectures automatically and open the black box of automated architecture design. To this end, this work proposes new encoding scheme for neural architectures, the Scheme (TA-GATES, TA-GATES encodes an NN architecture in way that is analogous to its training. Extensive experiments demonstrate that the flexibility and discriminative power of TA-GATES lead to better modeling of NN architectures. We expect our methodology of explicitly modeling the NN training process to benefit broader automated deep learning systems. The code is available https wwtp://owwwwwa/ingn.....moa aa__aa
Graph- based ArchiTecture Encoding
Introduction
The past decade has witnessed tremendous advances of deep learning (DL) methods using neural networks (NNs). One of the key driving forces behind NN's widespread applications is the clever design of NN architectures that are both effective and efficient [[3] In [ Nevertheless, long- tail distributed tasks and platforms in the real world pose challenges the scalability of manual architecture design workflows. One pathway to tackle these challenges to construct Automated Deep Learning (AutoDL) systems [38 32) M that can decide factors in all aspects of NN training and inference in an automated and algorithmic way. Neural Architecture Search (NAS) 57 46 is subfield of AutoDL that focuses automated architecture exploration.
As NAS shifts up the architecture design level from manual design to algorithmic design, NN architectures themselves can be viewed as data and need to be modeled. For example, predictor-based constructs a performance predictor consisting of an architecture encoding scheme and a prediction head, and uses its predictions for unseen architectures to guide architecture sampling. A better architecture encoding scheme enables more accurate predictions, and thereby enable us explore the space more efficiently by only sampling promising architectures Moreover, an encoding scheme can also be used to improve the widely-used parameter-sharig evaluation [55] help open the black box of automated architecture design 33 To this end, our work aims at designing a better encoding scheme for NN architectures.
NAS 27
2
42 20
[ 43
of NN architectures are graph- ones that view an architecture as a directed acyclic graph (DAG) of computations. As shown in Fig. T and described
27 -rate-of-the-art (SOTA) encoding schemes S4
"quual contribution.
Corresponding to: foxdoraame gmail.com, wu-wang@@singhuaa.co
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

ArchTectsre
[Information Flow Based) Architecture Encodar
oupprr
Arch Frreedding
NN Architecture
Fecture
(Arch-A)
(Arch-A)
CIITy 2+3
Corr
Comy
Current Arch Encoding Scheme Same Type Op, Same Embedding
X
Different Ops Need Different Encodings
Ops Across Positions
&
Ops Across Archs
Parameters Functions
(Arch-B)
(Arch-8)
Operation entrriding
Figure 1: Motivation illustration. Left: Crate-of-the-an information flow based encoders (architecture input, architecture encoding output) 154 27
In their encoding process, all Conv3x3 share the embedding in O, no matter which architecture or which position the operation is in. Right: NN architecture (data input, feature output). Conv3x3 operations across positions (e.g., the two Conv3x3 in Arch-A) and across architectures (e.g., the first Conv3x3 in Arch-A and Arch-B) obtain different parameters through NN training represent different functions. For example, different from plain Conv3x3, a Conv3x3 with a surrounding skip connection represents residual function.
in Sec. A. these encoding schemes map each type of computation or operation (e.g., MaxPool, Conv3x3) to the same embedding, and the operation type-wise embeddings O are trainable parameters of the encoder. And their encoding process of an architecture DAG could be seen as mimicking of how the architecture handles real data. More concretely, in the real NN forward propagation, the images are put at the input node, flow through the architecture, get processed by each operation, and output feature maps at the final output node. While in the encoding process of GATES (27]. analogically, the input node embedding flows through the architecture DAG, gets multiplied by the embedding corresponding each operation, and outputs the architecture encoding at the final output node. This encoding process mimicking the NN forward propagation is shown to outperform non-graph-based encoding schemes 2X 42
47 and vanilla GCN- schemes [35 [0]
However, these two schemes neglect an important characteristic of NN architectures: an architecture is not a DAG with fixed computations (e.g., sin, cos, MaxPool), but DAG with trainable operations (i.e.. parameterized operations, e.g.. Conv3x3). That to say. operations of the same type (e.g.. two Conv3x3) can represent different functions since they obtain different parameters through NN training. When encoding an architecture, the SOTA encodings [54] 27 make all operations of the same type across different locations and architectures use the same embedding, which does not take account of this characteristic of architectures. Fig. D illlsttrates that we need a scheme that can give contextualized embeddings for operations according to their architectural context.
To realize this, we base our design on the fundamental idea that an NN architecture not only depicts the computations in forward propagation, but also implies its learning dynamics. In view of this, we propose an intuitive method, Craining-Analogoun Graph- ArchiTecture Encoding Scheme (TA-GATES). During the encoding process of an architecture, TA-GATES mimics the learning dynamics of its parameterized operations. This aaaiiingannlogouu modeling enables TA-GATES to better encode NN architectures. We summarize the contributions of this paper as follows.
We propose TA-GATES, novel encoding scheme for NN architectures with analogous modeling of the NN training process. To encode an architecture DAG, TA-GATES conducts an iterative process of forward and backward passes on the DAG for several time steps, and uses the forward output at the final time step as the architecture encoding. In each time step, TA-GATES updates the embeddings of parameterized operations. This encoding process that iteratively updates the operation embeddings could be seen mimicking of the architecture training process that iteratively updates the operation parameters. this way, operations get contextualized embeddings according to their architectural context.

---

To further improve the discriminative power of TA-GATES (see Sec. we propose symmetry-breaking techniques for operation embeddings at the beginning of the encoding process (before the iterative time steps). This technique could be seen as a mimicking of the random parameter initialization in the actual architecture training process.
We apply TA-GATES for performance prediction in various architecture benchmarks. Exper- iments show that TA-GATES consistently surpasses baseline schemes across search spaces, for different tasks (ranking, regression, anytime prediction) and under different settings.
2 Related Work
Automated Deep Learning. In the complex pipeline build DL models, there are many steps and components that require substantial expert knowledge. The need to design DL models for vast tasks and platforms has aroused research interests in AutoDL. NAS S
proposed to automatically design the NN architecture. As for other training-time design choices, hyper-parameter optimization (HPO) [X is long-standing topic that dates back to 1990s 566 AutoAug
46
aims at designing automate the design of loss functions.
data augmentation. And AutoLoss methods [8
Architecture Benchmarks. Topological architecture search spaces can be classified into the operation-on-nodee (OON) ones and the operation-on-edge (OOE) ones. Architectures in OON or OOE spaces have operations on their nodes or edges, respectively. Researchers have established many benchmarks for the ease of comparing NAS methods [51 52 30
36
50 Benchmarks on
[52]. And as for OOE spaces, MAS-Bench-200 201 (NB201) 7 is benchmark a small search space of one cell architec-
OON spaces include NAS- AAS-Bench- 101 (NB101) [51], NAS- Bee-BBenhhheeeee 1Shol
ture. NDS ENAS B0 and MAS-Bench-301 (NB301) [6)] provide benchmarks on larger search
spaces
29
2
of two cell architecture (i.e., the normal and reduce cells).
Predictor-based NAS. A NAS method consists of two components: (1) The evaluation strategy gives out the performance of an architecture, e.g.. by training on the training dataset to get its parame- ters and then testing on the validation dataset [57]. (2) The search strategy samples architectures to evaluate and explores
architectures according to the feedback of the evaluation strategy.
To improve the exploration efficiency the large space, the predictor-based search strategy P 23 trains approximate performance predictor of architectures using some pairs of architecture and ground-truth (GT) performance, and uses its prediction scores unseen architectures to guide architecture sampling. An accurate predictor helps sample architectures more likely to be well- performing and improves NAS results. Lots of efforts have been devoted to developing the architecture encoding scheme, it is essential for the predictor's generalization ability to unseen architectures.
42 20
Existing encoding schemes of NN architectures include non- graph-based ones and graph-based ones. Non-graph-baad based schemes convert the computational DAG into a sequence 23 42 20
Architecture Encoding Schemes.
44 or image [47), and apply XGBoost, Multi-Layer Perceptron (MLP), Long Short- Term Memory (LSTM), Convolutional Neural Network (CNN). They do not explicitly use the graph topology, and have notable weakness such as improper isomorphism handling.
47
22
35
apply plain graph convolutional networks (GCNs) 14 to encode architectures. As plain GCNs cannot applied to OOE spaces directly, to encode architectures in OOE spaces, these methods either adopt the line-graph conversion trick K35
As for graph-based schemes, there are attempts 35. 10
45 or propose ad ad-hoc solutions 110 to convert architectures into OON graphs. NASBOWL proposes to use the WL graph kernel with multiple iterations in the Gaussian Process surrogate. These schemes are not dedicatedly designed for NN architectures. SOTA graph-based schemes, D-VAE computational DAG and follow its "information flow"
[
[
and GATES 27, view an NN architecture to encode it. Their analogous modeling of the NN forward propagation provides better encoding.
Another related work, CATE 49 proposes a seaefformer-baseed encoder. The motivation behind CATE's transformer design to capture "deep contextualized information" of operations. They use seachability-masker attention to aggregate operation embeddings and get contextualized operation embeddings. In this work, we propose a more intrinsical way to get contextualized operation embeddings, which also enable more potentials such anytime prediction.
3

---

3 Training-Analogour GATES
State-of-the -fttt---tttt-tttt-e encoding schemes 64 view NN architecture computational DAG on which the information flows and gets processed. We refer to them as the information flow-based schemes. As shown in Fig. [ (left), in their encoding process, a piece of virtual information (a trainable encoder parameter F: in Alg. D is used as the input node embedding of the architecture DAG. Then, this information flows along the DAG, and when the information arrives the information is transformed according to the embedding of that operation as shown in Equ. A2 and Equ.
operation,
A3 Finally, the information at the output node is adopted the architecture embedding (see also Sec.
A.I. Their encoding process of an architecture is analogous to its data processing.
We notice that they neglect an important characteristic of NN architectures: An NN architecture is not a DAG with fixed computations, but a DAG with trainable operations. These trainable operations parametrized, and their parameters obtained through training process, which implies that two operations of the same type with different parameters represent different functions.
As discussed in Fig. D neglecting this characteristic can result in improper modeling of operations and architectures. there an intrinsical way to solve this issue? Based on the fundamental idea that an NN architecture not only depicts the computations in the forward propagation but also implies the learning dynamics, TA-GATES encodes architectures by mimicking their training process.
In the following, Sec. 3.1 first describes the encoding process of TA-GATES. Then, we elaborate on our proposed "symmetry- breaking technique in Sec. 3.2 And Sec. 3.3 describes how the explicit modeling of the training process can empower the anytime performance prediction task.
3.1 Iterative Encoding in Analogy to Iterative Parameter Training
We give out the encoding process of TA-GATES in Alg. (Line 1-10) and an illustration in Fig. (upper). The learnable parameters and modules of TA-GATES and notations are also summarized in the first two sections in Alg. D We break down the encoding process as follows. First, we get the initial operation embedding embop ori = GetEmbeddingg(O. op) E TR M for all M operations in the
2
DAG according to their operation types (Line 1). Note that O R.N,x is an operation type- sipe-wise embedding matrix, where each row corresponds to one operation type. At this point, operations of the same type have the same embeddings in emb op Then, we apply SymmetryBreakknn on emb oni to
get emb (0) (Line 2), which will be discussed in detail in Sec. 3.2
op
Then, as shown in Fig. (upper left), we conduct an iterative process of forward and backward passes on the DAG a for T' time steps, in which the operation embeddings embop get iteratively updated. This process can be seen as mimicking the iterative parameter updates of the architecture in actual training. In each time step f, we first compute the information flow based GCNs on the architecture DAG a (Line 4-5). Specifically, we use global trainable parameter E as the input information of the architecture DAG (i.e., feed E into the input node of a), and call the InfoPropagation procedure with the current operation embedding emb(t-1). InfoPropagation is described in Sec. A. and implemented following [27 [54] After the forward GCN pass, convert the output information in
op
of the forward pass f by applying FBConvert (an MLP) to get b(0) [N). Then [
[N] used as the input information of the backward GCN pass in Line 7. Note that ((0) [N] is fed into the output node of a, i.e., the input node of aT. We call the two GCN passes in Line 5/7 the forward and backward passes as they propagate on the DAG a and its transposed DAG a", respectively.
After the forward and backward GCN pass in the t-th time step. TA-GATES computes the operation embedding updates using the propagated information in the forward and backward passes (Line 8). Specifically, GerOpEmbUpdian concatenates the operation embedding from the last time step emb(f-")
op 1) and the propagated information f(() ((() in the forward and backward passes the input, feed it into an MLP, and get the operation embedding updates g(t), Then, g(() is updated onto the operation embeddings to get emb(t) (Line 9). And the updated operation embeddings emb!*) (t) will be up
used in the following time step + 1.
After T time steps, the output of the T-th forward InfoPropagation pass (i.e., the node embedding at
the output node) ( [V] is used as the architecture encoding.

---

Algorithm The Encoding Process of TA-GATES
Learnable Parameter Modules of TA-GATES:
E: the input information
W Wh: the parameters of forward and backward GCNs (see Sec. A.1
O E R RNNxdaa the embeddings of N, types of operations
FBConvert: MLP module to convert the forward pass' output the backward pass' input
GerOOpEmbUpdate an MLP module to get updates of operation embeddings
InfoPropagation: the information flow based GCN computation (see Sec. A.
Symmetry Brankttggrr symmetry breaking using parameter-level zero-cost metrics (see Equ.
Other Notations:
T: the number of time steps
N: the number of nodes in a cell architecture
M: the number of operations in cell
No: the number of operation choices in the search space (the subscript o denotes operation)
d, the dimension of operation embeddings (the subscript denotes operation)
emb the operation embeddings at time step t
the information of the n-th node in the forward or backward pass at time step t (n = 1 denotes the input node, n = N denotes the output node)
Iut [n), 1 [n]:
a scale of operation embeddings" updates, set to 0.1 in all experiments except ablation studies
Input:
DAG demoting the NN architecture, a denotes its transposed DAG with all edges reversed } M,, the operation indexes in a
op E {o..... ,No)
Encoding Process:
1: emb up GetEmbeddding(,.cg
2: emb (0)
3: for op = Symmetry Braaking(ebbi
,,... do
4: [ [1] =
5:
fuld [2 .N InfoPropagation [1] emb((")) WT)
op
( [N] F
6:
6 [1: N 1] IntoPropaggtiion(( [N]; a", emb! (1-1) W)))
7:
5(1) = GerOppEabUUddaee(((( (emauueeeetteeeteeee 1 (
8:
9:
emb op -mb + 8(1)
10: end for
Output: f [ [N]
(L/12)1 (T T doing anytime performance prediction)
Analogy to NN training. The analogy between the training process and the TA-GATES's T T-step iterative encoding process of an NN architecture is illustrated Fig. 2 (upper). each time step of TTA-GATES, the forward and backward GCN passes (InfoPropagation) correspond to the NN forward and backward propagation, GerOppmmUpddate correspond to the gradient calculation w.r.t. parameters, and the updates of operation embeddings correspond to parameter updates.
Improving the modeling flexibility. The analogous modeling of NN training adds to the flexibility and discriminability of encoding, makes the embedding of each operation adaptive to the overall architecture (i.e., contextualized embedding). Thus, two operations with different architectural contexts can get distinguishable embeddings, even they are the same type (c.g., the 0- 1/0-2 Conv3x3 in Fig. 2 (upper)), which is more reasonable considering that they represent different functions in the actual NN. In this way, both the operation and architecture encodings become more discriminative, bringing benefits their downstream usage, i.e., architecture encoding for improving predictor- NAS and operation encoding for improving parameter-shane sharing evaluation (55)
Training of TA-GATES. We have described the encoding process (i.c., the inference) of TA- GATES, As for the training of TA-GATES's parameters (listed in the 1st section in Alg. I construct performance predictor consisting of TA-GATES and an MLP head, input the architectures.
and use their GT performances the labels for the predictor outputs MLP()((2) [N]) (see Sec. A.3)
5

---

Architecture Encoding
Dotaset
Architecture Training
Footure
Information
Arch Embedding
Forward Pass on
Forward Propagation
Info
Architecture
Loss
Embeddings
Update
Cara 343
Update Params
Architecture
to cundip
Backward Pass on
Tine stpp
Propagation
Tx Time Steps
Many Iterations
Symmetry Breaking
Cone
(Arch A)
(Arch A)
Introduce Zero-Cost
Same Embedding
Different Arch Equivalent Encoding
Different Arch Different Encading
Info
Info
(Arch B)
(Arch
Figure 2: Upper: The analogy between the training (Right) of an NN architecture with 4 nodes (0/1/2/3 circles) and the TA-GATES's iterative encoding process (Left). Lower: A single forward pass in the encoding process without (Left) / with (Right) the symmetry-breaking technique.
3.2 Symmetry Breaking in Analogy to Random Parameter Initialization
The iterative encoding process distinguish operations with different architectural contexts, even if they are of the same type. However, solely with the iterative encoding process. we cannot distinguish operations with fully symmetric architectural contexts and thus fail discriminate some architectures.
As shown Fig. (lower left), the two architectures are apparently different, and the upper one has larger capacity. However, their information-baut based encodings [54 27 are indistinguishable. As the three Conv3x3 in Arch-A/Arch-B have the same embeddings, and the operation between node and 2 in Arch-B is skip connection, node and in Arch-A and Arch-B B all have the same information r(t) (,(4) Therefore, the final architecture encoding Arch-A and Arch-B B is the same. Even if TA-GATES iteratively updates the operation embeddings in the architecture encoding process,
the embeddings of these Con3x3 operations remain the same across all time steps if their initial embeddings time step 0 are the same, since they have exactly the same paths to the input and output nodes (i.e., at fully symmetrical positions).
So, how to enable the encoding scheme to distinguish operations with fully symmetric architectural contexts? Actually, the reason why the Conv3x3s and the architectures in Fig. 2 (lower left) are not equivalent in NN training is that the random parameter initialization breaks the symmetry of the two Conv3x3 when the training begins. Based on this analysis, we propose to apply symmetry breaking to the initial operation embeddings emb(0) (0) (SymmetryBreakee Breaking in Alg. Line 2) to enable TA-GATES up to distinguish symmetric operations. We propose three types of symmetry -reakkng techniques:
1. Using random noises: The most straightforward way is to add random noises onto emb
2. Using zero-cost saliency metries: Instead of directly injecting randomness into operation em- beddings, we can inject randomness into NN parameters and aggregate some metries of parame- ters in each operation to refine its embedding. We propose to aggregate parameter- wise saliency metrics to break the symmetry operation embeddings. Specifically, SymmetryBreal in the encoding process of architecture first randomly initializes the parameters of the architecture. Then, for each operation, we calculate and aggregate five per-parameter saliency metries as 5-dim vector Z E Mx5 including grad norm, snip [6], grasp [1], plain 25].

---

and synflow 40 And the initial operation embedding emb (0) E IR Xxd, is refined as
emb (0) = emb ori + 3 MZ):
(1)
where the MLP maps from R$ to R and / is fixed scale.
Using zero-cost saliency metries in every time step: We also experiment with a variant of the 2nd technique: Instead of adding symmetry-breaking vector onto the initial operation embedding embap up. we concatenate Z onto emb in all time steps. up
Note that though the last two techniques aggregate parameter-level metrics, they only use randomly initialized parameters and do not require actual parameter training. Borrowing the terminology from zero-cost pruning and NAS 40 I
refer to them "zero-cost symmetry- breaking techniques". As illustrated in Fig. 2 (lower right), the encoding with symmetry breaking can differentiate the two operations at symmetric positions in Arch-A, and thereby the architectures Arch-A and Arch-B.
24
19 26.
3.3 Anytime Performance Training and Prediction
Besides improving the discriminative power of architecture encoding, the explicit modeling of NN training brings other interesting possibilities (see more discussions in Sec. (6. This work explores using TA-GATES to empower anytime performance training and prediction. The meaning of the anytime prediction task two-fold: (1) Training using performances at other epochs might help improve the prediction of final performances, as these supervisory signals bring more information without inducing additional training costs. (2) The task of predicting multiple performances across epochs has its applications, such as providing inspections into the learning dynamics, or making surrogate benchmarks [50
The basic strategy to amend existing encoders for anytime prediction is to make it output multiple scores as the predicted performances for multiple epochs. And as it comes to TA-GATES, we have a more natural choice time step in TA-GATES corresponds to a checkpoint in the NN training process. We'll demonstrate that this natural fit of TA-GATES indeed boosts the predictive power for anytime performances.
output scores of different time steps the predicted performances, each
3.4 Summary
To summarize, TA-GATES has three key steps in analogy with steps in NN training: (1) Zero-cost symmetry breaking of operation embeddings corresponds to the random initialization of parameters. (2) Forward and backward passes GCN correspond to the forward and backward propagation the NN architecture. (3) The updates of operation embeddings correspond to the updates of parameters.
Thanks to the analogous modeling of NN training, TA-GATES has higher flexibility and better discriminative power, and also brings interesting possibilities (see also Sec. 6 For example, TA- GATES provides a more natural and stronger solution. for the anytime performance prediction.
Experiments
We compare the predictive power of TA-GATES with baseline encoding schemes on four search spaces, including NB 52. NB201 M, NB301 H6). and ENAS [29]. Specifically, we use the performances provided by NAS benchmarks as the ground-trutt truth (GT), and split the GT architecture- performance pairs into training and test splits. After training the predictor with (a subset of) the training split, we measure the predictor fitness on the test split.
Following previous work on architecture encoding, we adopt Kendall's Tau (KD)
34 as the main
measure, We also use other measures, including Precision @ (P@K) 27 mean squared average error (MSE), and the Pearson coefficient of linear correlation (LC). We use the MSE regression loss train predictors in anytime training and use the pairwise hinge ranking loss all other experiments. Detailed experimental settings and runtime information can be found in Sec. B and Sec. C.2 and ablation studies can be found in Sec.
PPrecision @ K: The proportion of true top-K architectures in the predicted top-K K architectures.
7

---

Table 1: Kendall's Tau of using different encoders NB101, NB201, NB301 and NDS ENAS, The average result of experiments are reported, and the standard deviation is in the subscript.
Proportions of 7290 truining samples
Encoder
1%
5%
10%
50%
100%
MLP
42
0.531 (a092))
(
0.5703
0.6225(0.008))
0.6307,0.0009
LSTM
0.5476
0.5876
0.6040
0.6196...012)
(o.018)
NB101
GCN (global node)
(.5973 (u366)
0.6927manes)
0.7520
0.7689
NASBOWI.
0.0350
0.64 (,0006)))2
0.6536x0.0193)
0.6833
0.6872.0.000)
SemiNAS
0.527 (0.88)
0.6055(o.0294)
(.5953
0.6043 0.,040
XGBoost
45
004517
0.5987
0.5680p..115))
0.6175 0.56770.0007)
GATES
21
0.6321qao25h
0.7493
0.7690
0.7999
0.81 (9
TA-GATES
0.6686
0.7833 0.7744
(0.003)
(,0033)00053)
0.8217 (0.0057)
Proportions of 7813 training samples
Encoder
0.1%
0.5%
1%
5%
10%
MLP
4P4
0.0162q00850)
(.0863
0. 1.65.uu0)))))
0.3885
0.5492.u.002)
NB201
LSTM
42
0.1935ga 1006)
0.5079
0.569 quo0u
0.6690
0.7395ux061
I Line-Graph GCN
K5
0.246 (0.19)))
0.31 30.0626)
0.40800.0069
0,546 l.0038)
0.609500164)
NASBOWL
13
0,4980
0.6674 00777
0.5912
0. 0.7259 (0.0008)
0.7625 (0.08))
XGBoost
42
0,0706 12388
0.37 19 0se0)
0,4178 (.0088)
0.64 (2000055
0. 0.7084
GATES
0,4309o.uucz
0.6702o.1244)
0.757
0.8583u.0199
0,8823uo024)
TA-GATES
0.5382
0.6707 (000266)
0.773
0.8660
0.8890
Proportions of 5896 training samples
Encoder
0.5%
1%
5%
10%
50%
MLP
0.2750
0,401 8,ouz0
0,5687 0.5373
0.6249mu.uzi)
NB301
LSTM 26
0.5689 (0016)
0.6893
0.7144:0.002)
0.7572.0.019)
GCN 10
0.0951 (0,000)
0,267.3 (.128)
0.2835
0.3179(0.0033)
XGBoost
(.2725
0.3313 0.3059(on25s)
0..227.m.ozi))
0,3461
GATES
0.561 6Qu0251)
0.6064(o.227))
0.69 6110012)
0.7 0,7180
0.7595
TA-GATES
0.5728
0.635 I (00133))
0.7123
0.733
0.7685
Proportions of 500 training samples
Encoder
5.0%
10.0%
25.0%
50.0%
100.0%
MLP 42
0. (60601))
0.3543pp.0199) 0.2264ma0770)
0.3858
0.4141 (o.000)
ENAS
LSTM
0.2594
0.3406
(..55090.077))
0.4875
0.5517 (0.000))
GCN
10
(0,025)
0.3140
0.3367,0.000)
0.3715 0.3508
GATES
27
0.3400
0.4286
0.5274
0.6467 (0.67))
TA-GATES
0.3458,0088
0,4407
0.5485
0.6324
0.6683pp.00o.
4.1
Comparison with Baseline Encoders
Table T shows the Kendal's s Tau on test set using different encoders. Different columns show the results of training using different numbers of architectures. For example, proportion of 1% NB101 means that 1% x 7290=72 architectures are used for predictor training. The comparison of P@ K is shown in Fig. We can that TA-GATES achieves superior ranking quality consistently. We also conduct architecture search experiments using TA-GATES and discuss the results Sec.
4.2 Comparison of Symmetry-Breaking Techniques
As shown in Table 2 except for two exceptions when there only a few training architectures (39/29 training architectures on NB201/NB301), the "Add" symmerry-breaking technique brings improvements on TA-GATES without symmetry breaking ("None"). and achieves the best results. Therefore, we use the "Add" symmetry- breakkn technique without explicit statements.
4.3 Anytime Performance Training and Prediction
Intuitively, the analogous encoding process of TA-GATES has a natural fit for the anytime performance prediction task. We use MSE training loss to train encoders, and show the KDs and regression
measures (L.C and MSE) Table 3 Table A12 Table A13 and Table A14
We can see that
TA-GATES significantly boost the anytime prediction performances.
8

---

1.0
10
1.
14
Prectsion
Precision
Pracision
9.6
MIP
M8P
MIP
L5TH
LSTH
L5TH
GCN
9.2
GATES
GATES
GATES
GATES
62x
(a) NB101
(b) NB201
(c)
NB301
(d) NDS-ENAS
Figure 3: Precision @ K comparison on the validation split of four benchmarks. X-axis: K; Y-axis:
Precision. The training proportion is 5% on NB101, NB201, and NB301, and 50% on NDS ENAS.
Table 2: Kendall's Tau of using different symmetry- breakingg techniques. "None" indicates TA- GATES without symmetry breaking. "Random", "Add", "Concat" refer to TA-GATES with the three respectively.
breaeig techniques described in Sec. 3.2
NB 101 (7290 training)
NB201 (7812 training)
NB301 (S896 training)
Method
1%
5%
10%
50%
0.5%
1%
5%
10%
0.5%
1%
5%
10%
GATES
0.6321
0,7493
0.7690
0.7999
0.6702
0.7571
0.8583
0.8823
0.5616
0.6064
0.6916
0,7180
None
0.6510
0.7581
0.7704
0.8070
0.6838
0.7667
0.8623
0.8866
0.5735
0.6182
0.7020
0.7280
Random
0.6425
0.7612
0,7711
0,8020
0.6777
0.7688
0.8634
0,8836
0,5735
0.6207
0,7034
0.7257
Concat
0.6585
0,7689
0.7819
0,8086
0.6847
0.7708
0.8632
0.8856
0.5659
0.6230
0.7048
0.7261
Add
0.6686
0.7744
0.7839
0.8133
0.6707
0.7731
0.8660
0.8890
0.5728
0.6351
0.7123
0.7331
Table 3: Kendall's ' Tau of anytime training and prediction. Upper/Lower:
Kendall's Tau with the
half GT accuracy in the middle of training the final GT accuracy. Baselines: "Singlee" means to only use one supervisory signal to train a predictor. "Multi-" refers basic strategy described in Sec. 3.3
The predictor outputs multiple scores and is trained with multiple supervisory signals.
the half accuracy
NBIDI (7290 maining)
(7812 mminnng)
(5896 tranning)
Encoder
Training
15p
5%
10%
50%
0.5%
1%
5%5
10%
0.5%
156
5%
10%
Single-GATES
half
0.3636
0.3473
0.3147
0.4796
0,5882
0.6654
0.7317
0.7732
0,2028
0.2106
0.2807
0.3347
Muli-LSTM
half+Ginal
0.0123
0.0659
0.0723
0.3797
0.4771
0.1914
0,1466
0.1157
Multi GATES
half+final
0.2862
0.2912
0.2883
0.1413
0.5827
0.6574
0.7168
0.7745
0.1635
0.1654
0.148]
0.1263
TA-GATES
halfafinal
0
0.4615
0.4805
0.5674
0.6297
0.7110
0.7827
0,8140
0.2345
0.2322
0,3092
0.4249
with final accuracy
NB101 (7290
training)
NB201 (7812
training)
(5896
training)
Encoder
Training
1%
5%
10%
50%
0.5%
1%
5%
10%
0.5%
1%
5%
10%
Single-GAATES
final
0.3856
0.3820
0.5034
0.5903
0.4914
0.6915
0.7237
0.7806
0.1557
0.1549
0.1954
0.2432
Muli-LSTM
halftfinal
.0.0372
0.1028
0.2191
0.1473
0.4166
0.4795
0.5491
0.6062
0.2046
0.2153
0.2283
0.2318
Muli-GATTES
half4rinal
0.3455
0.3341
0.3370
0.1818
0.6145
0.6902
0.7306
0.7089
0.1190
0.1209
0.1025
0.0868
TA-GATES
balf+final
0.5463
0,5850
0,5950
0.6477
0.6648
0.7363
0,8213
0.8624
0.1963
0.2944
0.2928
0.4178
On one hand, TA-GATES enables the predictor to benefit from multiple supervisory signals by capturing the learning speed of architectures. Fig. A shows that a small architecture Arch- 1-A learns faster (higher half accuracy) but ends at low ninal accuracy. and a larger architecture Arch- 1-B learns slower (lower half accuracy) but ends a higher accuracy. TA-GATES can correctly predict the relative order of these two architectures both at the end or in the middle of the training process. In contrast, without explicit modeling of the learning dynamics, same relative order for the half and final accuracies, and thus fails to make correct comparisons.
tends to give out the
On the other hand, directly training baseline encoders with half and final accuracies simultaneously even leads to performance degradation. We observe a "trade-off" phenomenon where the prediction fitness for the half and final accuracy have opposite trends. See Sec. C.5 for detailed analyses.
Note that although our anytime training experiments use 2-step TA-GATES (T=2, t=1 for half and t=2 for final) and use the GT accuracies of two training epochs (half and final) for training, TA-GATES can be easily extended to anytime training with more time steps and more supervisory signals.
5 Conclusions
Neural architectures are data-proessiing DAGs with trainable operations. According to this nature, this work dedicatedly designs Training-Anen Analogous Graph-based ArchiTecture Encoding Scheme

---

(TA-GATES) for encoding NN architectures. The encoding process of TATA--GA TTA-GATES mimics not only how the information is propagated and processed by operations during the NN inference process, but also the learning dynamics of operations during the NN training process. In this way, every operation becomes distinguishable, i.e., gets "contextualized" embeddings according to its architectural context. Extensive experiments in various search spaces show that TA-GATES consistently improves the performance predictions. We also show how the explicit modeling of NN training in TA-GATES empowers the anytime performance prediction task.
6 Broader Applications, Limitations, and Future Directions
Here we discuss two applications that can potentially benefit from the "training-analogous" encoding methodology and one future extension direction of TA-GATES.
TA-GATES the learning curve extrapolator for early-stop NAS. Besides predicting the any- time learning curve from an architecture description, TA-GATES also has the potential to be used as black-box and learnable extrapolator of partial learning curves. An extrapolator be handy for accelerating AutoMI., where some NN training processes can be early stopped according to the extrapolated estimation targeting epochs.
The existing extrapolator 6 assumes the NN learning curve can be described by some parametric function families. And during the training process of an architecture, they fit the function family parameters using the validation performances of the early learning curve, and then extrapolate to get the estimations at targeting epochs. Different from existing extrapolators, TA-GATES can take the architecture description as input. That is to say, during the extrapolation, instead of solely relying on the early learning curve, this learnable extrapolator can utilize the learning curve knowledge of other architectures, and thereby has the potential to make better extrapolations in a data-driven way.
TA-GATES as the joint encoder for other DL factors in AutoML. As TA-GATES explicitly mimics the NN training process, it is easy to extend TA-GATES to model other training-time factors besides the architecture in elegant way. Thus, we expect TA-GATES to find its wide application in joint AutoDL systems, not restricting to NAS.
Here list some possibilities of integrating other factors into TA-GATES: (1) Training time auxiliary towers 39 or architectural reparametrization 5 can seen as introducing additional nodes and edges in the GCN passes when /. < T, while still using the inference-time architecture when / = T. (2) Different data augmentation can be modeled as different conversions of F.. (3) Different loss functions can be modeled as different FBConvert. (4) The influence of training hyper-parameters can be modeled by incorporating them into appropriate places in the encoding process. For example, the learning rate schedule is analogous a schedule of In a word, the methodology of TA-GATES (i.e., analogous black-box modeling of the NN training process) can empower joint AutoDL systems by combining information regarding both the inference-time and training time factors in an intrinsic way. TA-GATES could be used in joint AutoDL systems to help explore the inference-time architecture (NAS), the training- time architecture, the data augmentation (AutoAug), the loss function parameters (oss) dyper-paramett (HPO), and other factors in the complex DL. pipeline.
Enabling TA-GATES to conduct cross-space comparisons. Although TA-GATES can be applied for different search spaces, it's still limited to in-space comparisons. This is because TA-GATES (I) takes the cell DAG instead of the overall architecture DAG as the input and (2) encodes only the variable choices defined by the space instead of all operation properties. Applying the training- analogous methodology of TA-GATES to develop universal cross-space predictor [2] is an interesting future work, which can bridge the gap between zero-cost predictors (can conduct cross- space comparisons, non-satisfying predictions) and data-driven but space-specific predictors (cannot conduct cross-space comparisons, have stronger predictive power).
Acknowledgements
This work was supported by National Natural Science Foundation of China (No. U19B2019, 62171313, 61832007), Beijing National Research Center for Information Science and Technology (BNRist), Tsinghua EE Xilinx AI Research Fund, and Beijing Innovation Center for Future Chips.
10

---

References
[1] Mohamed S. Abdelfattah, Abhinav Mehrotra, Lukasz. Dudziak, and Nicholas D. Lane. Zero- Cost Proxies for Lightweight NAS. In International Conference on Learning Representations (ICLR), 2021.
[2] Wuyang Chen, Xinyu Gong, and Zhangyang Wang. Neural architecture search on imagenet in four gpu hours: A theoretically inspired perspective. In International Conference on Learning Representations (ICLR), 2020.
[3] Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoe V Le. Autoaugment: Learning augmentation policies from data. IEEE International Conference on Computer Vision (ICCV), pages 1 113-123, 2019.
[4] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. In IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 702-703, 2020.
[5] Xiaohan Ding, Xiangyu Zhang, Ningning Ma, Jungong Han, Guiguang Ding, and Jian Sun. Repvgg: Making vgg-style convnets great again. In IEEE Conference on Computer Vision and Pattem Recognition (CVPR), pages 13733 13742, 2021.
[6] Tobias Domhan, Jost Tobias Springenberg, and Frank Hutter. Speeding up automatic hy- perparameter optimization of deep neural networks by extrapolation of learning curves. In International Joint Conference on Artificial Intelligence (IJCAI), 2015.
[7] Xuanyi Dong and Yi Yang. Nas-bench-2011 Extending the scope of reproducible neural architecture search. In International Conference on Learning Representations (ICLR), 2020.
[8] Thomas Elsken, Jan Hendrik Metzen, Frank Hutter, al. Neural architecture search: A survey. Journal of Machine Learning Research (JMLR), 20(55): 2019.
[9] Daniel Golovin, Benjamin Solnik, Subhodeep Moitra, Greg Kochanski, John Karro, and David Sculley. Google vizier: A service for black-box optimization. In SIGKDD international conference on knowledge discovery and data mining, pages 1487-1495, 2017.
I Proceedings of the 23rd ACM
[10] Yong Guo, Yin Zheng, Mingkui Tan, Qi Chen, Jian Chen, Peilin Zhao, and Junzhou Huang. Nat: Neural architecture transformer for accurate and compact architectures. In Annual Conference on Neural Information Processing Systems (NeurIPS), pages 735-747, 2019.
[11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
[12] Daniel Hesslow and lacopo Poli. Contrastive embeddings for neural architectures. arXiv preprint 102:00200,.44288, 2021.
[13] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprini 1xXiiv/704.04800 2017.
[14] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations (ICLR),201)
[15] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Annual Conference on Neural Information Processing Systems (NIPS), volume 25, 2012.
[16] Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr. Snip: Single-shot network pruning based on connection sensitivity. In International Conference on Learning Representations (ICLR), 2019.

---

[17] Chuming Li, Xin Yuan, Chen Lin. Minghao Guo, Wei Wu, Junjie Yan, and Wanli Ouyang. Am-lfs: Automl for loss function search. In IEEE International Conference on Computer Vision (ICCV), pages 8410-841 8419, 2019.
[18] Hao Li, Chenxin Tao, Xizhou Zhu, Xiaogang Wang, Gao Huang, and Jifeng Dai. Auto seg- loss: Searching metric surrogates for semantic segmentation. In International Conference on Learning Representations (ICIR), 2020.
[19] Ming Lin, Pichao Wang, Zhenhong Sun, Hesen Chen, Xiuyu Sun, Qi Qian, Hao Li, and Rong Jin. nen-nas: zero-shot nas for perfomrrrrmee deep image recognition. In IEEE International Conference on Computer Vision (ICCV), pages 347-356, 2021.
[20] Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In European Conference on Computer Vision (ECCV), pages 19-34, 2018.
[21] Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint arXiv: 1806.0905..5555 2018
[22] Renqian Luo, Xu Tan, Rui Wang, Tao Qin, Enhong Chen, and Tie- Yan Liu. Semi-supervised neural architecture search. Annual Conference on Neural Information Processing Systems (NeurIPS), 33:10547-10557, 2020.
[23] Renqian Luo, Fei Tian, Tao Qin, Enhong Chen, and Tie-Yan Liu. Neural architecture opti- mization. In Annual Conference on Neural Information Processing Systems (NeurIPS), pages 7816-7827. Curran Associates, Inc., 2018.
[24] Joseph Mellor, Jack Turner, Amos Storkey, and Elliot J. Crowley. Neural architecture search without training. In International Conference on Machine Learning (ICML), 2021.
[25] Michael C Mozer and Paul Smolensky. Skeletonization: A technique for trimming the fat from a network via relevance assessment. In Annual Conference on Neural Information Processing Systems (NIPS), pages 107-115, 1989.
[26] Xuefei Ning, Changcheng Tang, Wenshuo Li, Zixuan Zhou, Shuang Liang. Huazhong Yang, and Yu Wang. Evaluating efficient performance estimators of neural architectures. In Annual Conference on Neural Information Processing Systems (NeurIPS), volume 34, pages 12265-
12277, 2021.
[27] Xuefei Ning, Yin Zheng, Tianchen Zhao, Yu Wang, and Huazhong Yang. A generic graph-based neural architecture encoding scheme for predictor-based nas. In European Conference Computer Vision (ECCV), pages 189-204, 2020.
[28] Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yong Guo, Peilin Zhao, Junzhou Huang, and Mingkui Tan. Disturbance-immune weight sharing for neural architecture search. Neural Networks, 144:553-564, 2021.
[29] Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. Efficient neural architecture search via parameters sharing. In International Conference on Machine Learning (ICML), pages 4095-4104. PMLR, 2018
[30] Ilija Radosavovic, Justin Johnson, Saining Xie, Wan-Yen Lo. and Piotr Dollár. On network design spaces for visual recognition. In IEEE International Conference on Computer Vision (ICCV), pages 1882-1890, 2019.
[31] Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe Le. Regularized evolution for image classifier architecture search. In AAAI Conference on Artificial Intelligence, volume 33, pages 4789 2019.
[32] Esteban Real, Chen Liang, David So, and Quoc Le. Automl-zero: Exolving machine learning algorithms from scratch. In International Conference on Machine Learning (ICMI., pages 8007-8019. PMLR, 2020.
12

---

[33] Binxin Ru, Xingchen Wan, Xiaowen Dong, and Michael Osborn.. Interpretable neural ar- chitecture search via bayesian optimisation with weisfeiler-Ichma kernels. In International Conference on Learning Representations (ICIR), 2021.
[34] Pranab Kumar Sen. Estimates of the regression coefficient based on kendall's tau. Journal of the American Statistical Association, 637324):1379-1339, 1968.
[35] Han Shi, Renjie Pi, Hang Xu, Zhenguo Li, James Kwok, and Tong Zhang. Bridging the gap between sample-based based and one-shot neural architecture search with bonas. Annual Conference on Neural Information Processing Systems (NeurIPS), 33, 2020.
[36] Julien Siems, Lucas Zimmer, Arber Zela, Jovita Lukasik, Margret Keuper, and Frank Hutter. Nas-bench-301 and the case for surrogate benchmarks for neural architecture search. arXiv preprint 20X1:20088,0977,, 2020.
[37] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409, 1556, 2014.
[38] Jasper Snoek, Hugo Larohelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. Annual Conference on Neural Information Processing Systems (NIPS), 25, 2012.
[39] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pages 1-9, 2015.
[40] Hidenori Tanaka, Daniel Kunin, Daniel L Yamins, and Surya Ganguli. Pruning neural net- works without any data by iteratively conserving synaptic flow. Annual Conference on Neural Information Processing Systems (NeurIPS), 33, 2020.
[41] Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. In International Conference on Learning Representations (ICLR), 2020.
[42] Linnan Wang, Yiyang Zhao, Yuu Jinnai, and Rodrigo Fonseca. Alphax: exploring neu- ral architectures with deep neural networks and monte carlo tree search. arXiv preprint 2018.
[43] Colin White, Willie Neiswanger, Sam Nolen, and Yash Savani. A study on encodings for neural architecture search. Annual Conference on Neural Information Processing Systems (NeurIPS), 23:20909-20319, 2020.
[44] Colin White, Arber Zela, Robin Ru, Yang Liu, and Frank Hutter. How powerful are performance predictors in neural architecture search? Annual Conference on Neural Information Processing Systems (NeurIPS), 34:28454-28461 2021.
[45] Wikipedia contributors. Line graph Wikipedia, the free encyclopedia, 2004. [Online; accessed 22-July-2004).
[46] Lingxi Xie, Xin Chen, Kaifeng Bi, Longhui Wei, Yuhui Xu, Lanfei Wang. Zhengsu Chen, An Xiao, Jianlong Chang, Xiaopeng Zhang, and Qi Tian. Weight-sharing neural architecture search: A battle to shrink the optimization gap. ACM Computing Surveys, 54(9), oct 2021.
[47] Yixing Xu, Yunhe Wang, Kai Han, Yehui Tang, Shangling Jui, Chunjing Xu, and Chang Xu. Renas: Relativistic evaluation of neural architecture search. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 441 1-4400, 2021.
[48] Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Guo-Jun Qi, Qi Tian, and Hongkai Xiong. Pe-darts: Partial channel connections for memory-effcient architecture search. International Conference on Learning Representations (ICLR), 2019.
[49] Shen Yan, Kaiqiang Song, Fei Liu, and Mi Zhang. Cate: Computation-awanl neural architecture encoding with transformers. In International Conference on Machine Learning (ICML.), pages 11670-11681. PMLR, 2021.
13

---

[50] Shen Yan, Colin White, Yash Savani, and Frank Hutter. Nas-bench-xl1 11 and the power of learning curves. Annual Conference on Neural Information Processing Systems (NeurIPS), 34, 2021.
[51] Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter. mas-bench-101: Towards reproducible neural architecture search. In International Conference on Machine Learning (ICML), pages 7105-7114 7114. PMLR, 2019.
[52] Arber Zela, Julien Siems, and Frank Hutter. Nas-bench- Ishotl: Benchmarking and dissecting one-shot neural architecture search. In International Conference on Learning Representations (ICLR), 2019.
[53] Chris Zhang, Mengye Ren, and Raquel Urtasun. Graph hypernetworks for neural architecture search. In International Conference on Learning Representations (ICLR), 2019.
[54] Muhan Zhang, Shali Jiang, Zhicheng Cui, Roman Garnett, and Yixin Chen. D-vae: varia- tional autoencoder for directed acyelic graphs. Annual Conference on Neural Information Processing Systems (NeurIPS), pages 1588-1600, 2019.
[55] Zixuan Zhou, Xuefei Ning, Yi Cai, Jiashu Han, Yiping Deng, Yuhan Dong, Huazhong Yang, and Yu Wang. Close: Curriculum learning on the sharing extent towards better one-shot nas. In European Conference Computer Vision (ECCV), 2022.
[56] Marc-André Zöller and Marco F Huber. Benchmark and survey of automated machine learning frameworks. Journal of Artificial Intelligence Research, 70:409-472, 2021.
[57] Barret Zoph and Quoe V. Le. Neural architecture search with reinforcement learning. In International Conference on Learning Representations (ICLR), 2017.
[58] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 8697-8710, 2018
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "lT8WhkYKKlw", "reviewer": "Reviewer_64UJ", "summary": "This paper recommends a new GCN encoder based performance predictor built on top of the GATES encoder/predictor. While most encoders assign the same embedding for a given operation irrespective of their position in the architecture, TA-GATES proposes a predictor that captures the training process of the architecture and assigns different embeddings for a given operation.\n\nGiven an adjacency matrix representation of the architecture $a$ and the input information E, for each time step, it computes the information flow of $a$ through a GCN similar to the GATES paper. This simulates the forward propagation of NN training. Then this information is transformed by passing through an MLP and  the resulting output along with $a^{T}$ is in turn passed through the GCN with different weight parameters for information propagation. To simulate backpropagation of the training, they compute derivates and use chaining rule in their second information propagation. The output of the backward propagation from the final timestep is the architecture embedding. Similar to GATES, they have a pairwise ranking loss and MSE loss. But rather than predicting the final accuracy, TA-GATES predicts the accuracy for every half epoch and full epoch time steps. This is seamlessly learnt because TA-GATES is trained for every time-step.\n\nFurther, to avoid the operations from having the same embeddings initially, they use some symmetric breaking techniques such as (1) adding random noise (2) add an vector comprised of zero-cost metrics such as grad_norm, snip etc. only initially or at every time step\n\n", "questions": "1) For table 1 and Table 3, Please compare against more recent performance predictors for some search spaces at least. [1] has open sourced several performance predictors. XGBoost seems to be competitive.\n\n2) For Table1, please include the kendall Tau metrics for 50% and 100% for NB201 and 100% for NB301. \n\n3) Similarly, please include Precision @ K for all the predictors when predicted on 50% and 100% of the training data.\n\n4) For Table3 also, it would be good to know how Multi-Gates fares against TA-Gates when trained on 100% of the training data.\n\n5) For Table A5, please list the accuracies of the architectures found when the predictors are trained on 100% of the training data.\n\n[1]How Powerful are Performance Predictors in Neural Architecture Search?, White et al.", "limitations": "See the weakness and questions section.\n\nWhile the idea of training the surrogate model that captures the training dynamics of the model is an interesting idea, it would be good to understand if it is really essential. Early stopping methods, some of which are listed in [1] are able to predict the accuracies at a given epoch by just extrapolating the learning curves. For your accuracy prediction at half of the training epochs, it would be compare against those for Table 3.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nTA-GATES is able to predict the accuracy for every epoch and not just the final accuracy. So in addition to using it as standalone for NAS, this would be useful to select models that perform well by training for fewer epochs where we have limited training budget.\nTA-GATES is able to out perform all the other baselines when trained on limited data.\n\nWeakness:\nDespite their new training strategy and improved correlation metrics, TA-GATES is not able to rate the top-k architectures significantly better than GATES as indicated by Precision @ K.\nThe range of accuracies of architectures found in NDS-Darts is a lot lesser than NAS-Bench 201. So in general, it would be good to demonstrate that the surrogate models work effectively there. NAO was also initially developed on the DARTS search space. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "dgQdQ_x_pdb", "reviewer": "Reviewer_zsxx", "summary": "The authors present a novel encoding scheme for neural network architectures based on the GATES method. They propose a significant change to the GATES approach that trains a different embedding for each of the operations even if they are of the same type. To achieve this they present a series of methods for symmetry breaking that allow the embeddings for each operation to vary independently. They perform an exhaustive series of experiments verifying and ablating their claims to show that their approach TA-GATES is indeed superior to the alternative encoding approaches they consider.", "questions": "See the Weaknesses part of the Strengths and Weaknesses section.", "limitations": "The authors have adequately addressed the limitations of their work.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n- This is a well-written paper. The prose is clear and the arguments are comprehensible. Enough detail is provided to understand the approach suggested and to understand the experimental setup.\n- The extensive experiments seem to indicate that indeed TA-GATES is a superior embedding to the suggested alternatives.\n- The area explored by the paper is extremely relevant as having better encodings for architectures can lead to better downstream models and better analysis of architectures based on the encodings. \n- The authors show the added benefits of TA-GATES for anytime performance evaluation that can be extremely useful when partial learning curve information is required.\n\n# Weaknesses\n- They do not reference other NN encoding schemes, like those suggested in [1].\n- Their argument that the architecture should include variation based on the specific parameters of the model seems suspect. While clearly using different embeddings for the same operations does seem to work well, the argument that the architecture should incorporate this seems incorrect since most of the time the specific parameters of the operation do not seem to be relevant when designing architectures.\n- The authors don't include the runtime of TA-GATES. While I assume that the runtime is comparable to GATES it would help to know the specific run times of their approach and compare it to the other approaches\n\n# References\n[1] White, C., Neiswanger, W., Nolen, S., & Savani, Y. (2020). A study on encodings for neural architecture search. Advances in Neural Information Processing Systems, 33, 20309-20319.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "LXxr3fNhfIA", "reviewer": "Reviewer_dXbo", "summary": "The authors consider encoding schemes for neural architecture search (NAS). While current encodings model the data flow through the architecture operations by encoding  each operation, the current authors propose a new encoding based on the idea that different operations need different encodings, since each operation has its own separate weights that are learned throughout architecture training. Specifically, they perform multiple forward and backward passes on the architecture and output the trained embeddings as the encoding. They also give symmetry-breaking techniques (similar idea as random initialization). They test their technique on NAS-Bench-101, 201, 301, and ENAS.", "questions": "Can the authors show the usefulness of TA-GATES in NAS by running full NAS experiments? Can the authors compare to more encoding-based methods such as the examples above?\n\nCan the authors include runtimes, CPU/GPU information, limitations/societal impact?", "limitations": "The authors did not discuss societal impact of their work, and I also did not see a discussion on limitations. Remember that the NeurIPS page limit was increased from 8 to 9 pages to give space for the questions in the author checklist, so these sections should be taken more seriously.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\nIntuitive and well-performing. The TA-GATES intuition, i.e., operations should not be encoded the same way because they have trainable parameters, makes sense, and TA-GATES is an intuitive solution to this. The symmetry-breaking technique also makes sense, and overall these techniques are shown to achieve strong performance on four benchmarks.\n\nThey released code with a detailed README, so reproducibility is high.\n\nGood ablations: ablation for number of time steps of TA-GATES, and ablation for symmetry-breaking technique.\n\nThe anytime performance prediction is a very interesting application of TA-GATES.\n\n### Weaknesses\n\nNo NAS experiments. All of the experiments compute rank correlation or precision for performance prediction. To show the usefulness of TA-GATES in NAS, the authors should run full NAS experiments. \n\nSimilarly, the method is only compared to four other performance prediction models: MLP, LSTM, GCN, and GATES. But, the authors could include even more. For example, there are other encoding-based works such as\n“NASBOWL”, “Contrastive Embeddings for Neural Architectures”, “Does Unsupervised Architecture Representation Learning Help Neural Architecture Search?”, “CATE”, and. Some of these works learn architecture embeddings, which is similar in spirit to TA-GATES, so it is important to discuss and compare to them.\n\nThe specific TA-GATES approach seems to be specific for each search space (based on all of the details in Section B.3) so it would need to be tuned for each new search space.\n\nRepeatedly refer to GATES and D-VAE as state-of-the-art techniques. But these have been out for more than two years. There are other methods now, such as the ones listed in the second point above.\n\nThe runtime of their method should be included in the main body, as well as total CPU and/or GPU usage.\n\nNo discussion on limitations or societal impact (see below).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept