Learning to Flow from Generative Pretext Tasks for Neural Architecture Encoding
Authors: Sunwoo Kim
Hyunjin Hwang
Kijung Shin
Venue: NeurIPS 2025 poster ()
Keywords: Graph representation learning
Neural architecture encoding
Self-supervised learning
Primary area: general_machine_learning
TL;DR: We propose a graph-based unsupervised pre-training method for neural architecture encoding.

Abstract: The performance of a deep learning model on a specific task and dataset depends heavily on its neural architecture, motivating considerable efforts to rapidly and accurately identify architectures suited to the target task and dataset. To achieve this, researchers use machine learning models—typically neural architecture encoders—to predict the performance of a neural architecture. Many state-of-the-art encoders aim to capture information flow within a neural architecture, which reflects how information moves through the forward pass and backpropagation, via a specialized model structure. However, due to their complicated structures, these flow-based encoders are significantly slower to process neural architectures compared to simpler encoders, presenting a notable practical challenge. To address this, we propose FGP, a novel pre-training method for neural architecture encoding that trains an encoder to capture the information flow without requiring specialized model structures. FGP trains an encoder to reconstruct a flow surrogate, our proposed representation of the neural architecture's information flow. Our experiments show that FGP boosts encoder performance by up to 106\% in Precision@1\%, compared to the same encoder trained solely with supervised learning.

Learning to Flow from Generative Pretext Tasks for Neural Architecture Encoding
Sunwoo Kim', Hyunjin Hwang', Kijung Shin
Kim Jaechul Graduate School of AI, Schooo of Electrical Engineering Korea Advanced Institute of Science and Technology (KAIST) [kswoo97, julia510, kijungs} @ kaist. ac. kr
Abstract
The performance of a deep learning model specific task and dataset depends heavily on its neural architecture, motivating considerable efforts to rapidly and accurately identify architectures suited to the target task and dataset. To achieve this, researchers use machine learning myyecslyy neural architecture en- coders-to predict the performance of neural architecture. Many state-of-the-art encoders aim to capture information flow within a neural architecture, which re- flects how information moves through the forward pass and backpropagation, via a specialized model structure. However, due to their complicated structures, these flow-based encoders are significantly slower to process neural architectures com- pared to simpler encoders, presenting notable practical challenge. To address this, we propose FGP, a novel pre- pre-training method for neural architecture encoding that trains an encoder to capture the information flow without requiring specialized model structures. FGP trains an encoder to reconstruct a flow surrogate, our pro- posed representation of the neural architecture's 's information flow. Our experiments show that FGP boosts encoder performance by up to 106% Precision %, compared to the same encoder trained solely with supervised learning.
Introduction
Deep learning has achieved outstanding performance across diverse machine learning tasks, includ- ing those in computer vision 57J. However, particular neural architecture (i.e., deep-learning model) that performs well on one task or dataset may underperform on others, highlighting the importance of choosing an architecture suited to the target task and dataset [29, 11, 35], A naive ap- proach-exhaustive exhaustively training and evaluating each neural architecture inncus substantial costs since each architecture requires expensive computational resources and time for training and evaluation on the target task and dataset.
promising approach for alleviating expensive training and evaluation costs is machine learning techniques to predict the performance of neural architectures. Due to their effectiveness, performance predictors widely leveraged for neural architecture search find good neural architectures rapidly and accurately [10, 45, 16, 33J. Recently, various neural architecture encoders, which are typically deep learning models, have been developed be used for performance predictors [34, 49, 15, 32].
The focus many state-of-the-art neural architecture encoders [34, 15) lies capturing the infor- mation flow within neural architecture- concept that describes how input data is propagated in the forward pass and how gradients flow during the backpropagation. To capture the information flow, these flow-based encoders complex model structures, especially graph neural networks with asynchronous message passing [40], specially designed to capture the flow within an architecture.
Challenge 1. However, these flow-based encoders face a practical challenge related to efficiency. The complicated structures of flow-based encoders result in significantly longer processing times
39th Conference on Neural Information Processing Systems (NeurIPS 2025).

---

Performane prodiction bask resulls (meliic: Kandall's tow..
Flow-besed encader (FlowerFormer)
(ReeGatedGCN)
Non-flow
Generative pretext on a molecule
Generative pretext task on neural architecture
higher the better)
78
x57
76
70
-14%
-5%
3x33
74
65
NAS-Bench-101
NAS-Bunch-201 dataset
1x1
3x33
10-
10
10
10a
In
Noural architecture processing time (sec)
(a) The total processing time FlowerFormer [15] and ResGatedGCN [3] for the 14K and 15K architectures in
(b) Example generative pretext tasks: deep learning models are trained to predict the masked atoms operations (colored in gray).
Nas-Bench- 10 and NAS- DAS-Bench-201. respectively.
Figure 1: Remaining challenges in neural architecture encoding. Challenge 1: Regarding model structures, as shown in (a), flow-based encoders, though more effective, are considerably slower than non-flow-basee encoders. Challenge 2: Regarding pretext objectives, in (b), while deep learning model learns chemical rules by identifying masked atoms in a molecule, it gains no clear training guidance when identifying masked operations in neural architecture, where most operations are possible options due to the absence of explicit rules.
compared to the simpler structures of non- non-flow-based encoders. particular, flow-based encoders sequentially perform message passing according to the graph's 's topological order (see Section 3.2. 1), instead of computing all messages simultaneously as in non-floo-basse encoders; this sequential process results in significantly greater computation time. Specifically,
shown in Figure (a), a flow-based encoder (spec., FlowerFormer [15]) is up to 57 times slower than non-flow-basee encoders [3, 47] (refer to Appendix B.1 for further results). Since performance predictors aim to quickly identify effective neural architectures, this slower processing time poses a considerable practical bottleneck when using flow-based encoders.
Another line research focuses on the effective pre-training of neural architecture encoders. Neural architecture encoders require sufficient training data- architectures paired with their ground-truth performance on the target task and dataset-since deep learning models are often vulnerable to the overfitting issue. However, obtaining enough neural architectures along with their performance entails heavy costs. To address this, various generative pre-training methods for neural architecture encoding have been explored [48, 17J. Note that in various domains, it is known that by using well-designed generative pretext tasks, deep learning models can capture data patterns [12, 6, 13, 19) that lead to performance improvements in label-scarce scenarios [55, 53, 18].
Challenge 2. However, existing generative pre-traue pre-training methods, which are often adapted from other domains [48, 17], may be sub-optimal for neural architecture encoding, Consider generative pretext task that involves predicting masked parts the input, used for both neural architectures [17] and molecules [13]. In Figure 1 (b), only a limited set of atoms can occupy the masked area due to chemical rules, which deep learning models learn through this task. In contrast, such rules are largely absent in neural architectures; as shown in Figure (b), most operations possible options for the masked part. This characteristic makes it unclear what specific advantage the model gains from this generative pretext task in the context of neural architecture.
To address both key challenges, we propose novel and specialized generative pre-training method for neural architecture encoding, FGP (Flow-based Generative Pre- Pre-trainne training). FGP enables an encoder to capture the information flow within neural architecture, even if the encoder structure is not specially designed to capture the information flow (e.g., the non-flow-based encoders [47, 3]). We train an encoder to reconstruct flow surrogale, a proposed representation of a neural architecture 's information flow, allowing the encoder to capture the information flow of the architecture. FGP addresses the two practical challenges outlined carlier: (1) it enables encoders to become flow-aware without requiring specialized model structure that would increase computational time significantly, and (2) it provides clear training guidance to the encoder, allowing it to effectively learn the information flow within a neural architecture through our pretext task.
We demonstrate the effectiveness of our pre-training method compared baseline pre-training methods across multiple downstream tasks, including performance prediction and neural architecture search. Specifically, compared to the strongest baseline method [56], FGP shows up to 46.5% gain in terms of Precision @ I % in performance prediction. Our key contributions are as follows:

---

We propose a novel generative are-training method, FGP, for neural architecture encoding. FGP trains a neural architecture encoder to learn the information flow within neural architecture even without specialized encoder structure (Section 3).
the performance prediction experiment, FGP outperforms the baseline pre-training methods in 23 out of 27 settings, demonstrating its efficacy (Section 4.2).
In the neural architecture search (NAS) experiment-a key application of neural architecture encoders- FGP outperforms the baseline pre-training methods (Section 4.3).
Our code and datasets are available at https:
cow/kswww37
2 Related Work and Preliminaries
In this section, we first review related studies, followed by an outline of the preliminary concepts essential to our study.
2.1
Related work
2.1.1
Neural architecture encoding.
Neural architecture encoding aims learn good representations of neural architectures [49, 15, 34, 32, 50]. Its notable application is neural architecture performance prediction, which can reduce the cost of finding proper neural architecture for the target task and dataset.
Neural architectures, including Transformers [5] and convolutional neural networks [51], can be expressed graphs where nodes represent operations (e.g.. 3x3 conv and max pooling) and edges represent connections between them. Therefore, graph neural networks (GNNs) [3, 47, 23] are widely used neural architecture encoders [42, 43, 48, 4, 10]. GNNs encode neural architecture graph by aggregating and transforming features from the neighborhood of each node.
Recent works have increasingly focused on capturing the information flow within given neural architecture by using specialized model structures [32, 34, 15]. A representative example is Flow- erFormer [15]. state-of-the-art neural architecture encoder that employs sequential information processing to imitate the forward- -pass and backpropagation in neural architecture. Although more effective, these flow-based encoders require significantly more processing time, due to their sequential processing, compared to simpler GNN-based encoders (refer to Figure (a)).
2.1.2 Pre-training for neural architecture encoding.
Due to the high cost of obtaining neural architecture's ground-truth performance on the target task and dataset (i.e., the architecture's 'label'), it is essential train accurate performance pre- dictors dictors-typice typically using neural architecture encoders in label-scarce scenarios, as discussed in Section 1. To this end, generative pre-training methods [22, 12], known for enhancing the generaliza- tion capabilities of deep learning models [55, 53, 18], have been adopted. These methods leverage unlabeled neural architectures (i.e., those with unknown ground-truti performance) pre-train neural architecture encoder and fine-tune the encoder the target downstream task, such as the performance prediction tasks [48, 17].
Arch2vec [48] is based on variational graph autoencoder [22], which trains neural architecture encoder to reconstruct edges in graphs representing neural architectures. GMAE [17] builds on masked autoencoder [12], where certain operations within neural architecture are masked, and the encoder trained predict these masked operations.
In contrast, Zhao et al. [56] leveraged zero-cost proxies [1, 25, 14), which are neural architecture's statistics correlated with its ground-truth performance. They trained neural architecture encoder to predict the zero-cost proxies of neural architecture, enabling it to learn which types of architectures more likely to yield high performance.
2.2 Preliminary
Neural architecture graph. A neural architecture is often modeled directed acyelic graph G (V, E), defined by set of nodes V {U1,".. fllis that represent operations, and set of directed
3

---

edges 8 C {V), Uy): 04, v E V) that represent connections between consecutive operations (refer to Figure 2 for an example). Specifically, when operation v, follows operation v, the two are connected by directed edge Uy), with v the source and vj as the destination. Fach node is assigned a feature vector, typieally a one-hot encoded vector indicating its operation type (refer to Figure 2). We denote the feature vector matrix as X E {O, 1 M where O represents the set of all possible operations. The i-th row of X. denoted by X.,,, corresponds to the feature vector of node U,. Therefore, neu- ral architecture can also be expressed as = EX,
Edges (E)
(vy, (v,, s.). (y,, (Pt, vyh (12, vel (Py.P4), (Ps, Ps), (Pa, (Ps, Pal Pt)
4 out
3x3
Node features (X)
in
1x1
3x3
mp out
1x1
V1
122
0
Neural architecture encoders. Neural ar- chitecture encoders transform a neural ar- chitecture graph 9 = (X, E) into its vector representation (i.e., embedding). Formally, given G as an input, a neural architecture encoder fa produces vector z E Rd (i.e.. fo(X. E) = z). For encoders that give em- beddings for each operation [47, 15] (i.e., E) Z' E II we apply mean pooling to obtain z, as in [15].
1x1
1/3
0
1
0
0
0
14
0
0
0
1
0
1x1
1/2
0
1
0
0 0
0 0
11
in
(a) An architecture from NAS-Bench-101
(b) Graph representation of the ample neural architecture in (a).
Figure 2:
Graph modeling of neural architecture.
An example of how neural architecture is modeled as a directed acyclic graph.
3 Proposed method
In this section, we introduce our proposed generative pre-training method for neural architecture encoding, FGP (Flow-based Generative Pre-training). FGP trains neural architecture encoder to learn the information flow within given neural architecture, even if the encoder lacks specialized structure for capturing this flow.
3.1 Motivation, challenge, and overview
Motivation. Information flow refers to how input data propagates through a neural network during the forward pass and how gradients are transmitted back during the backpropagation. This concept plays a fundamental role in understanding how a neural network learns and makes predictions. making critical feature of neural architectures [32, 34. 15]. Although state-of-the-ee thethhr neural architecture encoders successfully capture the information flow through specialized model structures [34, 15], complexity leads to significantly longer processing times compared to simpler, con-flow-base encoders (refer to Figure (a)). Therefore, we anticipate that training a simple neural architecture incoder-especiall without a specialized structure for information flow- to learn this flow can address the computational efficiency issue while enhancing the encoder's overall effectiveness.
Challenge. The main challenge in learning the information flow is defining an appropriate training objective. To address this, we introduce flow surrogate, vector representing the information flow within a neural architecture. This surrogate serves as a pre-training objective, guiding the encoder to understand the information flow in neural architectures. We obtain the surrogate by passing random vectors through a directed acyclic graph that represents an architecture, which is a highly simplified simulation of a neural network's ' forward pass and backpropagation! Notably, each architecture has its own flow surrogate, which can be obtained with a one-time computational effort without any learning procedure. This does not require prior knowledge of the architecture's 's ground-truth performance on the target task. We detail the process of obtaining the flow surrogate (i.e., pre-training objective) in Section 3.2.
Overview. We now provide an overview of FGP. Given a set of unlabeled neural architectures (i.e., architectures with unknown ground-truth performance on the target task and dataset), we first express each architecture as graph (i.c., neural architecture graph), as outlined in Section 2.2. Next, we obtain each architecture's 's flow surrogate (i.e., pre-training objective). as detailed in Section 3.2. Lastly, we train a neural architecture encoder to learn the information flow by reconstructing the obtained flow surrogate, as detailed in Section 3.3.
We analyze the alternatives and reliability of using random vectors in Appendix B.9.

---

Legend
Process Mimicking the forward pass
Operotion nodes and respective embeddinga (random vecfors)
QuT
Conversion operation
convolution
Process Mimicking backproppaaation
Neural architecture
Forward pans messages
Backpropagation me ssages
Message conversion
Flow surrogate (pre-tr fore-traming objective)
Conversion baxed on 'in openition and cotziniig the flow surrogate
Converalon based on 'Ix1 33.3" convolution operations
Reversing all edges
Figure 3:
Generation of the proposed pre-training objective. Red edges mark active computation for the current step: gray edges indicate inactive computation. Starting from 'in' (input), message (spec., randomly-dramd is converted depending each operation type. The two converted messages to 'out' (output) for further conversion. This process is then reversed, and the message flows back through the operations until returns 'in'. The resulting message, called the architecture's flow surrogate, serves as the pre-training objective.
embedding of 'in') flows through the 'Ix1' and '3x3' operations, where
summed and flow
3.2 Proposed pre-fraining objective
We describe the process for obtaining the flow surrogate, our proposed representation of neural architecture's information flow, which serves as our pre-training objective. We sequentially propagate random vectors through a directed acyclic graph that represents neural architecture. Specifically, the random vectors, called messages, are passed between nodes (operations) along edges (connections between operations). When node receives messages from other nodes, its message is converted based on its specific operation type, capturing unique operational characteristics. Notably, to simulate both the forward pass and backpropagation in highly simplified manner, we: (1) propagate messages in the forward direction along the edges to model the forward pass, and (2) repeat the process in the reverse direction to model backpropagation. An overview is in Figure 3. After completing both propagation steps, the final converted message(s) serve as the flow surrogate for the corresponding neural architecture. We detail this process in three steps: (I) assigning topological order, (2) mimicking the forward pass, and (3) mimicking the backpropagation. Note that this process resembles how flow-based encoders generate outputs. However, in approach, the process is used only once per architecture (with random vectors) to obtain pre- vre-trannng objectives and not repeated for training.
3.2.1 Assigning topological order
For given neural architecture, first obtain its graph expression G = E), as outlined Section 2.2. Recall that this graph is directed acyclic graph. In this graph, we assign a topological order to each node, indicating the sequence in which each node receives the information flow. An example of topological order assignment is provided in Figure 4. Order-1 nodes are defined nodes without any incoming edges. After finding order-1 I nodes, we remove them and the edges where they serve as source nodes from the graph. Next, order-2 nodes are those that, in the modified graph, have no incoming edges. We repeat this process until all nodes have been assigned order. We denote of order-t / nodes v(o) and denote the last order T. Thus, the node set is split into disjoint subsets ((1) (TT)
Order-4 node (p(*) = (vx))
map
Orden-3 (pt*) (v,))
1x
Orden-2 nodes (V (My Uy])
U1
in
Order-1 node (M()) (rx))
Figure Example of topological order assignment in a neural architecture graph. Assignment results are (V(!) {v1]. (22 {12, va). (33) {va).
and (44) {ve)). and T 4.
3.2.2 Mimicking the forward pass
Once the topological order assignment (Section 3.2.1) 1) complete, we model the forward pass of neural architecture. To this end, we propagate forward pass messages from node to node along
5

---

the edges of the neural architecture graph. Here, the forward -ass message of a node represents the hidden state at the corresponding node (i.e., operation) within the neural architecture, and each node has a distinct forward-pass message. We refer to these forward-pass messages fp-mesxages. When a node i receives fp-messages from other nodes along edges, the fp-message of vs converted according to the incoming fp-messages and the operation type of Ug. The process of propagation and conversion proceeds in the order of y(U), v22), ..... V(T), until the fp-messages of order-T nodes (i.e., Vy E )( are converted. An example is provided in the 'Process I' box of Figure 3.
We denote a fp-message of a node V/ as f, E IR* Also, each node v2 E V has an embedding, denoted by h; E IR. based on its operation type. Specifically, by using a matrix P E R | randomly sampled from .V(0, o²), we compute the node embedding matrix H = PX, where its i-th row corresponds U is embedding (i.e., H.. = h,). Note that nodes performing the same operation share the same embedding across all architectures. Additionally, we set the fp-messages of the order-1 nodes, which do not have any incoming edges, with a randomly initialized vector E R& (i.e.. f, = r, Vuy E V(U).. Note that fp-messages of all order-1 nodes are initialized as r in every neural architecture
We formalize the propagation and conversion process at node o, E VOO. When multiple fp-messages arrive at v/ (e.g., 'out' operation node in Figure 3), we first aggregate these arriving fp- fp-messages using sum pooling as m, Σe,ENto EN'O fy, where A/(ii = {u, (ty, uy) E E}. In this process, using mean or max (rather than sum) pooling for neighbor aggregation yields lower performance than our approach (refer to Appendix B.1.).
Next, we convert the pooled fp-messages m, E R according to the operation of 0s obtaining the converted fp-message of ti, denoted by f, E Rt, as am; + (1 R))RLUU (h:||m, [mm] where |al b] is a concatenation of vectors and b. W E _2** is a fixed projection matrix, and a E [0, 1] a weighing hyperparameter. Here, W and a are shared across all architectures and nodes. When the fp-messages of order-T nodes (i.e., Uj E V(T) are converted, the process of mimicking the forward pass is complete.
W),
3.2.3 Mimicking the backpropagation
Once the process of mimicking the forward pass (Section 3,2.2) is done, we do a reverse propagation process. This process aims to mimic the backpropagation step in a neural network. To this end, we propagate backpropagation messages, which we call bp-messages. The bp-message of a node represents the gradient at the corresponding node (i.e., operation) within the neural architecture, and each node has a distinct bp-message. The overall process (i.e., propagation and conversion) is similar to that for mimicking the forward pass (Section 3.2.2), but this process proceeds in the order of (TT), VXT-)), ...., D(1), until the bp-messages of order-1 nodes (i.e., U, v(1) are converted. An example is in the 'Process 2' box of Figure 3. We denote the bp-message of node v; b, E g² Furthermore, each order-T' node's 's bp-message is initialized to the corresponding node's fp-message (i.e., b, fy. Ve, E (TT)), to mimic the dependence of backpropagation on the forward pass.
We formalize the propagation and conversion process node 2t E V(t),, Here, bp-messages arrive from the nodes at the endpoints of 0y's 's outgoing edges, and when multiple bp-messages arrive at is (e.g., 'in' operation node in Figure 3), we aggregate these incoming bp- bp-messages using sum pooling as follows: m Σv,exto by, where XC06
{uy (v;, ui) E E.
We then convert the pooled by- bp-messages m E R according to the operation of Uj obtaining the converted bp- message of us denoted by b, E IRA, as b, = am + (1 - a ReLU ([b, [mm]m]] W). When the bp-messages of order-1 nodes (i.c., 0j E v()) are converted, the process of mimicking the backpropagation is complete.
After mimicking the forward-pass and backpropagation, we sum the by- ap-messages of the order-1 nodes to obtain the flow surrogate E R (i.e., s Σ Emeva by). The resulting vector serves as the pre-training objective for the corresponding architecture, the flow surrogate representing the information flow within G. Although this process can be repeated over multiple rounds, such methods underperform compared to our approach, as detailed Appendix 10 Further discussion of the surrogate's is expressiveness and theoretical properties is in Appendices B. 12 and B.13, respectively.

---

3.3 Proposed flow generative pre-training
We now present the details of FGP. our flow generative pre-training method for neural architecture encoding. As outlined in Section 3.1, FGP trains a neural architecture encoder to reconstruct the flow surrogate (i.e., pre-training objective)
IR (Section 3.2) of given neural architecture (Section 3.2). Then, any encoder, without requiring a specialized model structure, is trained to capture the information flow through three steps: (1) encoding, (2) decoding, and (3) computing the pre-training loss.
Encoding and decoding. For given neural architecture, we start by deriving its graph expression G = (x, E), as described in Section 2.2. We then encode this graph into an embedding z E R using a neural architecture encoder fo (i.e., fx( z), which aim to train. After, we decode the embedding by using MLP decoder JĄ to obtain the reconstructed surrogate $ R (i.e., (z(2) = 8), ensuring the encoder focuses on learning generalizable representations while the decoder handles task- transformations.
Pre-training loss. Then, we compute the reconstruction loss Cree by measuring the squared 6-- distance between the original surrogate (Section 3.2) and the reconstructed surrogate á (i.e., Cree ||s - -12). Notably, FGP can be combined with auxiliary learning objectives, such as predicting zero-cost proxies of neural architecture [1, 27]. The overall training loss is L X Cree + XzLaux where λ and X2 are loss loss- weighing hyperparameters. Details on our usage of auxiliary learning objectives provided in Appendix D.2. The learnable parameters 9 and o are optimized minimize via gradient descent. After pre- training, the trained encoder fo can be fine- fine-tuned to perform certain downstream tasks, such as neural architecture's performance prediction.
Experiment
In this section, demonstrate the effectiveness of FGP in several applications, including perfor- mance prediction and neural architecture search. Specifically, we answer the following questions:
RQ1. How effective is FGP in predicting the performance of neural architectures?
RQ2. How effective is FGP in neural architecture search?
RQ3. Can the proposed flow surrogate well represent the ground-truth performance of architectures?
RQ4. Are all the key components FGP essential?
RQ5. How long does it take to train an encoder by FGP?
4.1 Experimental setup
Datasets and splits. We leverage three computer vision neural architecture datasets, which are NAS-Bench-100 101 (NB-101) [51], NAS-Bench-201 (NB-201) [8], and NAS-Bench--N (NB-301) [39] datasets. In addition, we provide results on other domain datasets (spec.. natural language processing and graph representation learning) Appendix B.6. For NB-101 and NB-201 datasets, we follow the training and test splits provided in [34, 15]. For the NB-301 dataset, since the baseline method ZC-Proxy [56] requires certain numerical properties of architectures, we use a subset of the original NB-301 dataset where these properties are available. We sample 40 architectures from the test set to create validation set, following the approach in [15]. Further details are provided in Appendix A.I.
Backbone encoders. We use 3 backbone graph-based neural architecture encoders, ResGat- edGCN [3), GIN [47], and FlowerFormer [15]. Specifically, ResGatedGCN and GIN GNN-based encoders, which do not have a specialized structure for capturing the information flow (i.e., non-flow- based encoders). In contrast. FlowerFormer is the SOTA flow-based encoder with a specialized design to capture the flow. Moreover, we present analysis for non-graph-base based models in Appendix B. 14.
Baseline methods. We use five baselines: an encoder trained solely on supervised learning (N/A) and four pre-training methods for neural architecture encoding, which are the (1) graph-contrastive- learning method (GraphCL [52]) and (2) generative methods based on connections (Arch2vec [48]), operations (GMAE [17]). and zero-cost proxies (ZC-Proxy [56]).
Pre-training, fine-tuning and evaluation protocol. For each dataset and neural architecture encoder, we first pre-train the encoder using specific pre-training method across the whole dataset (i.e., both training and test sets). Note that ground-truth performances of any architectures are not used during
7

---

Table 1: Performance prediction results. Mean and standard deviation each dataset and metric.
The best performance is highlighted in blue N/A denotes encoder solely trained with supervised learning, without any pre-training method. Gain from N/A denotes the performance improvement of FGP compared to N/A. All values are multiplied by 100 to save space in the table. FGP outperforms the baseline pre-training methods in 23 out of 27 settings.
Pre-training method
Kendall's NBB201
Precision 1%
Precision (8
Encoder
NB-101
NB.301
NB. 101
NB-201
NB.301
NB. 101
NB.20
NB-301
AIA
(3.4 (..
(4.4
18.2 (7.9)
29.7 (11.)
12.
51.7 (6.2
GraphiCL
66.9 (5.0)
73.7
64.6
21.3 (82)
(126
46.5 (..4
52.3
39.7 (33)
Arch2vec
(5.8
74.1
57.7 (3.0)
21.8 (134)
28.3 (113)
(3.5
44,9 (88)
(2.5
42.9 (30)
ResGatedGCN
GMAE
68.1 (47)
22.0 (1.8
21.8 (11.7)
35.1 (112)
49.9 (2.2
41.8 (5.8
ZC-Proxy
(6.3
79.9 (03)
(2.5)
26.2 (6.2)
44.3 (11.0)
(3.0
52.1 (6.3)
63.5
FGP
74.8
(2.2.0.7)
(2.4
37.5 (130)
48.9
61.7 (4.2
(2.3
Gain from N/A
+15.1%
+12.0%
+7.4%
+106.0%
+64.6%
+26.1%
+33.6%
+20.5%
+6.4%
N/A
62.8
65.7 (1:4)
26.9 (2.3
(1:0)
48.1 (3.5
38.9 (5.6
GraphCLL
66.9 (6.9.09) (.8)
52.4 (2.8)
32.3 (18.)
18.9 (6.2)
20.2 (3.4)
50.2 (6.3)
(3.0
(..6
Arch2vec
(2.5
(270
30.0 (1.7
(8.6
49.4 (3.1
(2.6
(0.7
GIN
GMAE
66.4 (4.)
(2.6
(L.63
31.0 (13.4)
27.8 (144)
20.8
52.9 (3.00
49.7
(1.9
7C.Prrxy
(8.2
(233
22.7 (3.0
(1.3)
18.2
(139
(0.4
38.2
FGP
(278
(2.2 (1.2
33.2 (10.0)
(1.7)
23.5 (2.2)
55.88
54.1 (3.6)
42.0 (2.2)
Gain from N/A
+8.0%
+20.6%
+5.5%
+23,4%
+42,44
+22.4%
+16.0%
+13.7%
+8.0%
N/A
(4.0)
(1.3
$5.6 (4.7)
35.3 (13.9)
(141)
(4.5)
56.4 (5.2)
56.2 (4.3)
40.7 (4.2)
GraphCI.
(47)
(3.0
55.7 (3.)
35.3 (7.5)
34.1 (149)
21.2 (2.6)
(2.5)
55.7 (6.3)
Arch2vec
(2.
(2.2 (1.8
38.4 (9.1)
33.3 (15.))
24.2
59.8
57.5 (6.1)
44.3 (2.6)
FlowerFormer
GMAE
(123)
58.9 (1.1
33.6 (9.6)
26.1 (162))
56.2 (5.0)
58.7 (5.9)
44.5 (1.8)
ZC -CoPy
74.6
(2.3
58.5 (2.2)
37,8 (5.7)
45.0 (6.6)
25.3 (2.9)
58.1 (3.3)
60.8 (3.8)
41.9 (1.5)
FGP
(16)
(3.6
60.1 (2.9)
40.6 (13.1)
48.3 (1.9
(5.4.
58.9 (4.00
(4.0
(140
Gain from N/A
+3.1%
+8.2%
+8.1%
+15.0%
+35.7%
+24.7%
+4.4%
+15.75.
+10.6%
this pre-training stage, and despite this fact, it is also possible use only the training set even for pre-training (see Appendix B.3). Following pre-training, we fine-tune the encoder using only the training set optimize for the performance prediction task, using the ground-truth performance of the training set in this phase. After fine-tuning, we evaluate the encoder on the test set using three evaluation metrics: Kendall's tau [38], Precision- 1,, and Precision- @ @5%, all widely adopted for this evaluation process [34, 15]. We employ three distinct dataset splits and three model initializations, resulting in total of nine experimental configurations. Further details are Appendix A.
4.2 RQ1: Performance prediction experiments
We assess FGP in the performance prediction task compared to the baseline pre- pre-training methods.
Setup. In practice, only a small subset of the full neural architecture search space has known ground- truth performance values due to the high cost of training and evaluating architectures. To simulate this cireumstance, we use 1% of the training set for fine-tuning neural architecture encoders a common setting in neural architecture encoding research [34, 15]. In addition, analyses under varying (1) pre-training dataset sizes and (2) training dataset sizes are in Appendix B.4 and B.2, respectively.
Results. As shown in Table 1, FGP outperforms all baselines in 23 out of 27 settings, demonstrating the effectiveness of learning information flow through pre-traiming in performance prediction. Note that the performance improvement achieved by FGP in the performance prediction task is not restricted to specific dataset or encoder. While the improvement is greater in the non-flow-based encoders (i.e., ResGatedGCN and GIN), it can also improve the performance of the flow-based encoder (i.e., FlowerFormer); we further explore the potential reasons behind the improvement of FlowerFormer in Appendix B.1
4.3 RQ2: Neural architecture search experiments
Performance prediction is often integrated into neural architecture search (NAS) to automatically identify neural architectures suitable to the target task and dataset. Accordingly, we evaluate the efficacy of FGP in NAS, assessing its performance against those of existing pre-training methods.
Setup. As in [15], we use NPENAS [42] our backbone NAS algorithm, which uses performance predictor to evaluate neural architectures in the search process. Note that an accurate performance predictor enhances search efficacy, leading to the discovery of
architectures.
8

---

To assess each pre-training method, we (1) pre-train a performance predictor on neural architectures within the designated search space and (2) initialize NPE- NAS's predictor with this pre-trained model the be-
6.4
Arch2vec
Test error of top-1 neural architacture
GMAE
6.2-
ZC-Proxy
ginning of each search. We run 10 trials with different initializations.
6.0
FGP (Ours)
GraphCL
201 and ResGatedGCN serve the search space and performance predictor, respectively. Further details are in Appendix A.5.
5.8-
Lower the
better
5.6
Results. As shown in Figure 5, the NAS method, equipped with a performance predictor pre-trained using FGP, identifies the more effective neural archi- tecture than all competitors, each employing dis- tinct pre-training method. Notably, FGP consistently performs best at every NAS step.
20
40 60 80 100
120
140 160
NAS steps (# of trained neural architectures)
Figure NAS results. The mean test error of the top-1 architecture across 10 trials is reported. The backbone NAS algorithm NPE- NAS [42] adopts pre-trained performance pre- dictors, with each line representing a distinct pre-training approach.
4.4 RQ3: Flow surrogate analysis
We provide a qualitative analysis of the re- lation between the flow surrogate of neural architecture (Section 3.2) and the ground- truth performance of the corresponding chitecture. Specifically, we aim to ver- ify whether our flow surrogate well rep- resents the architecture's 's ground-truth per- formance, separating architectures with dif- fering performance..
High-porformance neural architectures Figure 6: Pre-training objective visualization. PCA visualization of flow surrogates (i.e., pre-training ob- jective), where colors represent the performance of the corresponding architecture.
Low-parformance
neural architectures
Setup. We visualize the distribution of the flow surrogate by using PCA. We color each data point according to the corre- sponding neural architecture's ground-truth performance.
Results. As shown in Figure 6, the proposed flow surrogate effectively represents the performance of neural architectures, distinguishing serh-peffomingg architectures from low-performing ones in separate regions. This visualization indicates that our pre-training objective, flow surrogate, can well guide a neural architecture encoder in identifying architectures likely achieve high performance.
4.5
RQ4: Ablation study
We demonstrate the necessity of the key components FGP in achieving high performance.
Setup. We use four variants of FGP:
Figure 7: Ablation study. Performance of the variants of FGP. All values are multiplied by 100 to save space. The best performance is highlighted in blue
V1. This does not perform the flow sur- rogate reconstruction (w/o Eyen).
Variants
Kendall's Tau
Precision@)15 1%
V2. This does not use the auxiliary train- ing objective (w/o Laux).
NB-101 NB-201 NB-301
NR-101
NB-201
NB-301
Eaen
68.3 (6.7) (08) 57.9
26.2.62)
44.3 (11))
23.0 (3.))
w/o Casz
(5.5 74.5 mm (8.0
35,6
36.5
23.6
V3. This skips mimicking the forward pass (Section 3.2.2) in obtaining the flow surrogate (w/o Forward).
wio Forward
(335 81. 57.1 (.8)
(6.3
23.4 44.6
(41)
Backward
72.4 81.4 (7.5
31.0
24.2 (27) (3.
(2.2 58.4
37.5 (13.0)
(4X)
V4. This skips mimicking the backpropagation. (Section 3.2.3) in obtaining the flow surrogate (w/o Backward).
We use ResGatedGCN as our backbone encoder and evaluate under the performance prediction task. Other settings are the same in Section 4.2.
Results. As shown in Table 7, FGP achieves superior performance over its four variants five out of six settings, underscoring the necessity of its core components for achieving high performance.

---

resulto (metiic: Kendalle ta))
prediction
Datasel:
NB-201
x4.3
ResGaledGCNN
Pre-training time (bec.)
1300
Archêvec
Performance wwataaaneeeeeeieeeeeeeee
60
+8.3%
ReeGGeleDCCN (Nor-Fiow- based) FGP
FlowerFormer (Flow-based)
1000
ZC-Proxy
GMAE
weo FGP
25
FlowerFormer (Flow-based)
GraphCL
FGP
PGP (Ouns)
Pre-training Neural architacture
processing time (sec.)
N6-101
NS-201
NI-3001
(a) Encoders' runtime ppre-training time included) and (b) Total pre-training time for each method using the
performance-predictin results (x 100).
ResGatedGCN backbone.
Figure 8: Speed analysis. (a) Pre-training the non-flow encoder (ResGatedGCN) with FGP improves both performance and speed over the flow-based encoder (FlowerFormer) without FGP. (b) FGP is
the recond-fastest among the pre-training methods for neural architecture encoding.
4.6 RQ5: Speed analysis
We provide speed analysis of FGP.
Setup. We analyze (1) the total runtime and performance prediction results with (w/) and without (w/o) FGP pre-training for the flow-based encoder (FlowerFormer) and the non-flow-based encoder (ResGatedGCN) and (2) the pre-training runtime of each pre-training method, equipped with Res- GatedGCN. We set the batch size and pre-training epochs to 256 and 200, respectively. Also, comparison of all methods under equal runtime Appendix B.5.
Results. First, as shown in Figure (a), pre-training the non-flow-basee encoder (ResGatedGCN) with FGP shows both performance and speed gain over the flow-based encoder (FlowerFormer) without FGP in performance prediction. That is, despite this performance superiority, the total runtime for the pre-training and architecture processing of ResGatedGGN remains significantly shorter than the processing time alone of FlowerFormer. Second, as shown in Figure (b), FGP the second-fasted pre-training method, comparable to the fastest one, ZC-Proxy. Thus, despite the large performance gain compared to the baselines, FGP does not impose a significantly greater computational burden.
Conclusion
In this work, we propose novel generative pre- pre-training method for neural architecture encoding, called FGP, which enables a neural architecture encoder to learn the information flow within a neural architecture, even without specialized model structure. To this end, FGP trains neural architecture encoder to reconstruct the flow surrogate, our proposed pre-training objective, based on information flow. Our extensive experiments demonstrate the efficacy of FGP in performance prediction and neural architecture search. Code and datasets are in https // gighhu.... com /ttmm///wooS//FGAAoooo
Acknowledgements
This work was supported by Institute of Information & Communications Technology Planning & Evaluation (ITTP) grant funded by the Korea govermment (MSIT) (No. RS-2022-11220871 Development of AI Autonomy and Knowledge Enhancement for AI Agent Collaboration, 60%) (No. RS-2022-11220157 Robust, Fair, Extensible Data-Centrie Continual Learning, 20%) (No. RS-2024-00457882. AI Research Hub Project, 10%) (No. RS-2019-11190075, Artificial Intelligence Graduate School Program (KAIST), 10%).
10

---

References
[1] M. S. Abdelfattah, A. Mehrotra, t., Dudziak, and N. D. Lane. Zero-cost proxies for lightweight nas, In ICLR, 2021.
[2] U. A. Bakshi and L. A. V. Bakshi. Electrical circuit analysis. Technical Publications, 2020.
[3] X. Bresson and T. Laurent. Residual gated graph convnets. arXiv preprint arXiv: 17
2017.
[4] Y. Chen, Y. Guo, Q. Chen, M. Li, W. Zeng, Y. Wang. and M. Tan. Contrastive neural architecture search with neural architecture comparators, In CVPR, 2021.
[5] K. T. Chitty-Venkata. M. Emani, V. Vishwanath, and A. K. Somani. Neural architecture search for transformers: A survey, IEEE Access, 10:108374-108412, 2022.
[6] J. Delvin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.
[7] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hicrarchical image database. In CVPR, 2009,
[8] X. Dong and Y. Yang. Nas-bench-201: Extending the scope of reproducible neural architecture search. In ICLR, 2020.
[9] A. Dosovitskiy, L. Beyer, Kolesnikov, D. Weissenborn, Zhai, T. Unterthiner, M. Dehghani. M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.
[10] L. Dudziak, Chau, M. Abdelfattah, Lee, H. Kim, and N. Lane. Brp-nas: Prediction-based nas using gens. In NewrlPS, 2020.
[11] T. Elsken, J. H. Metzen, and F. Hutter. Neural architecture search: A survey. Journal of Machine Learning Research, 20(55): 1-21, 2019.
[12] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. CVPR, 2022.
[13] W. Hu, B. Liu, J. Gomes, M. Zitnik, P. Liang, V. Pande, and Leskovec. Strategies for pre-training graph neural networks. In ICLR, 2019.
[14] Y.-C. Huang, W.-H. Li, C.-H. Tsou, J.-C. Chen, and C.-S. Chen. Up-nas: Unified proxy for neural architecture search. In CVPR, 2024.
[15] D. Hwang, H. Kim, S. Kim, and K. Shin. Flowerformer: Empowering neural architecture encoding using flow-aware graph transformer. In CVPR, 2024.
[16] G. Jawahar, M. Abdul-Mageed, L. V. Lakshmanan, and D. Ding. Llm performance predictors are good initializers for architecture search. In ACL, 2024.
[17] K. Jing, J. Xu, and P. Li. Graph masked autoencoder enhanced predictor for neural architecture search. In IJCAI, 2022.
[18] Kim, Kang, Bu, S. Y. Lee, Yoo, and K. Shin. Hypeboy: Generative self-supervised representation learning hypergraphs. In ICLR, 2024.
[19] S. Kim, S. Y. Lee, F. Bu, S. Kang, K. Kim, J. Yoo, and K. Shin. Rethinking reconstruction-bast
graph-level anomaly detection: limitations and a simple remedy. In NeurIPS, 2024.
[20] Kim, S. Y. Lee, Y. Gao, A. Antelmi, M. Polato, and K. Shin. A survey on hypergraph neural networks: an in-depth and step-by-step guide. In KDD, 2024.
[21] S. Kim, S. Y. Lee, J. Yoo, and K. Shin. 'hello, world!': Making gans talk with llms. In EMNLP,
2025.
[22] T. N. Kipf and M. Welling. Variational graph auto-encoders. In NeurIPS Bayesian Deep Learning Workshop, 2016.
[23] T. N. Kipf and M. Welling. Semi-supervised classification with graph convolutional networks. In ICLR, 2017.
[24] N. Klyuchnikov, I. Trofimov, E. Artemova, M. Salnikov, M. Fedorov, A. Filippov, and E. Bur- naev. Nas-bench-nlp: neural architecture search benchmark for natural language processing. IEEE Access, 10:45736-45747, 2022.

---

[25] A. Krishnakumar, C. White, A. Zela. R. Tu, M. Safari, and F. Hutter. Nas-bench-ssite-zen Accelerating research on zero cost proxies. In NeurIPS, 2022.
[26] A. Krizhevsky, G. Hinton, et Learning multiple layers of features from tiny images. 2009.
[27] Lee and B. Ham. Az-nas: Assembling zero-cost proxies for network architecture search. In CVPR, 2024.
[28] W. Lin, X. Peng, Z. Yu, and T. Jin. Hypergraph neural architecture search. In AAAI, 2024.
[29] H. Liu, K. Simonyan, and Y. Yang. Darts: Differentiable architecture search. In ICLR, 2019.
[30] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In ICLR, 2019.
[31] S. Lu, J. Li, J. Tan, S. Yang, and J. Liu. Tnasp: A transformer- nas predictor with a self-evolution framework. In NeurlPS, 2021.
[32] X. Ning, Y. Zheng, T. Zhao, Y. Wang, and H. Yang. A generic graph-based neural architecture encoding scheme for predictor based nas. In ECCV, 2020.
[33] X. Ning, C. Tang, W. Li, Z. Zhou, S. Liang. H. Yang, and Y. Wang. Evaluating efficient performance estimators of neural architectures. In NeurlPS, 2021.
[34] X. Ning, 7. Zhou, J. Zhao, T. Zhao, Y. Deng, C. Tang, S. Liang, H. Yang, and Y. Wang. Ta-gates: An encoding scheme for neural network architectures. In NeurIPS, 2022.
[35] Y. Ou, Y. Feng, and Y. Sun. Towards accurate and robust architectures via neural architecture search. In CVPR, 2024.
[36] J. I.. Peterson. Petri nets. ACM Computing Surveys, 9(3):223-252) 1977.
[37] Y. Qin, 7. Zhang, X. Wang, Z. Zhang, and W. Zhu. Nas-bench-graph: Benchmarking graph neural architecture search. In NewrIPS, 2022.
[38] K. Sen. Estimates of the regression coefficient based on kendall's tau. Journal of the American 1968.
statistical association, 63(324):
[39] J. Siems, L. Zimmer, A. Zela, J. Lukasik, M. Keuper, and F. Hutter, Nas-bench-30 -00 and the case for surrogate benchmarks for neural architecture search. In ICLR, 2021.
[40] V. Thost and Chen. Directed acyclic graph neural networks. In ICLR, 2021.
[41] Z. Tong, Y. Liang, C. Sun, X. Li, D. Rosenblum, and A. Lim. Digraph inception convolutional networks. In NeurlPS, 2020.
[42] C. Wei, C. Niu, Y. Tang, Y. Wang, H. Hu, and J. Liang. Npenas: Neural predictor guided
evolution for neural architecture search. IEEE Transactions on Neural Networks and Learning Systems, 34(11):84418455 2022.
[43] W. Wen, H. Liu, Y. Chen, H. Li, G. Bender, and P.-J. Kindermans. Neural predictor for neural architecture search. In ECCV, 2020.
[44] C. White, W. Neiswanger, S. Nolen, and Y. Savani. study encodings for neural architecture search. In NeurIPS, 2020.
[45] C. White, A. Zela, R. Ru, Y. Liu, and F. Hutter. How powerful are performance predictors in neural architecture search? In NeurIPS, 2021.
[46] F. Wilcoxon. Individual comparisons by ranking methods. In Breakthroughs in statistics: Methodology and distribution, pages 196-202. Springer, 1992.
[47] K. Xu, W. Hu, Leskovec, and Jegelka. How powerful are graph neural networks? In ICLR, 2019.
[48] S. Yan, Y. Zheng. W. Ao, X. Zeng, and M. Zhang. Does unsupervised architecture representation learning help neural architecture search? In NeurlPS, 2020.
[49] Y. Yi, H. Zhang, W. Hu, N. Wang, and X. Wang. Nar-forner: Neural architecture representation learning towards holistic attributes prediction. In CVPR, 2023.
[50] Y. Yi, H. Zhang, R. Xiao, N. Wang, and X. Wang. Nar-former v2: rethinking transformer for
universal neural network representation learning. In NeurIPS, 2023.
[51] C. Ying, A. Klein, Christiansen, Real, K. Murphy, and F. Hutter, Nas-bench- 101:: Towards reproducible neural architecture search. In ICML, 2019.
12

---

[52] Y. You, T. Chen, Y. Sui, T. Chen, Z. Wang, and Y. Shen. Graph contrastive learning with augmentations. In NeurIPS, 2020.
[53] Y. Yuan, C. Shao, Ding, D. Jin, and Y. Li. Spatio-temporal few-shot learning via diffusive neural network generation. ICLR, 2024.
[54] X. Zhai, J. Puigcerver, A. Kolesnikov, P. Ruyssen, C. Riquelme, M. Lucic, Djolonga, A.
Pinto, M. Neumann, A. Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint
2019.
[55] R. Zhang, Hu, Li, Huang, Deng, Qiao, Gao, and Li. Prompt, generate, then cache: Cascade of foundation models makes strong few-shot learners. In CVPR, 2023.
[56] J. Zhao, X. Ning, E. Liu, B. Ru, Z. Zhou, T. Zhao, C. Chen, J. Zhang, Q. Liao, and Y. Wang. Dynamic ensemble of low-fidelity experts: Mitigating nas "cold-start". In AAAI, 2023.
[57] L. Zhu, B. Liao, Q. Zhang, X. Wang. W. Liu, and X. Wang. Vision mamba: Efficient visual
representation learning with bidirectional state space model. ICML, 2024.
13

---

A Experimental details
Table 2: Dataset statisties. Total counts of training and test data, along with the average time (in
seconds) required to obtain the proposed flow surrogate (defined in Section 3.2).
Dataset
# of train
test
Time for surrogate (secs.)
Nas- mas--Bech--le 101 [51]
7,290
7,290
0,002
Nas- Nas-Bench-22 Rench-201 [8]
7,813
7,812
0.002
Nas-Benche Bench--nn [39]
5,611
5,610
0,005
A.1 Dataset details
In this section, we elaborate on the leveraged three convolution neural architecture datasets. Dataset statisties provided in Table 2. Specifically, represent each architecture based operation- on-node type graph [34], where each operation corresponds to a node, and each edge represents a connection between operations.
NAS-Bench-101 [51]. This dataset includes 423K convolutional neural architectures, where each data point represents specific neural architecture along with its test accuracy the CIFAR-10 10 image classification task [26]. We leverage a subset of the entire search space provided by Hwang al. [15], Ning al. [34].
NAS- MAS-Bench-221 [8]. This dataset includes 15K convolutional neural architectures, where each data point represents a specific neural architecture along with test accuracy on the CIFAR-10 image classification task [26].
NAS-Bench-301 [39]. This dataset includes 10 convolutional neural architectures, where each data point represents a specific neural architecture along with its surrogate performance (i.c., an estimated performance produced by performance predictor) on the CIFAR-10 image classification task [26]. Each architecture is represented by two graphs: normal cell and a reduction cell. Following Siems et al. [39], we connect the output node of the normal cell graph to the input nodes of the reduction cell, creating a single unified graph that represents the architecture. We leverage subset of the entire search space provided by Abdelfattah et al. [1], where zero-cost proxies are publicly available.
For non-flow-bee floww-baswdddd neural architecture encoders (i.e., ResGatedGCN [3] and GIN [47]), which lack specialized message passing for directed acyelic graphs, we transform each neural architecture graph into an undirected graph by adding a reverse edge for every existing graph.
A.2 Machines and implementation
We conducted our experiments machine with NVIDIA RTX 8000 D6 GPUs (48GB memory) and two Intel Xeon Silver 4214R processors. FGP is primarily implemented using the Pytorch (v1.12.1) and Pytorch Geometric (v2.2.0) libraries. We used AdamW [30] as the learning optimizer.
A.3 Fine-tuning protocol
In this section, we detail our fine-tuning protocol. For a given neural architecture encoder fo. which randomly initialized pre-trained with partieular pre-training method, fine-tune the neural architecture encoder to perform the performance prediction task.
Regression. Recall that neural architecture encoder outputs dd-iimeniional vector ( E Rd representing given neural architecture (()), To estimate the performance of the corresponding neural architecture, denoted (8) R. we employ a
regressor F3g R2d + R.
The performance estimate ((() is obtained by projecting (X) using this regressor (i.e., ((()) rg (z ()))
Learning objective. Similar to our use of a zero-cost proxy (refer to Appendix D.2), train model to predict which architectures are 'more likely' to perform well, we leverage margin-ranking
14

---

loss. For two neural architectures g and Gb) within the same batch, we compute the following prediction loss:
fine
max (0, m' - -0g() r(²))
(1)
(1d)))()3(0)
where m' is margin hyperparameter. The parameters A and & are trained to minimize fime (Eq. 1) via gradient descent. This updating process continues until it satisfies the validation stopping criterion reaches the maximum number of fine-tuning epochs.
A.4 Hyperparameters
Neural architecture encoders. For ResGatedGCN [3] and GIN [47], whose best hyperparameter configurations are not publicly provided, we tuned their hidden dimensions and maximum fine-tuning epochs within (64, 128, 256) and (300, 400, 500), according to the validation performance the performance prediction task. We fixed their number of layers, fine-tuning learning rate, and fine-tuning weight decay, as 3, 10-3 and 10-5 respectively, After finding the hyperparameter configuration that gives the best validation performance, the chosen configuration is used across all the methods (i.e., the same encoder structure and pre-training scheme leveraged across different pre-training methods). For FlowerFormer [15], we follow the hyperparameter configurations provided in their official implementation https: //gt thub. com/yOngj wwwniiusCVPRROO
B.DDERForrme.
Pre-training methods. For every pre-training method, including the baseline methods and our proposed method FGP, tuned the following hyperparameters:
Learning rate within (10-8,5 5 x 10-1 10-1,10-1) 1100))
Projection head dimension within {32, 64, 128, 256),
Projection head number of layers within (1,2,3).
Weight decay within {10-6 110-6,0.0) 0.0).
Additionally, for FGP, we tuned the learning objective coefficients {(2: (5:2).(2 (g. 3), (3.33) }).
(A1, within
A.5
Details regarding neural architecture search
In this section, we provide details regarding our neural architecture search experiment (Section 4.3). Note that the process below is based on NPENAS [42].
We first pre-train neural architecture encoder using a designated pre-training strategy, such as FGP. At the start of each round, the encoder is initialized with the pre-trained weights. The search begins with 20 neural architectures, each with known ground-trutt performance. Using these, we fine-tune the encoder predict performance. For each of the 20 architectures. we generate 5 valid candidates via mutation. Among the mutated architectures, we select the top 20 based on predicted performance and evaluate them. The selected architectures and their performances are added to the budget. With the updated pool, now containing 40 evaluated architectures, we repeat the process until the budget reaches predefined size.
B Additional experimental results
B.1
Analysis of encoding time of diverse architecture encoders
In this section, we provide further details regarding the encoding time analysis, which is provided in Figure 1. Specifically, we provide detailed mean encoding time architecture for the flow-aware neural architecture and the simple GNN-hased neural architecture.
Setup. We compare the encoding time of a flow-aware encoder (i.e., FlowerFormer) with non- flow-aware encoders (i.e., ResGatedGCN and GIN). Specifically, measure the time required to generate representations for all neural architectures-bon both training and test-and report the average per architecture.
15

---

Table 3: Performance prediction results on various training set ratios. Mean and standard deviation on each dataset and metric. The best performance is highlighted in blue All values are multiplied by 100. N/A denotes an encoder solely trained with supervised learning, without any pre-training. Gain from N/A denotes the performance improvement of FGP compared to N/A. FGP outperforms the baselines in 14/18 cases.
Training ratio
Pre-training method
Kendall's NB-201
Precision NB-201
Precision @0 NB-201
NB-101
NB-301
NB-101
NR-301
NB-101
NR-301
NIA
75.8 (34)
$4.6.02)
67.2 0.00
42.9 (1.3)
(3.)
31.9 (29)
64.5 (56)
69.4
51.9 (34)
GraphCL
76.0
8544007)
66.4 (1.3)
(88)
57.0
29.9
63.4
69.8
(1.6
Arch2vec
(.7
68.4 (7)
(2.8)
(5.)
34.5 (54)
65.1 (3.1)
(4.2 (01)
5%
GMAE
76.2 (m
67.8 (.8)
41.7 (8.1)
15.0
35.4 (89)
62.5 (56)
53.1 (..6
2C-Proxy
(..
81.7 (0.)
6..6
42.7 (7.)
56.4
33.9 (4.)
67.3 (300)
51.7 72.2
(42)
FGP
(3.)
(040
68.6 (2.4)
45.8 (11.)
59.7 (5.4)
34.8 (4.6
66.8 (3.2)
(3.6
(503
Gain from N/A
+4.7%
+4.0%
+2.1%
+6.8%
+9.9%
+3.6%
+6.1%
+2.7%
NIA
77.3 (5.1)
88.3 (0.3)
69.9 0.2)
43.2 (8
57.9 (2.2)
35.4 (5.2)
65.8 (4.0)
74.0 (1.0)
56.6 (2.0)
GraphiCL
77.3 (5.9)
88.9
69.1 (2.3)
42.7 (1.0)
58.6 (4.1)
35.6 (3.)
66.3 (3.8
55.3 (1.3)
(28)
Arch2vec
78.3
88.8
71.2
(11
(3.3)
37.2 (56)
66.5
73.9
56.7
10%
GMAE
78.1 (5.9)
89.9 (0.4)
70.8 (1.0)
(10.)
(5.3
38.8 (4.)
67.6 (5.)
75.4 (0.7)
(1.6
ZC-Pruxy
79.0
89.7 (0.2)
70.2 (0.9)
44.0 (100
58.2 (1.3)
37.4 (4.4)
67.2 (5.0)
74.5 (1.4)
56,8 (1.4)
FGP
79.9 (8.3)
89.9 (0.3)
71.5 (0.6)
45.8 (10.5)
(8.4
38.9 (5 0)
67.8 (34)
76.0
56.9 (2.)
Gain from N/A
+3.4%
+1.8%
+2.3%
+6.0%
+1.0%
+9.8%
+3.0%
+2.7%
+0.15
Results. As shown in Table 4, FlowerFormer, flow-aware encoder, takes at most 94x more encoding time compared to GIN, which is a sim- ple GNN encoder that does not have a special- ized architecture to capture the information flow. This result supports claim that flow-aware encoders are significantly slower than simple GNN-based encoders.
Table 4: Detailed mean encoding time per architec- ture of each neural architecture encoder
NB-101
NB-201
ResGatedGGCN GIN
0.0021
0.0022
0.0013
0.0014
FlowerFormer
0.1242
0.0849
B.2 Analysis under varying training set size
In this section, we present additional experimental results across different training set ratios. In the main paper, we focus scenario where only 1% of the training set is used for fine-tuning, simulating a label-scarce scenario (i.e., a limited number of sechitectur--peffommen Recall that this is because obtaining architecture- auchitecture-peffrrmncce pairs involves significant computational costs, making it crucial to develop a pre-training method that enhances neural architecture encoders in label-scarce scenarios. Furthermore, we evaluate each method in scenarios where additional mnchitecuuee-eeffrmmane pairs are available, corresponding to a larger training set ratio.
pairs available).
Setup. We explore two additional settings for fine-tuming using (1) 5% and (2) 10% of the training set, respectively. We use ResGatedGCN [3] as the backbone neural architecture encoder. Other experimental settings are the same those used in our neural architecture performance prediction experiments (Section 4.2).
Results. As shown in Table 3, FGP outperforms all baseline methods in 14 out of 18 settings. This result demonstrates that the superiority of FGP over existing pre- pre-training methods not restricted to specific training set ratio setting.
B.3 Analysis under using only the training set, including for pre-training
In this section, we present additional experiment results when we only leverage the training dataset to perform pre-training. In the main paper, we leverage the test dataset with the training during the pre-training, since architectures within the search space can be obtained with negligible cost. In the new setting, we aim to determine whether an encoder pre-trained with FGP can generalize to unseen neural architectures.
Setup. We do not use the test dataset for pre-training, and therefore, each test architecture becomes an unseen architecture. We use ResGatedGCN [3] as the backbone neural architecture encoder. Other experimental settings are the same as those used in our neural architecture performance prediction experiments (Section 4.2).
Results. As shown in Table 5, FGP outperforms all baseline methods in all the settings. This result demonstrates that the encoder trained with FGP generalizes well to unseen neural architectures.
16

---

Table 5: Performance prediction results when using only the training dataset even for pre- training. Mean and standard deviation on each dataset and metric. The best performance is highlighted in blue All values are multiplied by 100. N/A denotes an encoder solely trained with supervised learning, without any pre-training- Gain from N/A denotes the performance improvement of FGP compared to N/A. FGP outperforms the baseline methods in all the settings.
Pre-training method
Kendall's s Tau
Precision 1%
Precision@
NB-101
NB-201
NB-101
NB-201
NB-101
NB-201
N/A
(7.0
73.4 (0.7)
18.2
29.7 (3.7)
46.2 (12.)
51.7 (4.2)
GraphCL
65.2 (6.4)
74.3
20.1 (12.6)
36.4 (9.7)
44.4 (11.3)
54.5
Arch2vec
63.2 (7.8)
73.5
18.5 (9.3)
31.7 (119)
42.4 (12.4)
53.3
GMAE
67.3 (5.2)
(1.4)
25.5 (10.1)
34.8 (9.)
51.0 (70)
56.8 (4.9)
ZC-Proxy
69.8 (5.1)
79.8 (1.3)
29.8 (1L.1)
39.2 (12.)
56.4 (3.5)
58.9 (8:2)
FGP
(3.5)
81.4 (7.7)
(10.9
45.1 (7.9)
60.1 (1.0)
61.5 (2.8)
Gain from N/A
+13.1%
+10.8%
+103.3%
+51.9%
+30.1%
+19.0%
NAS-Bench-101
NAS-Bench-201
80
Kendall's Tau
GMAE
ZC-Proxy
76
66
FGP (Ours)
72
20%
40%
60%
80%
100%
20%
80%
100%
Ratio of training set used for pre-training
Figure 9: Performance under varying size of pre-training dataset. Performance prediction results when 20/40/60/80/ 000////////////////////// of the training set is used for pre-training of each method. FGP consistently outperforms the two strongest baseline methods (GMAE and ZC-Proxy) across all the settings.
B.4 Analysis under varying pre-training size
In this section, we present additional experimental results with varying pre-training dataset sizes, demonstrating that the effectiveness of FGP is not restricted to a specific data size.
of the training set and use the sampled neural architectures for pre-training of a neural architecture, based each pre-training strategy. We use ResGatedGCN as the backbone neural architecture encoder, and the two strongest pre-training method (i.e., ZC-Proxy and GMAE) as baseline methods. Other setups the same those used in our neural architecture performance prediction experiments (Section 4.2).
Setup. We randomly sample /0%//40///0////////00
Results. As shown in Figure 9, the superiority of FGP holds under varying size of pre-training dataset size, demonstrating that the effectiveness of FGP not limited a particular dataset size.
B.5 Analysis under a fixed pre-training time
Recall that in Section 4.6, we provide analysis regarding the runtime of FGP and other baseline methods. In this section, we provide performance comparison of pre-training methods under the fixed pre-trainig time.
Setup. We train a neural architecture for 100/200/300/400/500 seconds, by using each pre-training method. When the corresponding time limit is reached, we execute the pre-training process and fine- tune and evaluate the pre- me-trained encoder for the performance prediction task. We use ResGatedGCN as the backbone neural architecture encoder, and the two strongest pre-training method (i.e., ZC- -C-Proxy and GMAE) as baseline methods. Other setups are the same as those used in our neural architecture performance prediction experiments (Section 4.2).
Results. As shown in Figure 10, the superiority of FGP holds under varying pre-training time split, demonstrating that the effectiveness of FGP is not limited to a particular dataset size.
17

---

NAS-Bench-101
NAS-Bench-201
76
84
74-
82
Kendall's Tau
72
GMAE
80
70-
78
ZC-Proxy
68
76
66
FGP (Ours)
64
100
200
300
400
500
72
0
100
200
300
400
500
Pre-training time (seconds)
Figure 10: Performance under the fixed pre-training time. Performance prediction results under the fixed pre-training time of each pre-training method. FGP consistently outperforms the two strongest baseline methods (GMAE and ZC-Proxy) across all the settings.
Table 6: Performance prediction results under various domains, Mean and standard deviation
each dataset and metric. The best performance highlighted in blue All values are multiplied by 100. Since zero-cost proxy values for the NB-Graph dataset are not publicly available, we could not run the ZC-Proxy bascline on the NB-Graph dataset, which is marked by *-', FGP outperforms the baselines natural language processing datasets and graph representation learning datasets.
Pre-training method
Kendall's Tau
Precision @ 1%
Precision @ 5%
NB-NLP NB-Graph
NB-NLP NB-Graph
NB-NLP NB-Graph
N/A
28.5 (6.1)
37.1 (6.3)
(1.4)
4.3 (2.3)
14.8 (6.9)
17.1 (4.1)
GraphCL
28.7 (5.3)
38.3 (13.3)
3.1 (1.3)
3,8 (1.5)
14.9 (6.1)
16.1 (4.3)
Arch2vee
28.4 (5.7)
38.8 (9.6)
312)
3.2 (2.1)
15.1 (5.2)
15.9
GMAE
28.9 (6.6)
37.7 (8.6)
2.2 (1.0)
(2.1)
(5.3)
14.5
ZC-Proxy
28.7 (5.6)
(2.3)
15.5
FGP
29.7 (4.8)
41.1 (8.6)
4.3 (2.5)
(1.1)
16.2 (7.1)
18.3 (3.5)
B.6
Analysis under various domains
In this section, we explore the effectiveness of FGP in domains beyond computer vision, which are natural language processing and graph representation learning.
additional benchmark datasets: (1) NAS-Bench-NLP [24], dataset that consists of natural language processing models, and (2) WAS-Bench-Graph [37], a dataset that consists of graph representation learning models.
Setup. We
Results. As shown Table FGP outperforms the baselines in 5 out of settings, demonstrate that the effectiveness of FGP does not limit to computer vision domain.
B.7 Analysis under another task
Recall that NAS MAAS-Bench-101 NAS-Bench-201, and NAS-Bench-301 are benchmark datasets con- taining image classification performances various architectures. In this section, we assess each method's effectiveness in predicting architecture performance on different task-dataset transfer- ability-using new dataset.
Setup. Since no benchmark directly covers the transferability of architectures, we built small neural architecture benchmark dataset. To this end, we build two datasets: (1) TinyImageNet (TIN) [7] + DTD [54] and (2) ImnyImageNet (TIN) [7] OxfordPet [54]. To this end, we randomly sampled 100 architectures from M.S--Benhh-10 101, pre- preited each on TinylmageNet, and fine-tuned each on DTD or OxfordPet. We used each architecture's test accuracy on the target dataset as the architecture's 's transferability score. Architectures were split 50/50 into train/test sets. ResGatedGCN used as backbone neural architecture encoder. An architecture encoder was first pre-trained with a pre-training method (without transferability scores), then fine-tuned on the training set with transferability scores, and finally evaluated the test set.
18

---

Table 7: Performance prediction results under downstream task. Mean and standard deviation on each dataset and metric. N/A denotes a performance of an encoder without pre-training. All values are multiplied by 100. The best performance is highlighted in blue The superiority of FGP over baseline methods holds valid under a new downstream task, dataset transfer.
Kendall's Tau
Precision @ 10%
TIN + DTD TIN - OxfordPet
TIN - DTD TIN + OxfordPet
N/A
62.7 (4.6)
60.3 (4.4)
50.0 (31.1)
63.9 (12.4)
GMAE
63.4 (3.8)
61.0 (3.7)
55.5 (25.8)
63.9 (12.4)
ZC-Proxy
64.1 (4.9)
60.8 (8.1)
47.0 (29.7)
61.1 (11.7
FGP
65.3 (4.3)
64.1 (3.1)
63.9 (12.4)
66.7 (11.7)
Table 8: Performance prediction results under new neural architecture encoder. Mean and standard deviation on each dataset and metric. N/A denotes performance of an encoder without pre-training. All values are multiplied by 100. The best performance is highlighted in blue The superiority of FGP over baseline methods holds valid under a new encoder, DiGCN.
Kendall's
Precision @ % NB-101
Precision @ 5%
NB-101
NB-201
NB-201
NB-101
NB-201
N/A
65.2 (4.4)
65.7 (1.5)
25.6 (5.5)
14.1 (4.8)
49.0 (1.1)
43.9 (2.6)
GMAE
65.1 (3.9)
66.0 (1.4)
26.3 (5.4)
16.2 (5.2)
49.5 (2.7)
44.2 (2.1)
ZC-Proxy
66.9 (4.3)
72.1 (1.3)
26.0 (3.9)
18.8 (4.7)
50.5 (5.7)
47.1 (1.9)
FGP
73.3 (3.7)
77.9 (2.0)
37.3 (8.6)
23.8 (8.1)
57.5 (5.6)
50.9 (5.8)
Results. As shown in Table 7, FGP outperforms all the baseline methods all settings, demonstrating that its superiority extends beyond predicting image classification performance to also include architecture transferability prediction.
B.8 Analysis under new neural architecture encoder
In this section, explore the effectiveness of FGP under new directed-grapp seaeteddgraah--baaaeee based neural architecture encoder.
Setup. We use DiGCN [41], a praectedggrap--ppeciiiized neural architecture, as the backbone neural architecture encoder, and the two strongest pre-training methods (i.e., ZC-Proxy and GMAE) as baseline methods. Other setups are the same as those used in our neural architecture performance prediction experiments (Section 4.2).
Results. As shown in Table 8, FGP outperforms the baselines in all settings, demonstrate that the effectiveness of FGP is not limited particular neural architecture encoder.
Setup. We use two additional benchmark datasets: (1) NAS-Bench-NLP [24], dataset that consists of natural language processing models, and (2) NAS-Bench-Grapp [37], a dataset that consists of graph representation learning models.
Results. As shown in Table 6, FGP outperforms the baselines in 5 out of 6 settings, demonstrate that the effectiveness of FGP does not limit to a computer vision domain.
B.9 Analysis regarding the usage of random vectors
In this section, we present additional experiment results regarding the usage of random vectors and matrices for flow surrogate (Section 3). We analyze two questions: (Q1) Do the values of random vectors and matrices significantly impact the effectiveness of FGP in downstream tasks? (Q2) Does using random vectors and matrices for flow surrogates perform better than using representations of architectures obtained from flow-aware encoders?
B.9.1
Q1. Various random initializations
Setup. We use five different random initializations of vectors and matrices to compute flow surrogates. Then, we compare the performance of each initialization in the performance prediction task. To this end, we use ResGatedGCN [3] as the backbone neural architecture encoder. Other setups are the same as those used in our neural architecture performance prediction experiments (Section 4.2).
19

---

NB-101 Kendall's Tau
NB-101 Precision @ 1%
NB-201 Kendall's Tau
NB-201 Precision@ 1
0:00
6:33
837
335
0CC
C:93
8:25
935
0.8
000
6:53
2:33
3:53
0.0
6:32
8:39
3.5
937
644
837
3.14
2
0.02
8.
397
0.8%
6:20
2:74
33C
2
0.00
8:30
0.18
0.14
640
1.0
033
C22
0.22
153
3.0
6655
0.36
0.2
08.5
0.54
0.0
644
110
0.00
19.
0.5
0.3
109
0.00
090
651
135
375
0.00
10.
031
0.31
144
0.00
10.5
012
Figure 11: Statistical test across diverse random initialization for flow surrogates, p-values from the Wilcoxon Signed-Rand Rggned-Rank tests [46] between pairs of performance sets, each obtained from different initializations, are reported. At the significance level of a = 0.05, none of the pairs provide sufficient statistical evidence reject the null hypothesis. Thus, different random initializations for the flow surrogate yield the same performance distribution.
Table 9: Performance prediction results under different derivations of flow surrogates. Mean and standard deviation on each dataset and metric. All values are multiplied by 100. The best performance is highlighted in blue Using random vectors to obtain flow surrogates outperforms using trained TA-GATES as flow surrogates.
Kendall's Tau
Precision @ 1%
Precision 5%%
Flow surrogates
NB-101
NB-201
NB-101
NB-201
NB-101
NB-201
TA-GATEs [34]
72.2 (5.6)
74.7 (11.4)
29.2 (1.))
40.3 (8.5)
54.4 (7.3)
60.1 (5.1)
Random vectors propagated in Section 3
74.8 (4.8)
82.2 (0.7)
37.5 (13.)
48.9 (8.1)
61.7 (3.6)
62.3 (2.5)
We leverage the Wilcoxon-SigneddRann Rank test [46] to statistically compare the effects of different initialization. For each initialization setting, conduct 9 performance prediction trials using flow surrogates generated with the corresponding initialization. For each pair of initializations, we apply the Wilcoxon Signed-Rank test to compare the performance sets derived from the respective initialization. The null hypothesis posits that the two performance distributions are identical, while the alternative hypothesis suggests they are distinct. We set the significance level a 0.05. If the p-value of the test exceeds we lack sufficient statistical evidence to reject the null hypothesis, indicating that the performance sets from different initializations follow the same distribution.
Results. As shown in Figure 11, every p-value is smaller than 0.05, resulting in we do not have sufficient statistical evidence reject the null hypothesis. These test results indicate that the performance sets derived from different random initializations of vectors and matrices for flow surrogates share the same distribution.
B.9.2 Q2. Other choices for flow surrogates
Setup. For flow surrogates, use architecture embeddings obtained by the flow-aware neural architecture encoder, TA-GATEs [34], trained through supervised learning. In each experimental trial, we first train TA-GATES using the training dataset. Next, the trained TA-GATES are employed to obtain neural architecture representations for every neural architecture in the search space. Finally, these representations serve flow surrogates for their corresponding neural architectures, which then leveraged to pre-train another neural architecture encoder, ResGatedGCN with FGP.
Results. As shown in Table 9, flow surrogates derived using random vectors and matrices (as described in Section 3) outperform those derived using TA-GATES, highlighting the effectiveness of flowing random vectors. We hypothesize that this result stems from the potential overfitting of flow-aware encoders to the training dataset, causing them to focus on specific architectural patterns present in the training data. In contrast, using random vectors likely avoids this bias, enabling the capture of broader patterns that are not limited to a specific subset the dataset.
B.10 Analysis regarding alternative design choices of FGP
Recall that the proposed flow surrogate achieved after a single round of sequential message passing (refer to Section 3.2). However, it is possible to perform message passing for multiple rounds. In this section, we analyze the effectiveness of -orrsurro variants, that achieved via multiple rounds of sequential message passing.
20

---

Table 10: Performance prediction results under variants of the flow surrogate. Mean and standard deviation on each dataset and metric. All values are multiplied by 100. The best performance is highlighted in blue Using only single round flow surrogate yields the best performance.
Kendall's Tau
Precision @ 1%
Precision @ 5%
Propagation rounds
NB-101
NB-201
NB-101
NB-201
NB-101
NB-201
Using only round I (Ours)
74.8 (4.8)
82.2 (0:7)
37.5 (13.0)
48.9 (8.1)
61.7 (3.6)
62.3 (2.5)
Using rounds and
71.7 (5.4)
80.6 (0.5)
36.7 (5.9)
46.0 (6.0)
58.8 (3.4)
62.0
Using rounds 1, 2, and
70.7 (4.7)
81.6 (0.8)
34.0 (8.9)
47.3 (0.6)
58.2 (4.1)
62.5 (1.0)
Table 11: Performance prediction results under various pooling functions. Mean and standard deviation on each dataset and metric. All values are multiplied by 100. The best performance is highlighted in blue Kendall's Tau is used as evaluation metric. Sum pooling outperforms other pooling functions.
Mcan (variant 1)
Max (variant 2)
Sum (Ours)
NB-101
71.6 81.7 (1.0)
71.1 (6.4)
74.8 (4.8)
NB-201
81.5 (0.9)
82.2 (0.7)
Setup. We use two variants:
(Variant 1) This variant uses two surrogates: (1) flow surrogate obtained via a single message passing round and (2) flow surrogate obtained via two consecutive message passing rounds. It uses different projection heads for respective flow surrogate reconstruction. Reconstruction is performed in joint manner.
(Variant 2) This variant uses three surrogates: flow surrogate obtained via (1) single message passing round, (2) flow surrogate obtained via two consecutive message passing rounds, and (3) a flow surrogate obtained via three consecutive message passing rounds. uses different projection heads for respective flow surrogate reconstruction.
All methods use ResGatedGCN backbone neural architecture encoder. Other setups are the same as those used our neural architecture performance prediction experiments (Section 4.2).
Results. As shown in Table 10, a flow surrogate achieved via single round of message passing, which is our proposed method, outperforms the alternatives in five out of six settings, justifying our design choice of flow surrogate.
B.11 Alternative pooling strategies
Note that the pooling to aggregate the messages from the neighbors (Section 3.2). In this section, we analyze the alternative pooling functions that are widely used in GNN literature.
Setup. We use the two strategies: (1) mean pooling and (2) max pooling. Specifically, we replace the sum pooling function with one of the two functions and use the resulting flow surrogate as the final flow surrogate. We use ResGatedGCN as the backbone neural architecture encoder, and other settings are the same as Section 4.2.
Results. As shown in Table 11, sum pooling, which FGP uses to obtain flow surrogate, outperforms alternative strategies, demonstrating the effectiveness of design choice for pooling function.
B.12 Expressiveness our flow surrogate
In this section, we provide preliminary analysis regarding the expressiveness of our flow surrogate.
Setup. We analyze whether our flow surrogate can discriminate architectures containing different operations. We conduct three binary classification tasks to determine whether a given neural architec- ture includes: (1) 1xl convolution operation, (2) a 3x3 convolution operation, and (3) pooling operation. We use flow surrogate representations as input features and train an MIP as the classifier, performing 10 trials. The architectures are split into 80% for training and 20% for testing.
21

---

Table 12: Operation classification results. Mean and standard deviation each dataset and metric. All values are multiplied by 100. In every case, flow surrogates achieve accuracy higher than 92.
1 x 1 Conv.
3 x 3 Conv.
Pooling
NB-101
93.2 (0.3)
99.9 (0.1)
92.1 (0.7)
NB-201
98.9 (0.3)
99.9 (0.1)
1.0 (0.1)
Table 13: Performance under diverse backbone architecture encoders. Mean and standard deviation on each dataset and metric. All values are multiplied by 100. Kendall's Tau used as an evaluation metric. The best performance is highlighted in blue Since the MI.P-based model does not return node embeddings, GMAE is not applicable. Across diverse backbone encoders, FGP
consistently outperforms the baseline methods.
Backbone
Datasets
wo pre-training
GMAE
ZC-Proxy
FGP
NB-101
42.5 (5.4)
49.8 (7.3)
65.2 (4.1)
MLP-based
NB-201
59.6 (2.2)
70,2 (1.8)
72.3 (1.3)
NB-101
62.7 (3.6)
63.2 (4.5)
64.1 (8.5)
69.7 (6.3)
Transformer based
NB-201
63.2 (2.0)
65.7 (2.4)
69.2 (3.2)
74.3 (1.5)
Results. As shown in Table 12, our flow surrogate achieves higher than 92% accuracy in all the
cases. This result suggests that our flow surrogate can effectively distinguish architectures containing different operations, achieving
B.13 Theoretical properties our flow surrogate
this section, we investigate the theoretical properties of our flow surrogate. In nutshell, we provide preliminary theoretical analysis demonstrating that our flow-surrogate computation is invariant to node permutations (i.e., node indexing), a desirable property when representing the information flow of neural architccture ensure effective downstream task performance.
Motivation. [15, 34] demonstrates the importance of accurately capturing information flow for the downstream tasks (e.g.. performance prediction and neural architecture search). In the NAS-Bo AAS-Bench datasets used in our experiments, the information flow within neural architecture independent of the indexing of individual operations. Accordingly, it is desirable for the computation of flow with respect to these operations.
surrogates be permutation-invaaree
Proof sketch Thost and Chen [40] theoretically demonstrates that asynchronous message passing on directed acyclic graph-also used in our flow-surrogate computation-is invariant node permutations when the permutation invariant pooling function is employed for neighbor aggregation. Their theoretical result extends to our setting, indicating that our method likewise ensures permutation invariance regarding node indexing.
B.14
Alternative backbone neural architecture encoders
Recall that we use graph-based neural architecture encoders for our experiments, given that neural architectures are often expressed as graphs. In this section, we investigate the effectiveness of FGP under non- non-graph-Be based neural architectures.
Setup. We two non- con-graph--oo based neural architecture encoders: MLP-based model [44] and Transformer-based model [31]. Specifically, we first pre-train each architecture encoder via a certain pre-training method, and then fine-tune the pre-trained encoder with the performance prediction task. Other settings are the same as in Section 4.2.
Results. As shown in Table 13, FGP consistently outperforms the baseline methods across diverse non-graph-baan based backbone neural architecture encoders, demonstrating that the effectiveness of FGP is not limited particular graph-based backbone encoder.
B.15 Hyperparameter sensitivity
In this section, we investigate the hyperparameter sensitivity of FGP.
22

---

Table 14: Hyperparameter sensitivity analysis. Mean and standard deviation each dataset and metric. All values are multiplied by 100. Kendall's Tau is used as an evaluation metric. Across diverse settings of A and X2, FGP consistently outperforms the baseline methods.
(A1, (3
(A1, = (
(A1, Aa) = (3) ZC-Proxy
GMAE
NB-101
74.8 (4.8)
74.8 (4.4)
74.3 (4.5)
68.3 (6.7)
68.1 (4.7)
NB-201
82.2 (0.7)
82.3 (0.8)
81.7 (0.8)
79.9 (0.8)
74.8 (1.2)
Table 15:
FlowerFormer analysis under diverse pre-training data size. Mean and standard deviation on each dataset and metric. All values are multiplied by 100. Kendall's Tau is used as an evaluation metric. Performance of the FlowerFormer pre-trained with FGP tends increase as the size of the pre-training data increases.
Pre-training data ratio
0%
20%
40%
60%
80%
100%
NB-101 NB-201
74.0 (3.6)
74.6 (3.8)
75.9 (1.9)
75.8 (4.0)
76.2 (5.1)
76.3 (3.6)
77.3 (1.5)
81.1 (0.8)
82.3 (1.2)
82.4 (1.3)
82.7 (1.4)
83.5 (1.7)
Setup. We analyze the coefficients of each loss term, which are λι and X2 described in Sec- tion 3.3. As detailed in Appendix A.4, search space of the coefficients are as follows: (2) e {(3 (2. ())((,))) }. Therefore, measure the performance of FGP under each configuration. We use ResGatedGCN as the backbone neural architecture encoder, and other settings are the same as in Section 4.2.
Results. As shown in Table 14, across diverse settings of X and X2 FGP consistently outperforms the baseline methods, demonstrating the robustness of FGP under the choice of \ and X2
B.16 Analysis regarding performance gain in FlowerFormer
Recall that FlowerFormer [15] is flow-based encoder, which captures information flow within the neural architecture by using its architectural design. However, as shown in Table 1, the performance of FlowerFormer even improves with FGP. In this section, we further investigate this phenomenon. To this end, we first present the high-level hypothesis regarding the reason behind this phenomenon, and provide experiments that further support our hypothesis.
Hypothesis. We hypothesize that FGP enables FlowerFormer to learn a broader range of information flow beyond what is present in labeled architectures by leveraging large number of unlabeled architectures, thereby enhancing its ability to capture more diverse flow patterns. When trained solely on labeled samples, FlowerFormer tends to learn information flow patterns limited that specific set. In contrast, FGP leverages a much larger pool of neural architectures whose ground-trui ttunh performance is unknown, exposing FlowerFormer to wider range of information flow patterns. This exposure enables Flowerforrmr to learn more diverse and generalizable information flow representations.
Setup. By varying the number of neural architectures used during FGP pre-training, we aim aim-though not precisely-to control the diversity of information flow patterns to which FlowerFormer is exposed. A positive correlation between dataset size and FlowerFormer performance suggests that FGP helps FlowerFormer encounter a broader range of information flow patterns, leading to improved accuracy in performance prediction. We vary the proportion of the pre-training dataset used for FGP training- 0% (no pre-trate training), 20%, 40%, 60%, 80%, and 100%- and evaluate FlowerFormer's s performance on the performance prediction task. Kendall's Tau is used as the evaluation metric, and all other settings follow those described in Section 4.2.
Results. As shown in Table 15, the performance of FlowerFormer pre-trained with FGP tends to increase as the size of the pre-training dataset increases. This result supports our hypothesis on the performance improvement of the flow- encoder (FlowerFormer) with FGP.
23

---

C Flow surrogate details
C.1 Details of node embeddings and messages
In this section, we elaborate on the node embeddings (i.e., operation representations) h, E RR, Ve; V, messages of the order-1 nodes r, and projection matrix W IR 2
Node embeddings. We first randomly generate transformation matrix P RIO| where O is a set of all the possible operations. Each entry of P is sampled from (0,, o²) independently to each other. Then, we multiply P with node features X, obtaining node embedding matrix (i.e., H XP), where H..- h, holds.
Messages of the order-1 nodes. We randomly sample each element of r from the uniform distribution (0,11) 1).
Projection matrix. We randomly sample each element W from (N,
C.2 Hyperparameters for obtaining flow surrogate
We provide search space fixed value for each hyperparameter related to the flow surrogate:
Standard deviation of Gaussian distribution a. This is tuned within [10-!,10%)
Dimension k. This is tuned within (4, ... ,
Identity coefficient a. This is fixed as a = 0.5.
D
Zero-cost proxy details
this section, we provide further details regarding zero-cost proxy, which we use its prediction as an auxiliary training objective of FGP.
D.1 Descriptions of zero-cost proxies
metrics that represent certain characteristics of neural architecture [1]. For given neural architecture, these metrics can be obtained by simply building a deep learning model having the corresponding neural architecture (e.g., 2 norm of parameters) or performing a single forward pass and gradient computation (e.g., le norm of parameters' gradients). Therefore, compared to actual model training training-which requires a substantial number of training iterations (i.e., forward passes and packpropagation--ootiiinn zero-cost proxies significantly more economical [1].
Overview. Zero-cost proxies are
Usage of zero-cost proxies. Numerous studies on Neural Architecture Search (NAS) have demon- strated that these metrics exhibit a strong correlation with the ground-truth performance of neural architectures [1, 56, 14]. Consequently, various NAS approaches leverage zero-cost proxies to quickly assess the potential effectiveness candidate architectures for the target task and dataset [14, 27]. Similarly. Zhao et al. [56] proposed pre- vre-training neural architecture encoder predict the zero-cost proxies of a neural architecture, enabling the encoder to identify architectures more likely to achieve high performance. Note that this zero-cost-predee cenest-prriccto-baasseeee pre-train pre-training [56] has been leveraged as our baseline method, which is called ZC-Proxy. Specifically, ZC-Proxy pre-trains the encoder following the scheme described in Appendix D.2.
D.2 Our usage of zero-cost proxy
Overview. Motivated by the recent success of zero-cost proxies in NAS, we use predicting zero-cost proxy of a neural architecture as an auxiliary learning objective Laue- We use Jacobian Covariance zero-cost proxy [1]. measuring how well a neural architecture distinguishes distinct inputs. We train neural architecture encoder to predict the Jacobian Covariance of given neural architecture. For NAS-Bench- 101 [51] and NAS-Bench-301 [39], we use proxies provided by Krishnakumar et al. [25]. For
[8]. we use proxies provided by Abdelfattah et al. [1].
Note that all the order-1 nodes share the same embedding

---

Usage. Consider (()) E RR, an embedding of neural architecture G( obtained by neural architecture encoder fa. We use pero-cost-proxy regressor Up predict the architecture's zero-cost proxy, denoted as e R. Specifically, the predicted zero-cost proxy ali) E IR is obtained as follows: a(1) up ). Since the objective of learning zero-cost proxies is train model to identify which architectures are more likely to perform better, we adopt a ranking- based loss. This loss encourages model to focus on relative performance, specifically identifying which architecture outperforms another. To achieve this, we use margin ranking loss, which is widely used in training neural architecture encoder [34, 15]. The loss function formalized as:
Laux
Σ
max (0, (2(0) - =(5))),
(e)))
where g and go) are two neural architectures within the same batch, and m a margin hyperpa- rameter. The parameters 0 and p are optimized using gradient descent to minimize the auxiliary loss Lauz (Eq. 2).
E
Discussions
In this section, we discuss limitations and broader impacts of our research.
E.1
Limitations and potential future works
While our work demonstrates empirical effectiveness neural architecture encoding across various benchmark datasets and applications (Section 4), the theoretical properties of FGP remain under- what types of flows it can or cannot capture. While we provide preliminary analysis regarding these concepts in Appendices B.12 and B.13, a further rigorous and in-depth analysis regarding them would enrich our understanding of our method. Thus, further theoretical investigation into flow surrogate offers promising directions for improvement. In addition, given findings that LLMs understand graphs and produce interpretable graph representations [21], incor- porating them into FGP is a promising path improve interpretability. Moreover, several recent methods represent neural architectures in diverse formats, such as hypergraph [28], which model interactions among multiple nodes [20]. Therefore, extending FGP to such approaches would be interesting direction.
E.2 Broader impacts
While our work focuses neural architecture encoding, approach has the potential to generalize to other data structures that exhibit a notion of flow, such as electrical circuits [2] and Petri Nets [36]. Accordingly, our flow surrogate presents practical opportumities for representing such structures in these domains.
25

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We propose generative- method for neural architecture encoding.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss them in Appendix E.
Guidelines:
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer:
[Yes]
Justification: We discuss them Appendix 1.1..
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experi- mental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: We present detailed experimental settings and hyperparameters, together with the source code.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We make all our code and datasets publicly available through https: com/kswoo97 /FGP/APPART Anom.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We provide detailed settings and hyperparameter search space of our experiments.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We provide mean and standard deviation of our experimental results obtained via multiple trials.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experi- ments?
Answer: [Yes]
Justification: We provide detailed information regarding the machines used in experiments.
9. Code of ethics
26

---

Question:
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurip:: http://p...ppp...t..a..uuu.ua.aa
Answer: [Yes]
Justification: We carefully checked the NeurIPS Code of Ethics, and believe that we did not violate any of the ethical terms.
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss them in Appendix E.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: We believe that our work does not pose such risks.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We adequately cited existing works.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide detailed README instructions for our code https://githhe com/ kawoo97/FGPAnon
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: We not contain any crowdsourcing and human-subjectt--relted experiments.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper deseribe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: We do not contain any crowdsourcing and Human-subbjccs--relteee experiments.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, non- standard component of the core methods this research? Note that if the I.LM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Question:
Answer: [NA]
Justification: We only used LLMS improve the writing.
27


Reviews:
[{"review_id": "8jalb6GHY5", "reviewer": "Reviewer_RQZD", "summary": "This paper proposes FGP (Flow-based Generative Pre-training) which is a novel pre-training method for neural architecture encoding. The key idea is to train a neural architecture encoder to capture information flow within a neural architecture without requiring specialized and computationally expensive model structures. Specifically, FGP trains an encoder to reconstruct this flow surrogate, enabling it to learn information flow without complex model structures.", "questions": "Address the points raised above under weaknesses", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- The paper argues that FGP can make simpler flow-aware encoders and provides clear training guidance. The claim is backed by the experimental results to show that FGP significantly boosts encoder performance with gains of up to 106% when compared to encoders trained solely with supervised learning. \n- FGP is substantially faster and consistently outperforms baseline pre-training methods in a majority of settings. \n \nOverall, the paper presents a well-argued and experimentally supported novel pre-training method for neural architecture encoding that addresses critical challenges of efficiency and effectiveness in the field.\n\nWeaknesses:\n- The paper acknowledges that FGP demonstrates empirical effectiveness across various benchmark, further theoretical investigation into their flow surrogates would make this work very strong.\n- Even though the authors have touched on the issue that they have limited understanding of the types of information flows that FGP could capture, it would be good to see a thourough discussion around it. \n- Without the above two points, the claims are purely emperical and not strong.", "was_revised": true, "final_justification": "Thanks for addressing the issues raised in my feedback. I've raised my score.", "rebuttal": "Dear Reviewer RQZD,\n\nWe deeply appreciate your invaluable feedback on our theoretical analysis and the discussion of our flow surrogate.\n\nBelow, we provide detailed responses to each of your comments.\n\nBest regards,\n\nThe Authors\n\n---\n\n# Weakness 1\n\n`The paper acknowledges that FGP demonstrates empirical effectiveness across various benchmark, further theoretical investigation into their flow surrogates would make this work very strong.`\n\n# Response 1\n\n- **[Permutation invariance]** In response to the reviewer’s suggestion, we provide a preliminary theoretical analysis demonstrating that our flow-surrogate computation is invariant to node permutations (i.e., node indexing), a desirable property when representing the information flow of a neural architecture to ensure effective downstream task performance.\n    - **[Intuition]** Prior works [1, 2] have shown that accurately capturing the information flow within a neural architecture is critical for downstream tasks, such as performance prediction. Therefore, for our flow surrogate to be effective in such tasks, it should accurately represent this information flow.\n    - **[Crucial aspect of flow]** In the NAS-Bench datasets used in our experiments, the information flow within a neural architecture is independent of the indexing of individual operations. Accordingly, it is desirable for the computation of flow surrogates to be permutation-invariant with respect to these operations.\n    - **[Proof sketch]** Thost and Chen [3] theoretically demonstrate that asynchronous message passing on a directed acyclic graph—also used in our flow-surrogate computation—is invariant to node permutations when the permutation-invariant pooling function is employed for neighbor aggregation. Their theoretical result extends to our setting, indicating that our method likewise ensures permutation invariance. We will add this analysis to our revised manuscript.\n\n---\n\n# Weakness 2\n\n`Even though the authors have touched on the issue that they have limited understanding of the types of information flows that FGP could capture, it would be good to see a thourough discussion around it.`\n\n# Response 2\n\nOur response is two-fold: (1) empirical evidence supporting the effectiveness of the flow surrogate in capturing information flow, and (2) a discussion of the challenges in providing more rigorous theoretical justification.\n\n- **[New experiments: Preliminary analysis]** We present a new preliminary analysis suggesting that our flow surrogate can differentiate neural architectures with distinct operations, resulting in different patterns of information flow.\n    - **[Setting]** We conduct three binary classification tasks to determine whether a given neural architecture includes: (1) a 1×1 convolution operation, (2) a 3×3 convolution operation, and (3) a pooling operation. We use flow surrogate representations as input features and train an MLP as the classifier, performing 10 trials. The architectures are split into 80% for training and 20% for testing.\n    - **[Results]** As shown in Table 1, our flow surrogate achieves over 92% accuracy across all cases, indicating its ability to distinguish architectures with different operations that give distinct patterns of information flow.\n\n**Table 1.** Operation classification results. The mean and standard deviation over 10 trials are reported.\n\n|  | 1x1 Conv | 3x3 Conv | Pooling |\n| --- | --- | --- | --- |\n| NAS-Bench-101 | 93.2 (0.3) | 99.9 (0.2) | 92.1 (0.7) |\n| NAS-Bench-201 | 98.9 (0.3) | 99.9 (0.1) | 1.0 (0.0) |\n- **[Challenges]** Unfortunately, a more rigorous theoretical analysis is non-trivial, primarily due to two key challenges:\n    - **[Definition of flow]** It is challenging to formally define information flow within a neural architecture with a single mathematical formulation. This is because the actual information flow is highly dependent on the specific parameter values, which can vary significantly across models.\n    - **[Non-linearity]** Both the actual neural architecture and our flow-surrogate computation involve multiple non-linear functions, which complicates the derivation of rigorous theoretical conclusions.\n\n---\n\n# Weakness 3\n\n`Without the above two points, the claims are purely emperical and not strong.`\n\n# Response 3\n\nGiven our responses 1 and 2, while our additional results may not fully meet the reviewer’s expectations, we believe they adequately address the two concerns raised.\n\n- **[Regarding point 1]** In response to the concern about the lack of theoretical analysis, we provide a theoretical justification showing that our flow surrogate computation is permutation invariant with respect to node ordering, which is an essential property for effectively capturing information flow.\n- **[Regarding point 2]** In response to the concern regarding the lack of thorough analysis on the information flow captured by the surrogate, we provide a preliminary analysis indicating that our flow surrogate can distinguish information flows involving distinct operations.\n\n---\n\n# References\n\n- [1] FlowerFormer: Empowering Neural Architecture Encoding using a Flow-aware Graph Transformer\n- [2] TA-GATES: An Encoding Scheme for Neural Network Architectures\n- [3] Directed Acyclic Graph Neural Networks"}, {"review_id": "0qCWtTk4yD", "reviewer": "Reviewer_df83", "summary": "This work addresses the problem of training neural architecture encoders, or models that can predict the downstream task performance given a proposed architecture description. Specifically, this work proposes an representation that captures the \"flow\" of a target model, where the flow refers to information flow w.r.t. the forward and backward passes. The proposed pre-training objective in this work allows an encoder to learn to predict the flow, which previously required specific architecture components to capture. \nThis is accomplished by first assigning a topological ordering over elements in the architecture, then tracing the forward and backward pass of the candidate architecture. This information is aggregated into an embedding that an encoder-decoder pair are trained on with a reconstruction objective. \n\nThe proposed method is evaluated for downstream performance prediction, as well as utility in predictor-aware-NAS. \nAn ablation study over the different constituent components is used to validate the efficacy of each.", "questions": "See weaknesses section", "limitations": "Yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n* The proposed method seems novel with respect to related work. Framing the pretraining objective on the flow-based surrogate itself seems like a more flexible design space compared to modeling it directly with the encoder\n* The paper is well-written and easy to follow\n* Experiments show a consistent (sometimes sizeable) increase in performance \n\nWeaknesses :\n* The values of \\lambda_1 and \\lambda_2 are not mentioned, only their search space. I'm concerned that the performance of the method might be tuned separately for each test setting, or that performance is not robust to the choice of \\lambda\n* It seems to me that a major part of this work is that no \"special components\" are needed to predict the flow, as that information is now captured in the embedding. That unlocks a large design space of less-constrained encoders. However, I don't see any mention of what other encoder designs / architectures were considered. \n\n\nOther comments:\nline 111: variation -> variational", "was_revised": true, "final_justification": "After a detailed rebuttal regarding issues with hyperparameter specification and encoder design experiments, I have raised my score.", "rebuttal": "Dear Reviewer df83,\n\nWe sincerely appreciate your invaluable feedback on our hyperparameter and encoder analyses.\n\nBelow, we provide detailed responses to each of your comments.\n\nBest regards,\n\nThe Authors\n\n---\n\n# Weakness 1\n\n`The values of \\lambda_1 and \\lambda_2 are not mentioned, only their search space. I'm concerned that the performance of the method might be tuned separately for each test setting, or that performance is not robust to the choice of \\lambda`\n\n# Response 1\n\nOur response is two-fold.\n\n- **[Clarification]** We apologize for the missing detail. All hyperparameters were tuned using the validation set, which is described in Section 4.1 of our manuscript.\n    - **[Configuration upload]** We will report the validation-best hyperparameter configurations to our revised manuscript and upload them to Anonymous GitHub after the rebuttal phase since the rebuttal policy prohibits uploading additional content during this period.\n- **[New Experiments: Robustness Analysis]** Our method is robust to the choice of the $\\lambda_{2}$ and $\\lambda_{2}$ hyperparameters.\n    - **[Setting]** Recall that our search space for $\\lambda_{1}$ and $\\lambda_{2}$ includes three configurations: $(\\lambda_1$, $\\lambda_2) \\in \\lbrace (\\frac{1}{3}, \\frac{2}{3}), (\\frac{2}{3}, \\frac{1}{3}), (\\frac{1}{2}, \\frac{1}{2}) \\rbrace $, as detailed in Appendix A.4 of our manuscript. In this additional experiment, we report the performance prediction task results under each configuration. We use ResGatedGCN [1] as our backbone architecture encoder, and other settings are the same as those in Section 4.2 of our manuscript.\n    - **[Results]** As shown in Table 1, the performance variations across these settings remain marginal and consistently outperform the strongest baselines (GMAE [2] and ZC-Proxy [3]). These results highlight the robustness of our method to hyperparameter choices.\n\n**Table 1.** Results of performance prediction under each hyperparameter configuration.\n\n|| $(\\lambda_1, \\lambda_2) = (\\frac{1}{2}, \\frac{1}{2})$ | $(\\lambda_1, \\lambda_2) = (\\frac{1}{3}, \\frac{2}{3})$ | $(\\lambda_1, \\lambda_2) = (\\frac{2}{3}, \\frac{1}{3})$ | ZC-Proxy | GMAE |\n| --- | --- | --- | --- | --- | --- |\n| NAS-Bench-101 | 74.8 (4.8) | 74.8 (4.4) | 74.3 (4.5) | 68.3 (6.7) | 68.1 (4.7) |\n| NAS-Bench-201 | 82.2 (0.7) | 82.3 (0.8) | 81.7 (0.6) | 79.9 (0.8) | 74.8 (1.2) |\n\n---\n\n# Weakness 2\n\n`It seems to me that a major part of this work is that no \"special components\" are needed to predict the flow, as that information is now captured in the embedding. That unlocks a large design space of less-constrained encoders. However, I don't see any mention of what other encoder designs / architectures were considered.`\n\n# Response 2\n\nOur response is two-fold.\n\n- **[Clarification]** Sorry for the missing detail. Since a neural architecture can be represented as a graph (see Section 2.2 of our manuscript), we adopt graph neural networks (ResGatedGCN and GIN) and a graph transformer (FlowerFormer) as backbone architecture encoders. We will incorporate these details in our revised manuscript.\n    - **[Detail]** This choice aligns with common practice in neural architecture encoding work [2, 3, 4], where graph neural networks and graph transformers are widely used to encode neural architectures.\n- **[New Experiments: Additional Encoders]** The superiority of our method over baseline pre-training methods holds across different types of neural architecture encoders.\n    - **[Clarification]** As the reviewer noted, our method is compatible with a wide range of neural architecture encoders.\n    - **[Setting]** To validate this, we additionally evaluate our method using two alternative encoders: an MLP-based encoder [5] and a transformer-based encoder [6]. All other experimental settings follow those described in Section 4.2 of our manuscript. We compare against the two strongest baselines, GMAE and ZC-Proxy.\n    - **[Results]** As shown in Tables 2 and 3, our method (FGP) outperforms both baselines, demonstrating that its effectiveness holds valid under other choices of backbone neural architecture encoders.\n\n**Table 2.** Results of performance prediction using the MLP-based architecture encoder. Since this encoder does not produce node embeddings, the node-feature-generative method (GMAE) is not applicable.\n\n|  | w/o pre-training | GMAE | ZC-Proxy | FGP (ours) |\n| --- | --- | --- | --- | --- |\n| NAS-Bench-101 | 42.5 (5.4) | - | 49.8 (7.3) | **65.2 (4.1)** |\n| NAS-Bench-201 | 59.6 (2.2) | - | 70.2 (2.8) | **72.3 (1.3)** |\n\n**Table 3.** Results of performance prediction using the Transformer-based architecture encoder.\n\n|  | w/o pre-training | GMAE | ZC-Proxy | FGP (ours) |\n| --- | --- | --- | --- | --- |\n| NAS-Bench-101 | 62.7 (3.6) | 63.2 (4.5) | 64.1 (8.5) | **69.7 (6.3)** |\n| NAS-Bench-201 | 63.2 (2.0) | 65.7 (2.4) | 69.2 (3.2) | **74.3 (1.5)** |\n\n---\n\n# Weakness 3\n`Other comments: line 111: variation -> variational`\n\n# Response 3\n\nThank you for pointing out the typos. We will revise all identified typos, including the one mentioned by the reviewer.\n\n---\n\n# References\n\n- [1] Residual Gated Graph ConvNets\n- [2] Graph Masked Autoencoder Enhanced Predictor for Neural Architecture Search\n- [3] Dynamic Ensemble of Low-Fidelity Experts: Mitigating NAS “Cold-Start”\n- [4] Does Unsupervised Architecture Representation Learning Help Neural Architecture Search?\n- [5] A study of neural architecture encoding\n- [6] TNASP: A Transformer-based NAS Predictor with a Self-evolution Framework"}, {"review_id": "OkyrWblJak", "reviewer": "Reviewer_VhSE", "summary": "This paper aims to predict the performance of neural architectures effectively and efficiently. Specifically, the flow surrogate is proposed to simulate the ground-truth information flow, which helps to train the architecture encoder effectively. Then, the trained encoder can be used for the performance prediction of most of the neural architectures.", "questions": "Please refer to the Weaknesses.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 1, "clarity": 1, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Weaknesses:\n\n1. There are some typos in this paper. Line 23: 'extensive' computational resources.....\n\n2. The insight of this paper is not solid enough. The proof that the 'flow surrogate' can effectively represents the information flow is lacked. Moreover, after reviewed the details of the generation of 'flow surrogate', there is no constraint that ensures different information surrogates for two architectures with largely different performance, especially when the the inputs are random initialized.\n\n3. There is no clear explanation that why the current flow-based methods are computationally heavy. Given the inference time of GNN, it is hard to understand why it takes such a long computation time.\n\n4. Figure 3 is confusing. In the middle sub-figure of Process 1, why the 'In' information is directly used for the computation of the 'Out'? Even they are not directly connected. Is this means that all historical information will be used when computing the high-level information? Will this cause a requirement of large memory?\n\n5. The sum pooling for multiple incoming information is not reasonable, since the summation can not distinguish the incoming source which is too rough for simulating the real information flow.", "was_revised": true, "final_justification": "Please ensure that the clarifications from the rebuttal are added in the final version.", "rebuttal": "Dear Reviewer VhSE,\n\nWe sincerely appreciate your valuable feedback on the flow surrogate analysis and presentation.\n\nBelow, we provide detailed responses to each of your comments.\n\nBest regards,\n\nThe Authors\n\n---\n# Weakness 1\n`There are some typos in this paper. Line 23: 'extensive' computational resources`\n\n# Response 1\nThank you for pointing out the typos. We will revise the manuscript to address all identified typos, including the case you mentioned (replacing ‘extensive’ with ‘expensive’).\n\n---\n# Weakness 2.1\n`The insight of this paper is not solid enough. The proof that the 'flow surrogate' can effectively represents the information flow is lacked.`\n\n# Response 2.1\n- **[New preliminary analysis]** While rigorous proof is challenging, we present a new analysis showing that our flow surrogate can distinguish neural architectures with distinct operations, resulting in different patterns of information flow.\n  - **[Setup]** We conduct three binary classification tasks to determine whether a given architecture includes: (1) a 1×1 conv. operation, (2) a 3×3 conv. operation, and (3) a pooling operation. We use flow surrogate representations as input features and train an MLP as the classifier, performing 10 trials. The architectures are split into 80%/20% for training/testing.\n  - **[Result]** As shown in Table 1, our flow surrogate achieves over 92% accuracy across all cases, indicating its ability to distinguish architectures with different operations that give distinct patterns of information flow.\n\n**Table 1.** Operation classification results. \n||1x1 Conv|3x3 Conv|Pooling|\n|---|---|---|---|\n|NB-101|93.2 (0.3)|99.9 (0.2)|92.1 (0.7)|\n|NB-201|98.9 (0.3)|99.9 (0.1)|1.0 (0.0)|\n\n---\n# Weakness 2.2\n`Moreover, after reviewed the details of the generation of 'flow surrogate', there is no constraint that ensures different information surrogates for two architectures with largely different performance, especially when the inputs are random initialized.`\n\n# Response 2.2\nWe acknowledge the reviewer’s concern that the flow surrogate may lack a strong constraint to effectively distinguish architectures with distinct performance. However, we offer evidence suggesting that our flow surrogate can make such distinctions. Our response is two-fold.\n\n- **[Effectiveness of flow]** We claim that the ability to distinguish architectures based on their information flow allows us to differentiate between high- and low-performing architectures.\n  - **[Evidence 1]** As shown in Fig. 6 of our paper, our flow surrogates effectively separate high- and low-performing architectures in the embedding space, even without any knowledge of the architecture performance.\n  - **[Evidence 2]** As detailed in Sec. 1 and 2 of our paper, SOTA architecture encoders designed to capture information flow [1, 2] have shown remarkable success in performance prediction tasks, suggesting that the flow is a key feature for accurately estimating architecture performance.\n- **[Effectiveness of random vectors]** We also claim that random vectors can still effectively capture information flow.\n  - **[Evidence 1]** Prior work [3] shows that random vectors can yield meaningful representations when combined with adequate computational structures (e.g., convolutional layers). Similarly, our propagation mechanism enables random vectors to capture the flow.\n  - **[Evidence 2]** Compared to learning-based alternatives, our flow surrogate outperforms architecture embeddings derived from the supervised-trained flow-aware encoder TA-GATEs [2] (see Table 2). See Appendix B.9.2 for details.\n  - **[Evidence 3]** Note that our method is robust to the choice of random vectors, with different initializations consistently yielding similar performance. See Appendix B.9.1 for details.\n\n**Table 2.** Comparison with a learning-based alternative.\n|Metric: Kendal Tau|Learning-based|Random-vector-based (ours)|\n|---|---|---|\n|NB-101|72.2 (5.6)|**74.8 (4.8)**|\n|NB-201|74.7 (1.4)|**82.2 (0.7)**|\n\n---\n# Weakness 3\n`There is no clear explanation that why the current flow-based methods are computationally heavy. Given the inference time of GNN, it is hard to understand why it takes such a long computation time.`\n\n# Response 3\n- **[Clarification]** Sorry for the missing detail. The computational burden of flow-based encoders comes from their asynchronous message passing, which is difficult to parallelize. We will elaborate on these details in the revised manuscript.\n  - **[Mechanism]** To capture information flow, they sequentially pass messages according to the topological order of nodes in a directed acyclic graph (DAG), instead of passing messages between all linked node pairs at once.\n  - **[Detail 1]** As detailed in Sec. 3.2.1, each node is assigned a topological order based on its position in the DAG. Nodes with no incoming edges (typically the input node) are assigned order 1. Nodes with incoming edges only from order-1 nodes are assigned order 2, and nodes with incoming edges only from order-1 and -2 nodes are assigned order 3, and so on.\n  - **[Detail 2]** Message passing begins at the order-1 nodes, which send messages to order-2 nodes, where the messages are updated. These updated messages are then moved to order-3 nodes. The process continues through the DAG, passing messages to later-order nodes until reaching the highest-order node (typically the output node). The process is then reversed: messages are propagated backward through the graph until they reach the input node again. This process constitutes a single message-passing layer in the flow-based encoder. Note that this process is hard to parallelize, as each sub-step depends on the output of the preceding one.\n  - **[Summary]** Thus, unlike typical GNNs that perform message passing in a single step per layer, flow-based encoders need multiple steps per layer proportional to twice the length of the longest path from the input node to the output node, which is hard to parallelize.\n\n---\n# Weakness 4\n`Figure 3 is confusing. In the middle sub-figure of Process 1, why the 'In' information is directly used for the computation of the 'Out'? Even they are not directly connected. Is this means that all historical information will be used when computing the high-level information? Will this cause a requirement of large memory?`\n\n# Response 4\nSorry for the confusion. Our response is three-fold, which we will elaborate on in the revised manuscript.\n- **[Figure clarification]** We represent deactivated computations using grey-colored arrows. Thus, when updating the embedding of a given node, only the embeddings of nodes directly connected via incoming edges are used; others are not used.\n  - **[Detail]** The middle sub-figure of Process 1 shows the computation for updating the embeddings of ‘1×1’ and ‘3×3’ operations. Here, the embeddings of the respective operations and the ‘in’ node are used, while the ‘out’ node is not involved.\n- **[Embedding clarification]** We do not store all historical embeddings. In the forward pass, order-k nodes’ embeddings are updated using only order-(k-1) and current order-k nodes’ embeddings; in the backward pass, only order-(k+1) and order-k nodes’ embeddings are used.\n  - **[Detail]** Node ordering is detailed in our Response 3 and Sec. 3.2.1 of our paper.\n- **[New experiments: Memory & time]** Our flow-surrogate computation requires less than 10MB of memory and takes under 1 minute to process 14K architectures.\n  - **[Setup]** We measure the difference in memory usage before and after the computation, along with the execution time.\n  - **[Result]** As shown in Table 3, our method processes 14K architectures (# of architectures in NB-101 and 201) in under 1 minute while using less than 10 MB of memory.\n\n**Table 3.** Memory and time consumption in obtaining flow surrogates.\n||Memory usage (MB)|Time consumtion (sec.)|\n|---|---|---|\n|NB-101|9.2|31|\n|NB-201|9.4|32|\n\n---\n# Weakness 5\n`The sum pooling for multiple incoming information is not reasonable, since the summation can not distinguish the incoming source which is too rough for simulating the real information flow.`\n\n# Response 5\n\nOur response is two-fold.\n- **[Clarification]** While sum aggregation (pooling) is permutation-invariant, our flow surrogate computation mechanism allows the distinction between different sources of incoming information.\n  - **[Goal]** We need to distinguish information from different sources when each source involves distinct operations (since sources with identical operations yield indistinguishable information, we do not attempt to differentiate them).\n  - **[Detail 1]** As detailed in Sec. 3.2, messages (i.e., vector representations of incoming information) are transformed based on the operations they pass through, with each operation type having its own projection function. Thus, messages that have undergone different operations are likely to be distinct.\n  - **[Detail 2]** Therefore, since messages are sufficiently distinct when originating from different operations, the aggregation function does not need to explicitly account for the specific operation each message has undergone.\n- **[New experiments: Alternatives]** We show that sum aggregation outperforms other commonly used aggregation functions in GNNs.\n  - **[Setup]** We use 2 variants of flow surrogate where the sum aggregation is replaced with other common aggregations in GNNs: mean and max. Other settings are the same as in Sec. 4.2.\n  - **[Result]** As shown in Table 4, our method that uses sum aggregation outperforms its alternatives.\n\n**Table 4.** Comparison with other aggregations.\n|Metric: Kendal Tau|Sum (ours)|Mean (variant 1)|Max (variant 2)|\n|---|---|---|---|\n|NB-101|**74.8 (4.8)**|71.6 (5.8)|71.1 (6.4)|\n|NB-201|**82.2 (0.7)**|81.7 (1.0)|81.5 (0.9)|\n\n---\n# References\n\n- [1] FlowerFormer: Empowering Neural Architecture Encoding using a Flow-aware Graph Transformer\n- [2] TA-GATES: An Encoding Scheme for Neural Network Architectures\n- [3] Deep Image Prior"}, {"review_id": "urCFdjmcvC", "reviewer": "Reviewer_zBpp", "summary": "The authors propose FGP, a pretraining strategy to train neural architecture encoders to perform NAS. The proposed strategy allows capturing flow based information to predict neural network performance while being computationally cheaper. The authors use a proxy flow based objective to achieve this and show improved downstream performance with their method on predicting neural network performance as well as finding better neural architectures.", "questions": "See above", "limitations": "The flow training objective seems to be limited to GNN based encoders, which can be computationally expensive. Perhaps as a more general direction, is there an extension to transformer based encoders which could be more efficient?", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. The proposed FGP method is able to predict neural network performance via a flow based objective which predicts NN performance by mimicking the forward and backward passes via a graph based step.\n2. The experiments are well structured and highlight the effectiveness of FGP in predicting neural network performance as well as finding better architectures. \n\nWeaknesses\n1. While a ResGatedGCN with FGP outperforms Flowerformer in Fig 8a, training with FGP increases the encoding time of the ResGatedGCN significantly (from $10^2$ to $10^3$) presumably due to the flow based objective, which suggests that the Flow based objective still introduces significant pretraining overhead compared to other non flow based methods. However in Fig 8b, FGP is faster than other pretraining methods. It is unclear which pretraining method has been compared for the ResGatedGCN in Fig 8a and how that is different from the baselines in Fig 8b. Overall, the time taken for pretraining with FGP seems to significantly increase over other pretraining methods. It would be nice to have a Pareto curve for performance vs time taken for each pretraining method to highlight the efficiency of FGP.\n2. While FGP is claimed to capture the flow based information implicitly, it is unclear how combining FGP with FlowerFormer is able to increase performance. It would suggest that the FGP objective adds information other than the flow to training, can the authors ablate this effect?\n\nThe empirical evaluations of the proposed method shows that the flow based objective can improve encoding performance, but it seems to come at additional cost. The exact tradeoffs between performance and efficiency, specifically due to the flow based objective can be made clearer by the authors.", "was_revised": true, "final_justification": "The authors provide additional experiments distinguishing the effects FGP pre-training on FlowerFormer, which was one of my main concerns, They have also explained their computation time analysis in their response.", "rebuttal": "Dear Reviewer zBpp\n\nWe deeply appreciate your invaluable feedback on the further investigations of computation speed and backbone neural architecture encoders.\n\nBelow, we provide detailed responses to each of your comments.\n\nBest regards,\n\nThe Authors.\n\n---\n\n# Weakness 1.1\n\n`While a ResGatedGCN with FGP outperforms Flowerformer in Fig 8a, training with FGP increases the encoding time of the ResGatedGCN significantly (from to) presumably due to the flow based objective, which suggests that the Flow based objective still introduces significant pretraining overhead compared to other non flow based methods.` \n\n`However in Fig 8b, FGP is faster than other pretraining methods. It is unclear which pretraining method has been compared for the ResGatedGCN in Fig 8a and how that is different from the baselines in Fig 8b.`\n\n\n# Response 1.1\n\nWe apologize for the confusion. Our response is three-fold, and we will revise the manuscript by adding these details.\n\n- **[Clarification on Figure 8a]** The X-axis values in Figure 8a represent the total time required for both architecture processing (encoding) and the pre-training phase using our method (FGP). Therefore, (1) the additional time observed should not be interpreted solely as increased architecture processing time, and (2) other pre-training methods are not used in Figure 8a.\n    - **[Detail 1]** For ‘ResGatedGCN (non-flow-based) w/o FGP’ and ‘FlowerFormer (flow-based) w/o FGP’, the X-axis values reflect only the architecture processing time for ResGatedGCN [1] and FlowerFormer [2], respectively.\n    - **[Detail 2]** In contrast, for ‘ResGatedGCN (non-flow-based) w/ FGP’ and ‘FlowerFormer (flow-based) w/ FGP’, the X-axis values include both the pre-training time and the architecture processing time. Thus, the time gap between the w/ FGP and w/o FGP settings reflects the cost of pre-training, not additional architecture processing time.\n- **[Clarification on Figure 8b]** Figure 8b presents the pre-training time required for the ResGatedGCN architecture under each pre-training method. It shows that FGP achieves a pre-training time comparable to that of the fastest baseline method.\n    - **[Detail]** Compared to the fastest pre-training method (ZC-Proxy [3]), our method (FGP) requires only 10 additional seconds to train 14K architectures over 200 epochs—a relatively minor overhead in practice. Despite this marginal overhead, FGP outperforms ZC-Proxy in 26 out of 27 settings.\n    - **[Note]** The sum of the pre-training time for FGP in Figure 8b and the architecture processing time for ResGatedGCN w/o FGP in Figure 8a is equivalent to the time for ResGatedGCN w/ FGP in Figure 8a.\n- **[Speed clarification]** Pre-training with FGP does not introduce additional overhead in architecture processing. Moreover, while it requires some extra pre-training time, this cost amounts to only a few minutes and results in substantial performance gains, as detailed in our response to Weakness 1.2.\n\n---\n\n# Weakness 1.2\n\n`Overall, the time taken for pretraining with FGP seems to significantly increase over other pretraining methods. It would be nice to have a Pareto curve for performance vs time taken for each pretraining method to highlight the efficiency of FGP.`\n\n`The empirical evaluations of the proposed method shows that the flow based objective can improve encoding performance, but it seems to come at additional cost. The exact tradeoffs between performance and efficiency, specifically due to the flow based objective can be made clearer by the authors.`\n\n# Response 1.2\n\nOur response is three-fold.\n\n- **[Comparison with baselines]** Our method (FGP) achieves a pre-training time comparable to that of the fastest baseline method.\n    - **[Detail]** Compared to the fastest pre-training method (ZC-Proxy [3]), our method (FGP) requires only 10 additional seconds to train 14K architectures over 200 epochs—a relatively minor overhead in practice. Despite this marginal overhead, FGP outperforms ZC-Proxy in 26 out of 27 settings.\n- **[Experiments]** As detailed in Appendix B.5 of our manuscript, our proposed method (FGP) outperforms the strongest baseline methods (ZC-Proxy and GMAE [4]) under a fixed pre-training time constraint.\n    - **[Setting]** We fix the pre-training time for each method and use ResGatedGCN [3] as the backbone neural architecture. All other experimental settings follow those described in Section 4.2.\n    - **[Results]** As shown in Tables 1 and 2, FGP consistently outperforms all baseline methods across all the settings.\n- **[Implication]** These results indicate that substantial performance gains can be achieved with just a few minutes of pre-training.\n\n**Table 1.** Performance prediction results in the NAS-Bench-101 dataset under diverse fixed pre-training time.\n| Pre-training time (secs.) | 0 | 100 | 200 | 300 | 400 | 500 |\n| --- | --- | --- | --- | --- | --- | --- |\n| GMAE | 65.0 (7.8) | 65.7 (6.1) | 66.6 (6.8) | 67.2 (7.4) | 67.8 (4.6) | 68.0 (4.9) |\n| ZC-Proxy | 65.0 (7.8) | 67.9 (5.3) | 68.8 (5.3) | 69.2 (4.7) | 69.0 (4.4) | 68.9 (4.8) |\n| FGP (Ours) | 65.0 (7.8) | **71.8 (4.8)** | **73.2 (3.9)** | **74.0 (4.0)** | **74.7 (4.4)** | **74.8 (4.3)** |\n\n**Table 2.** Performance prediction results in the NAS-Bench-201 dataset under diverse fixed pre-training time.\n| Pre-training time (secs.) | 0 | 100 | 200 | 300 | 400 | 500 |\n| --- | --- | --- | --- | --- | --- | --- |\n| GMAE | 73.4 (1.5) | 74.0 (1.6) | 74.4 (1.6) | 74.8 (1.2) | 74.6 (1.5) | 74.8 (1.9) |\n| ZC-Proxy | 73.4 (1.5) | 76.3 (1.7) | 79.5 (1.2) | 79.3 (1.4) | 79.6 (1.2) | 79.8 (1.3) |\n| FGP (Ours) | 73.4 (1.5) | **79.8 (1.1)** | **81.4 (1.1)** | **82.0 (1.0)** | **82.2 (0.4)** | **82.2 (0.7)** |\n\n---\n\n# Weakness 2\n\n`While FGP is claimed to capture the flow based information implicitly, it is unclear how combining FGP with FlowerFormer is able to increase performance. It would suggest that the FGP objective adds information other than the flow to training, can the authors ablate this effect?`\n\n# Response 2\n\nOur response is two-fold.\n\n- **[Hypothesis]** We hypothesize that FGP enables FlowerFormer to learn a broader range of information flow beyond what is present in labeled architectures by leveraging a large number of unlabeled architectures, thereby enhancing its ability to capture more diverse flow patterns.\n    - **[Detail]** When trained solely on labeled samples, FlowerFormer tends to learn information flow patterns limited to that specific set. In contrast, FGP leverages a much larger pool of neural architectures whose ground-truth performance is unknown, exposing FlowerFormer to a wider range of information flow patterns. This exposure enables FlowerFormer to learn more diverse and generalizable information flow representations.\n- **[New experiments: Empirical evidence]** Our hypothesis is empirically supported by our new experiments, which reveal that the performance of FlowerFormer tends to be positively correlated with the size of the pre-training dataset.\n    - **[Intuition]** By varying the number of neural architectures used during FGP pre-training, we aim—though not precisely—to control the diversity of information flow patterns to which FlowerFormer is exposed. A positive correlation between dataset size and FlowerFormer performance suggests that FGP helps FlowerFormer encounter a broader range of information flow patterns, leading to improved accuracy in performance prediction.\n    - **[Setting]** We vary the proportion of the pre-training dataset used for FGP training—0% (no pre-training), 20%, 40%, 60%, 80%, and 100%—and evaluate FlowerFormer’s performance on the performance prediction task. Kendall’s Tau is used as the evaluation metric, and all other settings follow those described in Section 4.1 of our manuscript.\n    - **[Results]** As shown in Table 3, FlowerFormer’s performance tends to improve with larger pre-training dataset sizes, providing empirical support for our hypothesis.\n\n**Table 3.** Performance prediction results of FlowerFormer under varying sizes of pre-training dataset\n\n| % of the dataset used for pre-training. | 0% (No pre-training) | 20% | 40% | 60% | 80% | 100% |\n| --- | --- | --- | --- | --- | --- | --- |\n| NAS-Bench-101 | 74.0 (3.6) | 74.6 (3.8) | 75.9 (1.9) | 75.8 (4.0) | 76.2 (5.1) | 76.3 (3.6) |\n| NAS-Bench-201 | 77.3 (1.5) | 81.1 (0.8) | 82.3 (1.2) | 82.4 (1.3) | 82.7 (1.4) | 83.5 (1.7) |\n\n---\n\n# References\n\n- [1] Residual Gated Graph ConvNets\n- [2] FlowerFormer: Empowering Neural Architecture Encoding using a Flow-aware Graph Transformer\n- [3] Dynamic Ensemble of Low-Fidelity Experts: Mitigating NAS “Cold-Start”\n- [4] Graph Masked Autoencoder Enhanced Predictor for Neural Architecture Search"}]

Author rebuttal:

Meta-review: The initial reviews mentioned good structure, novelty, good argumentation and well-thought experiments as the main strengths of the paper. At the same time, the criticism was centred around the results being purely empirical, lack of explanations for certain statements and generalisability/robustness of the results.

During the rebuttal, the authors have provided additional explanations, including a preliminary analysis of certain (theoretical) properties of their method (in particular: invariance to node permutations and ability to distinguish architectures with distinct operations), as well as ablation studies and comparisons. In the end, all reviewers were convinced to recommend acceptance. Although confidence is low in certain cases, after reading the paper myself I concur that it constitutes an interesting contribution and would like to recommend acceptance.
Decision: Accept (poster)