ReG-NAS: Graph Neural Network Architecture Search using Regression Proxy Task
Authors: Boyi Wei
Cong Hao
Venue: Submitted to ICLR 2023 (2023)
Keywords: Neural Architecture Search
GNN
Machine Learning

Abstract: Neural Architecture Search (NAS) has become a focus that has been extensively researched in recent years. Innovative achievements are yielded from the area like convolutional neural networks (CNN), recurrent neural networks (RNN) and so on. However, research on NAS for graph neural networks (GNN) is still in a preliminary stage. Because of the special structure of graph data, some conclusions drew from CNN cannot be directly applied to GNN. At the same time, for NAS, the models' ranking stability is of great importance for it reflects the reliability of the NAS performance. Unfortunately, little research attention has been paid to it, making it a pitfall in the development of NAS research. In this paper, we proposed a novel NAS pipeline, ReG-NAS, which balances stability, reliability and time cost to search the best GNN architecture. Besides, for the first time, we systematically analyzed factors that will affect models' ranking stability in a given search space, which can be used as a guideline for subsequent studies. Our codes are available at https://anonymous.4open.science/r/ReG-NAS-4D21

Under review as a conference paper at ICLR 2023
REG-NAS: GRAPH NEURAL NETWORK ARCHITEC- TURE SEARCH USING REGRESSION PROXY TASK
Anonymous authors Paper under double-blind review
ABSTRACT
Neural Architecture Search (NAS) has become a focus that has been extensively researched in recent years. Innovative achievements are yielded from the area like convolutional neural networks (CNN), recurrent neural networks (RNN) and so on. However, research on NAS for graph neural networks (GNN) still in a pre- liminary stage. Because of the special structure of graph data, some conclusions drew from CNN cannot be directly applied to GNN. At the same time, for NAS, the models' ranking stability is of great importance for it reflects the reliability of the NAS performance. Unfortunately, little research attention has been paid to it, making it pitfall in the development of NAS research. In this paper, proposed a novel NAS pipeline, ReG-NAS, which balances stability, reliability and time cost search the best GNN architecture. Besides, for the first time, we systemat- ically analyzed factors that will affect models' ranking stability in a given search space, which can be used as guideline for subsequent studies. Our codes are available at htttps::/aanoymouus 4open. science/ /reerr//eee ReS/RRRG-S---DD
INTRODUCTION
Graph neural networks (GNN) have received a lot of attention for their broad applications in social networks (Guo & Wang 2020 Gao et 2021 Zhong al. 2020), molecule properties prediction
(Shui& & Karypis] 2020
Ma et al. 2020 Yang et al. 20221) traffic prediction (Dichl et al., 2019 Bui 2021 and so on. With the goal of "faster and more accurate" people always have a pursuit to find a better structure of GNN. However, similar to neural networks like CNN and RNN, searching an ideal GNN architecture manually also is challenging. Neural architecture search (NAS) for GNN is absolutely a key point to the future development of GNN.
ct al. 2021] Zhang et al.
To design a NAS architecture, an intuitive yet primitive idea is enumerate all models in given search space, and evaluate each model's 's performance according to the metric specified by the down- stream task (Ying et al., 2019 Dong & Yang 2019 You et al. 2020). However, is extremely time- consuming and needs huge amount of computational resources. To make NAS more efficient, several searching methods are proposed. Most GNN NAS architectures can be divided into five classes. (1) werrrforemmee--earnnnng----- 2020a), where these architectures have controllers defined as neural network that dynamically change the parameters according the evaluation of the performance of the generated model; (2) Bayesiannoppimizztion--ee
methods (Zhou et al. 2019 Gao et al.) 2020 Zhao al.,
methods (Yoon et al. 2020 Tu et al. 2019). which builds a probability distribution over sampled candidates and uses a surrogate function test; (3) Evolution-learning-
based methods (Shi et al.) 2022 Li & King 2020). among which the genetic algorithm is the most commonly used for GNN NAS frameworks Oloulade et al. 202T). (4) Differentiable-see weaelll--eachh-bere
methods Zhao et al. 2020b: Huan et al. [021 Ding et al. 2021 Li et al. 2021b; Cai et al. 2021), which learns one or two blocks that are repeated in the whole neural network, and for GNN block is generally represented as a direct acyclic graph consisting of an ordered sequence of nodes. (5) sandoom-search-e
methods (Gao et al. 2020 Zhao et al., 2020al Tu et al., 2019),, which gen- erates random submodels from the search space. However, these methods are still time consuming and can take hours to days.
To reduce the search time, a popular way in NAS is proxy-task, usually much smaller than the groundtruth task (e.g., 10 is a proxy for ImageNet). The representativeness of the proxy- task is crucial, i.e., how similar results can be obtained from proxy-task and from groundtruth task.

---

Under review as a conference paper at ICLR 2023
One prevailing method quantify the similarity between the two is using "ranking correlation", which refers to the similarity between two rankings of all the networks' performance in the search space, usually uses Spearman's 's p and Kendall's as indicators (Abdelfattah et al., 2021 Liu et al. 2020 Zela et al. 2019). The larger and T are, the more similar the two rankings are, meaning the more representative of the proxy task is. By achieving large p and between groundtruth ranking and prediction ranking, one can significantly improve NAS quality (Zhou et al.) 2020: Chu et al., 2021 Li & Talwalkar 2020). A lot of zero-cost few-shot NAS works have been proposed to design a good proxy, so that the ranking correlation between proxy and groundtruth can be high within only a few training steps (Mellor et al. 2021 Dey et al. 2021 Li al., 2021a).
In our work, we propose proxy-task based GNN architecture search, aiming to reduce the GNN architecture search time. Similar to previous works in DNN architecture search Zhou et al. (2020); Chu et al. (2021) [L & Talwalkar (2020); MMllor et al. (202T): ; Dey et al. ( et al. (2021a) we also use ranking correlation to evaluate the performance of our proposed proxy-based NAS. In addition, ranking correlation can be used to quantify the ranking stability of the networks by computing P and r between different repetitions of the same training pipeline, Large p and T imply that the variation of the model's s relative ranking is small, ie.. the ranking is stable. We clarify two metries that will be used hereafter:
Ranking Correlation: Correlation of two network rankings on two different tasks, e.g., ground truth task and proxy task, or proxy task one and proxy task two; quantified by p and 7.
Ranking Stability: Correlation of two rankings on the same task but of two repetitions, either the same or different initialization and training hyperparameters. Also quantified by p and
Observing the two metries, we found interesting phenomenon: in GNN search space, the rank- ing stability for classification lasks can be much lower than for regression tasks. For classification groundtruth tasks, the ranking correlation between two repetitions of all GNN architectures in the same search space can be as low as 0.57, while for regression tasks, the ranking correlation between two repetitions can be up to 0.99. Inspired by this observation, together with recent regression- based proxy task in CNN, GenNAS (Li et al. 2021c), we propose a self-supervised regression- based proxy task for GNN NAS. We observe that using our proposed regression based proxy task, both the ranking stability and ranking correlation are higher. In addition, regression- proxy task converges faster than classification groundtruth task, thus reducing the search time. However, generate a representative proxy task is non- non-trivial. There is rich study for proxy task generation or selection for DNN NAS but no related research on GNN NAS. The only regression- NAS work, GenNAS, can automatically search for a good proxy task, but requires knowing 20 chitectures with groundtruth ranking (Li et al.) 2021c). This is a strong premise and still can be time consuming when targeting a new search space or dataset, i.c., at least 20 architectures must be well-trained and then ranked.
DNN
To address the above challenges, we proposed novel NAS method, using Reression-based proxy task for Graph Neural Architecture Search, ReG-NAS. We summarize our contributions as follows:
ReG-NAS is the first GNN NAS using pivression-based proxy task. We propose a GNN NAS pipeline that can transform a groundtruth classification task to a regression proxy task, which leads to much higher ranking stability and faster convergence.
We systematically study the ranking stability and ranking correlation under various training en- vironments, and uncover the fact that directly searching on classification groundtruth task is un- reliable because of the low ranking stability. This observation challenges one common practice in NAS that, as long as the ranking correlation between proxy and groundtruth task is high, it is regarded as an effective proxy.
We propose simple yet effective proxy task to guide GNN NAS, which does not require groundtruth labels but only one well-trained GNN model as proxy task generator. The gener- ator is not necessarily the best GNN but can be any GNN within the search space. Using the proposed proxy task, we turn the groundtruth classification problem into regression, leading to much higher ranking stability and faster search.
2

---

Under review as a conference paper at ICLR 2023
2
RELATED WORK
2.1 GRAPH NEURAL NETWORKS
Graph a kind of data structure that defines a set of nodes and their relationships (Waikhom & Patgiri 202T). A graph is set of V nodes and set of E edges, with optional labels y or features x attached to nodes, links, or the whole graph. Therefore, a graph can be represented as G = (V, E). Graph Neural Networks (GNN) is a type of Deep Neural Networks (DNN) that is suitable for analyzing graph-structured data. If we use X, and Xue to represent node v's and link (u, v)'s feature vectors, use h, and hwr to represent node v's and link (u, v)'s hidden representations in GNN, the GNN's message passing and update process can be described as:
h = fnode he:
hav
(1)
uEN(w)
h/
fedge
(hw hy: Xav)
Where (v) denotes the number of in in-neighbor nodes node u, and fnode and fedge are message passing functions that gather information from node's 's neighborhood and previous layers.
GNN can be classified according to variants of graphs, downstream tasks, learning methods and so Main types of graphs includes undirected/direeted graph, heterogeneous graph, dynamic graph, attributed graph and so on. Downstream tasks usually include classification task and regression task, each of which can be subdivided into graph-level, link-level and node-level problems. In our work, we mainly focus on undirected graphs with graph-level tasks.
2.2 GRAPH NEURAL ARCHITECTURI SEARCH AND GRAPIGYM
Graph Neural Architecture Search (GNN NAS) means automatically find the best GNN models for targeted tasks (Oloulade et 202T). Like NAS for other neural networks, GNN NAS samples an architecture from a predefined search space, then the NAS network will evaluate the performance of sampled architecture as a feedback returned to the search algorithm.
GNN NAS can be categorized ac- cording to search space, search al- gorithm (see Section [1] and perfor- mance evaluation (Oloulade et al.]
Pre- message passing layer
Pre- process layers
202T]. Recent studies for GNN NAS
Pre layer
(Zhou et al. 2019 Gao et al. 2020
Linear
Tu et al.
2019:Shh et al. 2022
Huan
Batch Normalization
Message pessing layer
Layer
et al.]
2021
Li et al.) 2021b You
Dropout
Message passing layer
connectivity
al.
2020) have not only achieved
Message
promising performance for many ap- plications GNNs but also showed potential as a unanimous approach to constructing GNN models.
Activation
Message passing layer
passing layers
Aggragation
Post-message passing layer
Post: process layers
Post message passing layer
Among all GNN NAS frameworks, we want to introduce GraphGym Youu et al. 2020) in detail. In Graph- Gym, a GNN design space consists of 12 design dimensions for intra-layer design, inter-layer design and learn-
Training configuratins Batch size, Leaming rale, Optimizer Trinning epochs
Figure 1: Structure of and Design Space of GNN
ing configuration. A single GNN layer has a sequence of modules:(1) Linear layer ( (R) +b.).
(2) batch normalization BN(-) (Toffe & Szegedy| (015);(3) dropout operation DROPOUT()) Sri- vastava et al.) 2014);; (4) nonlinear activation function ACT(-); (5) aggregation function AGG(.). Therefore, the k-th GNN layer can defined as:
h. (k+1) AGG ((ACT (DROPOUT (BN (w (k) (ov())(++())).eeeee.. ,uEN(o}))
(3)
3

---

Under review as a conference paper at ICLR 2023
where he (1) is the k-th layer embeddings of node e, W (()) are trainable weights. For inter- layer design, Graphgym gives three ways to connect GNN layers: STACK (directly stack multiple GNN layers) (Welling & Kipf, 2016 Velickovic et al. 2017): SKIP-SUM (residual connections) (He et al. 2016) and SKIP-CAL (concatenate embeddings in all previous layers) (Huang et al.) 2017). GraphGimm also adds Multilayer Perceptron (MLP) layers before/after GNN message pass- ing. Training Configurations includes batch size, learning rate, optimizer type and training epochs. The overview of GraphGym'ss 's design space and structure is shown Fig In our works, we utilize GraphGym as a basic framework and propose our new NAS architecture by modifying it.
2.3 GENERIC NEURAL ARCHITECTURE SEARCH VIA REGRESSION (GENNAS)
Generic Neural Architecture Search (GenNAS) is the a CNN and RNN based NAS framework that uses self-supervised regression proxy task instead of classification for NAS (Li et al. 2021e). Com- pared to other NAS frameworks, it has several advantages: (I) By using regression as the self- supervised proxy task, it is downstream-agnossii to the specific downstream tasks. (2) has near- zero training cost, which is highly efficient for neural architecture search.
Here we mainly focus on CNN regression architectures of GenNAS, as shown in Fig 2(a) GenNAS constructs a Fully Convolutional Network (FCN) (Long et al. 2013) by removing the final classifier of a CNN, and then extract the FCN's intermediate feature maps from multiple stages. The number of stages is denoted N. If the input tensor is I, each stage's feature map tensor is F, the synthetic signal is Fi, the regression pipeline will reshape F; into F, (F.)., and compute MSE loss defined as = Σi= E F:) during training process. GenNAS will rank models' perfor- mance according final MSE values, and select the model with the lowest MSE value the best. GenNAS uses Ranking Correlation (See Section I for NAS evaluation.
3 PROPOSED REG-NAS
3.1 THE BARRIERS TO GENERATE PROXY TASK
As mentioned in Section I ReG-NAS uses a regression based proxy task to search for GNN struc- tures. However, different from grid-data like images, which can use the combination of signals with different frequency but the same shape (i.e., data dimension) as proxy task, generate proxy task for GNN is much harder.
First, even in the same dataset, different graphs usually have different topology structures. Therefore, for graph datasets, should generate proxy task for each graph individually, and there is no called "global" signal (Li al. 2021c) in the process of generating proxy task.
Second, although according spectral graph theory (Shuman et al. 2013), any graph signal can be projected on the eigenvectors of the Laplacian Matrix L and the "frequency" of each eigenvector the corresponding eigenvalue, we cannot simply use these vectors as our proxy task. For graphs, Laplacian Matrix only contains a graph's structural information, while many graphs also have other non-structural information such as node features and edge features. Direct linear combination of Laplacian Matrix's eigenvectors will lose graphs' original information and weaken NAS perfor- mance. Therefore, in our generating process, we should aggregate both structural and non-structural information into proxy task.
3.2 PROPOSED REG-NAS PIPELINE
In ReG-NAS, we propose a simple, yet effective proxy task generator for graph datasets, as shown in Fig 2(b) In a given search space with number of post-process layers equals to 2, we first randomly select GNN architecture, then use this architecture to train the dataset for A epochs. In the end we use this well-trained model for inference, and extract model's hidden node feature from the first Post-process layer as our proxy task F,, The reason why we set the number of Post-process layers into 2 is to make sure that all graphs' proxy tasks are in the same shape. For example, if the input graph G's G's original node feature is I R xdo (n is the number of nodes), and in the message passing layer i, node feature is F, E I x the first proceps layer will reshape F, into F, E Rlxd, and F, is the proxy task for graph. The first post-process layer uniformly converts node feature's
4

---

Under review as a conference paper at ICLR 2023
Classfication Task
Stage
Stagee 2
Stage
Model being evaluated CNN,
Bork
y:
Input:7 E R xxx3xxxw
Famking correlation anaiysii
T. fxxxk
i, F Bhxggshann P3 C kheg
Regression Tink
Spearman's ' Kendall's
Procy Task Search
Mi(FD
S M20%)
F Mc(Fs)
Sourch:
Subsot of NN
Tinxy
Groudhruh
architactures
Ranking
Ranking
Networki
Network2
Network3
( FD)
(f -1)
(f, PA)
Networks
Roxeixat
Tg Rix BOXGNEE
(a) Overview of GenNAS structure
Classification Task berng evaliaated GNN,
Pre MP
Message Passing Layers (MP) GR Roaa,
Post
MP
G GVV,
Graph
Deepatt
Autiunten
Arthato
Aggang
Doppott
Encoder
122
Teput Festture
Enseded Foctuni
Ranking
2
coreelation analysis
R Regression Task
Speremnn's & Kendall's
Praxy gesemtor GNN,
M(F
M(F2)
M(F))
$; E R
Rank
Model
Poat NBP
MP
(f. Fp)
(f, F,
(f, F,)
1
Ingut Node 5 geo0
Pruxy Tiss g20d,
data.nur nodesan
(b) Overview of ReG-NAS structure
Figure 2: Overview of GenNAS structure (a) and ReG-NAS structure (b). For ReG-NAS, this figure only shows the cases when groundtruth task is classification. fact, ReG-NAS is also applicable for regression-based groundtruth task, which is almost the same as classification-batt groundtruth task architecture searching pipeline.
shape into dp despite the variance of de and d,. which facilitates the proxy training process in the next.
The way we generate proxy task does not need know model's performance ranking, even subset of the search space. As the final goal of NAS is to rank GNN's relative performance, the generator's performance wouldn't 't affect the final result as long the generated proxy task is informative (See Section 4.1.2). In fact, from the experiments discussed later, we will find that the selection of model has little effect on final results. This is different from the way that GenNAS did (Li et al.] (221cc:: In GenNAS, before generating proxy task. need to know the relative performance ranking a subset (c.g 20) the neural architectures in NAS search space, which is strong assumption and thus is not practical when being applied to new datasets or tasks.
After the proxy task is generated, we will attach to the graph dataset. In the regression training process, ReG-NAS will extract node feature F; from each Message Passing layer, and reshape into F; RR 1xxp in order to match the shape of F,- We pooling method (SUM, MEAN, MAX)

---

Under review as a conference paper at ICLR 2023
to reshape F, The evaluation metric is the final MSE loss between F; and Fp- In the end will compute Ranking Coefficient (Spearman's 's p and Kendall's 7) between proxy (regression- ranking and groundtruth (clasificattion-aasee)
regression-based ranking.
In summary, the whole process of ReG-NAS contains several steps below:
Randomly select GNN model from the search space as proxy task generator GNNp;
2. Generate proxy task F E R' x
3. Select model be evaluated GNN, from the search space:
4. Extract Message Passing layer's node feature F; E Rnxdd reshape it into Fi E R 1x,, compute
MSE loss C = Σ E( - F;))):
5. Evaluate and rank model's performance according to final L value.
4 EXPERIMENTS
Table 1: Basic information and hyper-parameter configurations for ogbg-molhiv and ZINC
Dataset
ogbg-molhiv
ZINC
# Graphs
41,127
249,456 (12,010) for subset)
Nodes per graph
25.5
23.15
Edges per graph
24.9
Task type
Binary-classificction
Regression
Metric
ROC-AUC
MAE
Pipcline
Groundtruth (classification)
Proxy (Regression)
Groundtruth (Regression)
Proxy (Regression)
Base learning rate
0.0007
0.075
0.0006
0.0009
Batch size
128
128
Drupout
(False)
(False)
(Tulse)
(False)
Loss function
Cross entropy
MSE
MSE
MSE
Optimizer
ADAM
SGD
ADAM
ADAM
Node encoder
Atom
Atom Bond
Atom
Atom
Edge encoder
Bond
None
None
# Post-mp layers
2
2
2
Train Epochs
100
80
100
80
To fully evaluate the performance of ReG-NAS well analyze the factors that will affect GNN ranking stability, we conduct 3 types of experiments: Ranking stability analysis, Effectiveness eval- uation and Efficiency evaluation. In these experiments we use ogbg-molln molhiv (Hu al.) (020) as classification based dataset, and use ZINC (Gómez- Bombarelli et al. 2018) regression-basee dataset. The basic information of these datasets and seper-parameter confi configurations are listed in
Table 1 Our search space contains 216 GNN models, as shown in Table 2
All hyper- seper-parameters (base learning rate, optimizer, batch size etc.) are op- timized to ensure that the model con-
Table 2: GNN Search Space for experiments Variable
verges at optimal rate. In our ex- periment, the learning rate is annealed via cosine decay to 0 in order to reduce the variance between multiple indepen- dent training runs (Loshchilov & Hutter 2016). And for ZINC, we use its sub- which contains 12,000 graphs with 10,000 train graphs. 1,000 test graphs and 1,000 validation graphs in our experiment to reduce train- ing cost. The reason why we set proxy training epochs equals is that we find that all models' loss converges at about 80 epochs, thus there's no need to train extra 20 epochs.
Range
# Message Passing layers
2,
Stage
STACK, SKIP-SUM, SKIP-CAT
Inner Layer Dimension
32, 64, 128, 256 ReLU, SWISH. PReLU MHAN, MAX, SUM
Activation
Aggregation
4.1 GNN RANKING STABILITY ANALYSIS
In this section we aim to find factors that will affect GNN ranking stability and try to evaluate the stability of our proposed NAS pipeline. Therefore, we first test GNN ranking stability on two groundtruth task (classification task on ogbg-molhive regression task on 2INC), then we test ranking stability on our NAS pipeline. At the same time, will also discuss how proxy task will

---

Under review as a conference paper at ICLR 2023
affect the proxy ranking stability. For single experiment, repeat the training and evaluation of all architectures 3 times, then compute Ranking Stability among them. For different repetitions of the same task, their intiialization is different.
Sposmmnn's 0,5733, Kendall's 0,4014
Spoarmen's p 0.9142, Kondall's 0.5427
Spearmen's KLnslllKmdlll'' Keniill's 0.939
50
Catundrtth forking (epoot))
200.
2004
200-
180-
100-
Mo-
140-
140-
170-
100-
300
Ger.nerth
60.
(00
40-
10
30
20
100
00
to 100
Grounitrth DeniingrRepppr
Geourdouth Panking (Repostt
Prixy Fariong (Repest
(b)
(c)
Sprarman's 0,9962, Kondail's 7 0,9500
Speacman's 's p 0,8071, Kendall's - 0.6250
Spearman's 't 0-120, Remdall's r 0.3990
pianyy Ranking (162983
1000-
11
1604
360
-32
100
180-
30
100-
Groundrahh Tembong
340-
120-
844
1204
DoN Barkin
120
300
101-
20-
20 ao do ao tio 10020000 Prony Ronking (Rrgpott))
20 di 6o oo 300 a80 150200 Grouddouth Ranking (Poprat
40 to 100 120 160 Prony (Pepoot))))
(d)
(c)
(f)
Figure 3: Groundtruth task's and Proxy task's Ranking Stability analysis. (a) Classification-
based groundtruth task's Ranking Stability ogbg-molhiv (Use ROC-AUC ranking metric); (b) Regression- based task's Ranking Stability on ZINC (Use MAE as ranking metric); (c) Proxy task's Ranking Stability on ogbg-molhiv (Use MSE as ranking metric); (d) Proxy task's Ranking Stability on ZINC (Use MSE as ranking metric); (e) Classification- based groundtruth task's Rank- ing Stability on ogbg-molhiv (Use Loss as ranking metric); (f) Proxy task's Ranking Stability on ogbg-molhiv (Use random-
random-generated generated vectors as proxy task)
The Ranking Stability analysis between two repetitions of the same experiment are shown in Fig 3 The r-axis represents the ranking of ith experiment, and the y-axis represents the ranking of jth experiment. For example, if model ranks 3rd in the ith experiment and ranks 5th in the jth experiment, then the model's coordinate (3, 5). We make Ranking Stability analysis for every 2 experiments and place all points the same figure, represented
heat map. The heat map will show the density of the points and therefore reveals the pattern of task's 's Ranking Stability for a training pipeline with given configuration. We also compute p and from the heat map.
4.1.1 GROUNDTRUTH RANKING AND PROXY RANKING STABILITY ANALYSIS
In the situation when we don't modify ranking metries on groundtruth ranking and proxy ranking. and use proposed method to generate proxy task, as shown in Fig 3(a) 3(d) from the result we can clearly see that regression-bee using groundtruth training pipeline or proxy training pipeline. This phenomenon may due to the choice of downstream task as well as ranking metric. For classification task, in the training process, the variable being directly optimized is loss value, while the final ranking metric is ROC-AUC (for sgbggmolhhvye At the same time, for regression task, in the training process the variable being directly optimized is still loss value, but the final ranking metric is MAE (for INC). An intuitive explanation that, network performance for regression based task is evaluated directly on the regression loss; the network performance for classification based task, on the other hand, is evaluated on ROC-AUC, which is an indirect metric. To further validate our hypothesis, based on ogbg-mo 1 há v, we rank models according their final loss value, and analyze Ranking Stability,
ranking are more stable than classification-bue based ranking. despite
7

---

Under review as a conference paper at ICLR 2023
shown in Fig 3(e) From the result find that compared to "ROC-AUC-basee ranking, flos-bbsee ranking $ p and T are much better, which affirms our hypothesis.
4.1.2 THE RELATIONSHIP BETWEEN THE CHOICE OF PROXY TASK AND PROXY TASK'S RANKING STABILITY
Although all the experiments mentioned above illustrate regression- rankings are more stable, this may not be applicable to all cases. For example, if we use randomly generated vectors as our proxy task and use it to rank models, we will find it's is only 0.424, even it is a regression-base ranking, as Fig (() shows. From the results we can find many points fall in the upper left and lower right corners of the figure, which means that the relative rankings of these models differ lot in the two repeated experiments. This is an anomaly even for classification cases.
To explain this phenomenon, we should know the way how the Deep Learning optimization works. Deep Learning optimization is trying to globally optimize a function by using local gradient infor- mation (Bottou & Bousquet 2007). which means if a learning problem is characterized by non- informative gradients, then no deep learning architecture will be able to learn it. Back to the topic, clearly manoom-generater vectors cannot provide informative gradients, due which it is not sur- prising that a model's s ranking changes drastically in different repetitions. Therefore, before we draw conclusion that regression-based rankings are more stable, it should based premise that the learning problem should be informative or reasonable.
4.2 REG-NAS PERFORMANCE EVALUATION
4.2.1
EFFECTIVENESS OF REG-NAS
We compute Ranking Correlation between groundtruth rankings and proxy rankings to show the effectiveness of ReG-NAS. To shed light on how the variance of groundtruth tasks and proxy tasks affect the effectiveness of ReG-NAS, we conduct 8 types of experiments. For
MP
ogbg-me mo hiv, we conduct 5 types of experiments, the
Others
P, E IR 1xdd
F1 E Rixd1
LE
difference among them is the way how proxy task is gen- crated: (1) Use Random-selected Model as proxy task generator (RM); (2) Use model with the best groundtruth performance in the search space ("Golden" Model) as proxy task generator (GM); (3) Use model with the worst groundtruth performance in the search space ("Poorest" Model) as proxy task generator (PM); (4) Use Random- generated Vectors as proxy task (RV); (5) Use Laplacian
F E R
MP
Others LE
P2 E R 1xdp
F2 E Raxd,
F E R nx
MP
Matrix's eigenvectors as proxy task (LE). For LE-based pipeline, as the shape of Laplacian Matrix's 's eigenvectors (n 1) not equal to 1 x dp. so the reshape process is different from other proxy training pipeline, as Fig 4 shows. For ZINC, we conduct types of experiments: (1) RM-based pipeline; (2) GM-based pipeline; (3) PM-based pipeline. For each groundtruth task and proxy task, we repeat the exper- iment three times, and compute the average value of Spearman's 's p and Kendall's т among them as final results, as shown in Table 3
Comparison between LE pooling and other pooling process
Figure 4:
Table 3: Proxy-Groundtruth Ranking Correlation analysis
Datasets
ogbg-molhiv Classification
ZINC Regression
Groundtruth task type
Proxy task
RM
GM
PM
RV
LE
RM
GM
PM
Proxy task's shape
1 x dp
1 x dp
1 x dp
1 x dp
n X 1
1 x dp 0.421 0.303
1 x dp 0.435 0.314
1 x dp
Spearman's
0.362
0.367
0.356
-0.064
-0.038
0.443
Kendall's
0.260
0.259
0.227
-0.041
-0.026
0.324
From the results we can draw three conclusions: (1) Random-generatee vectors and Laplacian Matrix's eigenvectors not suitable for proxy task. As mentioned before, for proxy training
8

---

Under review as a conference paper at ICLR 2023
pipeline, the learning problem should be informative and reasonable (Section 4.1.2 Therefore, cundom-generatee vectors cannot used as proxy task. Meanwhile, from Section 3.1 we know that Laplacian Matrix's 's cigenvectors doesn't 't contain graph's 's non-structural information, which is pipeline reached ideal Spearman's p and Kendall's r, which can be used to approximate the performance of GNN structure. (3) The choice of models as proxy task generator has little effects on the final results, regardless the type of groundtruth task. For ogbg mo lhiv, the difference between the of GM-based pipeline and PM-based pipeline is 0.011, which is in the fluctuation range of (See Section Se.tion[4.2.2): For 2 INC, the IM-based pipeline's p even higher than GM- GM-based pipeline's s, which further proves that we don't have to select "Golden" model as task generator.
the key reason to the poor performance of LE pipeline. (2) RM/GM/PM-
4.2.2 EFFICIENCY OF REG-NAS
oggg ongg LAwenge (Avased Bengy
Dataset
ogbg-molhiv
ZINC
Ground truth
Ground truth
Task Type
Proxy
Proxy
Spamertss
Required
0.3
Training
100
20
100
40
epochs
0.2
Average
0.1
Single-
epoch
20.52
24.43
2.03
2.22
training
Speedup
4.2x
2.3x
Epoch
(a)
(b)
Figure 5: ReG-NAS convergence speed analysis. (a) Spearman's convergence curve, is the Ranking Correlation between proxy ranking at epoch n and final groundtruth ranking at epoch 100.
(b) ReG-NAS speedup compared to Groundtruth training time.
Here we compute Spearman's 's p between proxy ranking at epoch n and final groundtruth ranking at epoch 100 (constant), and draw the relationship between p and epoch. Since we repeat each periment times, we use Filled Area Graph to show the fluctuation range of rho at each epoch, as shown in Fig S(a) According to the figure
find that compared to 2INC, oglog cooog--moovv'ss fluctuates more drastically, which possibly due to the instability of groundtruth ranking. Mean- while, for ogbg-mo molii, converges at about 20 epochs, and for ZINC, p converges at about 40 epochs. Fig ((b) illustrates the speedup of ReG-NAS during the search of best GNN architecture by testing the average single-epoch training time of groundtruth pipeline and proxy pipeline. For ogbg-molhiv, ReG-NAS can save up to 76.2% training time; For 2 INC, ReG-NAS can save 56.5% training time.
To sum up. ReG-NAS is downstream-agee agnossic (applicable on both classification task and regression task), stable, effective and efficient GNN NAS architecture. By using ReG-NAS, we can approxi- mate GNN's relative performance in short time, which is especially useful for searching GNN in large search space.
CONCLUSION
In this work, we proposed ReG-NAS, a GNN NAS architecture which uses regression proxy task. has several advantages: (1) Stable. The Ranking Stability (Spearman's 's p) between two repetitions can reach up to 0.99; (2) Downstream-agnostic.. regression task; (3) Effective. It has high mrxxygroundtrrute ranking similarity which can be work as a reference of GNN's relative performance; (4) Efficient. Compared to traditional NAS searching method, it
applicable to both classification task and
up to 76.2% of training time. At the same time, for the first time, analyzed the factors that will affect GNN ranking stability, which provides a new insight designing stable GNN.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Mohamed S Abdelfattah, Abhinav Mehrotra, Lukasz Dudziak, and Nicholas D Lane. Zero-cost proxies for lightweight nas. arXiv preprint arXiv:2101.081 334, 2021.
Léon Bottou and Olivier Bousquet. The tradeoffs of large scale learning. Advances in neural information processing systems, 20, 2007.
Khac-Hoai Nam Bui, Jiho Cho, and Hongsuk Yi. Spatial-temporal graph neural network for traffic forecasting: An overview and open research issues. Applied Intelligence, pp. 1-12, 2021.
Shaofei Cai, Liang Li, Jinean Deng, Beichen Zhang, Zheng-Jun Zha, Li Su, and Qingming Huang. Rethinking graph neural architecture search from message-passing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6657-6666, 2021.
Xiangxiang Chu, Bo Zhang, and Ruijun Xu. Fairnas: Rethinking evaluation faimess of weight sharing neural architecture search. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12239-12248, 2021.
Debadeepta Dey, Shital Shah, and Sebastien Bubeck. Fear: A simple lightweight method rank architectures. arXiv preprint arXiv:2106.04010 2021.
Frederik Diehl, Thomas Brunner, Michael Truong Le, and Alois Knoll. Graph neural networks for modelling traffic participant interaction. In 2019 IEEE Intelligent Vehicles Symposium (/V), pp- 695-701. IEBE, 2019.
Yuhui Ding, Quanming Yao, Huan Zhao, and Tong Zhang. Diffig: Differentiable meta graph search for heterogeneous graph neural networks. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 279-288, 2021.
Xuanyi Dong and Yi Yang. Nas-bench-201. Extending the scope of reproducible neural architecture search. In International Conference on Learning Representations, 2019.
Jun Gao, Jiazun Chen, Zhao Li, and Ji Zhang. Ics-gnn: lightweight interactive community search via graph neural network. Proceedings of the VLDB Endowment, 1466:1006-1018, 2021.
Yang Gao, Hong Yang, Peng Zhang, Chuan Zhou, and Yue Hu. Graph neural architecture search. In Proceedings of the Twenty-Ninth International Joint Conference Artificial Intelligence, IJCAI- 20, pp. 1403-1409, 7 2020.
Rafael
Jennifer N Wei, David Duvenaud, José Miguel Hernández-Lobato, Benjamín Sánchez- Lengeling, Dennis Sheberla, Jorge gguilera-Iparragung Timothy Hirzel, Ryan P Adams, and Alán Aspuru-Guzik. Automatic chemical design using data-driven contin- uous representation of molecules. ACS central science, 4(2):268-276, 2018.
Zhiwei Guo and Heng Wang. A deep graph neural network-based mechanism for social recommen- dations. IEEE Transactions on Industrial Informatics, 17(4):2776-278) 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp- 770-778, 2016.
Weihua Hu, Mauthis Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems, 33:22115 21-21113,, 2020.
ZHAO Huan, YAO Quanming, and TU Weiwei. Search to aggregate neighborhood for graph neural network. In 2021 IEEE 37th International Conference on Data Engineering (ICDE), pp. 552-563.
IEEE, 2021.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Weinberger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
10

---

Under review as a conference paper at ICLR 2023
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 1448-466 PMLR. 2015.
Guihong Li, Sumit K Mandal, Umit Y Ogras, and Radu Marculescu. Flash: ast neura I a rchitec- ture earch with h ardware optimization. ACM Transactions Embedded Computing Systems (TECS), 20(5s): 2021a.
Liam Li and Ameet Talwalkar. Random search and reproducibility for neural architecture search. In Uncertainty in artificial intelligence, pp- 367-377. PMLR, 2020.
Yanxi Li, Zean Wen, Yunhe Wang, and Chang Xu. One-shot graph neural architecture search with dynamic search space. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 35, pp. 8510-8517, 2021b.
Yaoman Li and Irwin King. Autograph: Automated graph neural network. In International Confer- on Neural Information Processing, pp. 189-201. Spriner, 2020.
Yuhong Li, Cong Hao, Pan Li, Jinjun Xiong, and Deming Chen. Generic neural architecture search via regression. Advances in Neural Information Processing Systems, 34:20476-20490 2021c.
Chenxi Liu, Piotr Dollár, Kaiming He, Ross Girshick, Alan Yuille, and Saining Xie. Are labels necessary for neural architecture search? In European Conference on Computer Vision, pp. 798- 813. Springer, 2020.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3431-3440, 2015.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint 1608 03983, 2016.
Hehwan Ma, Yatao Bian, Yu Rong, Wenbing Huang, Tingyang Xu, Weiyang Xie, Geyan Ye, and Junzhou Huang. Multi-view graph neural networks for molecular property prediction. arXiv preprint a00iv: 13607, 2020.
Joe Mellor, Jack Turner, Amos Storkey, and Elliot Crowley. Neural architecture search without training. In International Conference on Machine Learning, pp. 7588-7598. PMLR, 2021.
Babatounde Moctard Oloulade, Jianliang Gao, Jiamin Chen, Tengfei Lyu, and Raeed AI-Sabri. Graph neural architecture search: A survey. Tsinghua Science and Technology, 27(4):692-708, 2021.
Min Shi, Yufei Tang, Xingquan Zhu, Yu Huang, David Wilson, Yuan Zhuang, and Jianxun Liu. Genetic-gnn: Evolutionary architecture search for graph neural networks. Knowledge-Baud Sys- tems, 247:108752, 2022.
Zeren Shui and George Karypis. Heterogeneous molecular graph neural networks for predicting molecule properties. In 2020 IEEE International Conference on Data Mining (ICDM), pp. 492- 500. IEEE, 2020.
David Shuman, Sunil K Narang, Pascal Frossard, Antonio Ortega, and Pierre Vandergheynst. The emerging field of signal processing on graphs: Extending high-dimensional data analysis to net- works and other irregular domains. IEEE signal processing magazine, 30(3):83-0)) 98, 2013.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-1958 2014.
Ke Tu, Jianxin Ma, Peng Cui, Jian Pei, and Wenwu Zhu. Autone: Hyperparameter optimization for massive network embedding. In Proceedings of the 25th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pp. 216-225, 2019.

---

Under review as a conference paper at ICLR 2023
Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. stat, 1050:20, 2017.
Lilapati Waikhom and Ripon Patgiri. Graph neural networks: Methods, applications, and opportu-
nities, arXiv preprint arXiv:2108 10733, 2021.
Max Welling and Thomas Kipf. Semi-supervised classification with graph convolutional net- works. In J. International Conference on Learning Representations (ICLR 2017), 2016.
Qiong Yang, Hongchao Ji, Hongmei Lu, and Zhimin Zhang. Prediction of liquid chromatographic retention time with graph neural networks to assist small molecule identification. Analytical Chemistry, 93(4):2200 22020000022222222222022222200200 2021.
Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter. Nas- 10nch-101: Towards reproducible neural architecture search. In International Conference on Machine Learning. pp. 7105-7114. PMLR, 2019.
Minji Yoon, Théophile Gervet, Bryan Hooi, and Christos Faloutsos. Autonomous graph mining algorithm search with best speed/accuracy trade-off. In 2020 IEEE International Conference on Data Mining (ICDM), pp. 751-760. IEFE, 2020.
Jiaxuan You, Zhitao Ying, and Jure Leskovec. Design space for graph neural networks. Advances in Neural Information Processing Systems, 15:17009-170211. 2020.
Arber Zela, Thomas Elsken, Tonmoy Saikia, Yassine Marrakchi. Thomas Brox, and Frank Hutter. Understanding and robustifying differentiable architecture search. arXiv preprint arXiv: 09056, 2019.
Chenhan Zhang, Shuyu Zhang, JQ James, and Shui Yu. Fastgnn: A topological information pro- tected federated learning approach for traffic speed forecasting. IEEE Transactions Industrial Informatics, 17(12)84644-8474 2021.
Huan Zhao, Lanning Wei, and Quanming Yao. Simplifying architecture search for graph neural network. CoRR, abs/2008. 1652, 2020a.
Yiren Zhao, Duo Wang, Xitong Gao, Robert Mullins, Pietro Lio, and Mateja Jamnik. Probabilistic dual network architecture search on graphs. arXiv preprint arXiv:2003.096 2020b.
Ting Zhong, Tianliang Wang, Jiahao Wang, Jin Wu, and Fan Zhou. Multiple-aspect attentional graph neural networks for online social network user localization. IEEE Access, 8:95223-95234, 2020.
Dongzhan Zhou, Xinchi Zhou, Wenwei Zhang, Chen Change Loy, Shuai Yi, Xuesen Zhang, and Wanli Ouyang. Econas: Finding proxies for economical neural architecture search. Proceed- ings of the IEEF/CVF Conference computer vision and patlern recognition, pp. 11396-11404, 2020.
Kaixiong Zhou, Qingquan Song, Xiao Huang, and Xia Hu. Auto-gnn: gnn: Neural architecture search of graph neural networks. arXiv preprint arXiv: 1909.03/84 2019.
12


Reviews:
[{"review_id": "mJ80a-VM5Yk", "reviewer": "Reviewer_bV9M", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new graph neural architecture search pipeline named Reg-NAS. Specifically, the authors find that regression tasks are more reliable than classification tasks in estimating architecture performance and advocate that a regression self-supervised proxy task be used to estimate the architecture performance.", "summary_of_the_review": "See above", "strength_and_weaknesses": "Pros:  \n1. The study about the stability of using classification and regression tasks to estimate GNN architecture performance is interesting and useful, which can inspire future work in graph NAS.  \n2. The paper is clearly written and easy to follow.  \n  \nCons:   \nThough I find the paper studies an interesting perspective of graph NAS, the current experiments are highly insufficient with the following concerns:   \n1. I’m concerned that only using SSL tasks to estimate the architecture performance may not work well for GNNs. Specifically, it is well known that the performance of architectures heavily relies on the downstream task, e.g., see [1] for an empirical study and [2] for a theoretical justification. For example, what if the SSL task is fundamentally different from the downstream tasks?  \n[1] Design space for graph neural networks, NeurIPS 2020.   \n[2] How neural networks extrapolate: from feedforward to graph neural networks, ICLR 2021.  \n  \n2. The experiments in Section 4 are not convincing enough. The authors only use two molecule datasets, neglecting that graph datasets cover a wide range of domains, e.g., social networks, citation networks, financial networks, etc. Different data sources can cause the graph to behave fundamentally differently. It is doubtful that the same conclusion holds in other areas.  \n\n3. The Spearman correlation coefficient and Kendall Tau shown in Table 3 are not very high, so I wonder whether they can support a good NAS method. Moreover, the authors do not show the quantitative results of the pipeline for the final downstream task, i.e., the real effectiveness of the proposed method is not validated. Besides, I would suggest the authors additionally show the correlation of the top-performing architectures instead of all architectures in the search space, considering that the ultimate goal of NAS is to find good architectures rather than evaluation all possible architectures.  \n\n4. The statement in Section 4.1.2 regarding using random labels seems to be in contradiction with NAS results in other domains, e.g., in computer vision [3]. The authors may need to explore deeper into why random labels do not work for graph tasks.  \n[3] Neural architecture search with random labels, CVPR 2021.   \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "j6UZFct_8JY", "reviewer": "Reviewer_i2b5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper first proposes a proxy regression task, which is called REG-NAS, to replace the metric of the ground-truth task and evaluate the GNN architecture performance. The proxy regression task and its metric leading lead to higher ranking stability and faster search.", "summary_of_the_review": "This paper provides a proxy task approach to simplified evaluation metrics for GNN NAS. However, it doesn't provide sufficient proof that the proxy task improves search efficiency for GNN NAS.", "strength_and_weaknesses": "### Strengths:\n1. REG-NAS only needs one well-trained proxy model, unlike the previous work GenNAS which needs at least 20 models for CNN and RNN. \n\n2. The proposed proxy task requires fewer epochs to converge than the ground-truth task which reduces the search time.\n\n### Weaknesses:\n1. My most concern is the integrity of the paper. From Table 3, we can observe that the Spearman or Kendall Correlation Coefficient is not good enough (e.g., 0.362/0.260 for ogbg-molhiv and 0.421/ 0.303 for ZINC). I have no idea that the metric of the proxy regression task can really replace the downstream metric (AUC, MAE). Thus, in my opinion, the authors should show how the proposed REG-NAS improves the search efficiency of the NAS methods (e.g., RL, EA) on the downstream metric.\n\n2. There is the same question for another contribution: ranking stability. This paper takes a lot of space to talk about the proxy metric is stable and the ground-truth classification metric (AUC) is unstable under different initializations. However, in the practical application, we focus more on ground-truth metrics. If the ground-truth metric is unstable, why do we need a stable proxy metric to replace the ground-truth metric?\n\nIn my opinion, our problem is a GNN NAS problem and I think the proxy task may focus on improving search efficiency for GNN with ground-truth metric. However, it seems that this paper doesn't give sufficient proof for this aspect.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "fcE9mlM9D3", "reviewer": "Reviewer_xCbv", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposed to search GNN architectures by searching on a proxy regression task. The proxy regression task is generated by regressing some intermediate input features to output features from a pretrained GNN model which is randomly selected from the search space. The results are evaluated based on Ranking Coefficient Spearman’s ρ and Kendall’s τ.", "summary_of_the_review": "The manuscript can be improved for future submission.", "strength_and_weaknesses": "Strength:\n\nThe idea of using regression as a proxy task is interesting.\n\nConcerns:\n\n- Why the ranking stability for classification tasks can be much lower than for regression tasks? Any justification?\n\n- The authors claim to be the first to analyze factors that will affect models’ ranking stability for GNNs. However, to the best of my knowledge, this has been systematically studied in SGAS [1] which is not discussed. I believe a dedicated section for the comparison and discussion with SGAS is needed. \n\n- The authors propose to generate the proxy task by a randomly selected GNN architecture. However, it would cause significant biases toward similar architectures as the selected GNN architecture. For instance, if a GNN with sum aggregations is selected. The search algorithm will favor GNNs with sum aggregations because of the expressiveness of GNNs.\n\n- At the 3rd step of ReG-NAS, a model is selected to be evaluated GNNe from the search space. I wonder how the model is selected. Is it brute force, RL-based, evolution-based, or differentiable-based?\n\n- The final results of obtained architectures on ogbg-molhiv and ZINC are not reported. There is no comparison with SOTA models. \n\n[1] Li, G., Qian, G., Delgadillo, I.C., Muller, M., Thabet, A. and Ghanem, B., 2020. Sgas: Sequential greedy architecture search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 1620-1630).\n\n[2] Xu, K., Hu, W., Leskovec, J. and Jegelka, S., 2018. How powerful are graph neural networks?. arXiv preprint arXiv:1810.00826.\n\n\n======== post rebuttal ========\n\nThanks for the authors' reply. However, my main concern is that the method is only evaluated on a small search space with a brute-force search approach. It is nonclear if it can be generalized to a large search space. I keep my rating unchanged.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-rk_aaVDI71", "reviewer": "Reviewer_TB9W", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper designs a proxy task for GNN architecture search. Specifically, a GNN is randomly selected from the search space as the proxy generator. For each candidate architecture in the search space, MSE error between its feature map and that of the proxy generator is computed and used to rank this candidate. This paper then analyses the stability and correlation of such ranking method using two graph-level datasets.", "summary_of_the_review": "Please see Strength And Weaknesses", "strength_and_weaknesses": "I have several concerns about this paper:\n\n1. Finding a proxy task for GNN architecture search is not appealing, because GNN is typically not deep (especially in your search space) and its training is much cheaper compared with training a deep CNN or a transformer. Only moderate search time is needed to find a good GNN reliably (e.g., with evolutionary search). It is questionable whether reducing search time via a proxy task is worth the extra problems it brings.\n\n2. The description of the proxy pipeline is not clear enough. For example, for each candidate in the search space, is it trained to minimize the MSE error, or is it trained with the original loss function (MSE error is only used as the final evaluation)?\n\n3. The proxy design needs further justification. Feature maps from different message-passing layers are compared to the same F_p of the proxy generator GNN, which is counterintuitive.  Analysis from GNN's perspective should be added.\n\n4. The experimental results are not convincing enough. The correlation in Table 3 is poor and cannot support the proposed proxy task. Small search space, small graph size, only 2 datasets.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject