Efficient Automatic Machine Learning via Design Graphs
Authors: Ying-Xin Wu
Jiaxuan You
Jure Leskovec
Zhitao Ying
Venue: Submitted to ICLR 2023 (2023)
Keywords: Automated Machine Learning
Sample efficiency
Design graph
Graph Neural Networks

Abstract: Despite the success of automated machine learning (AutoML), which aims to find the best design, including the architecture of deep networks and hyper-parameters, conventional AutoML methods are computationally expensive and hardly provide insights into the relations of different model design choices. To tackle the challenges, we propose FALCON, an efficient sample-based method to search for the optimal model design. Our key insight is to model the design space of possible model designs as a design graph, where the nodes represent design choices, and the edges denote design similarities. FALCON features 1) a task-agnostic module, which performs message passing on the design graph via a Graph Neural Network (GNN), and 2) a task-specific module, which conducts label propagation of the known model performance information on the design graph. Both modules are combined to predict the design performances in the design space, navigating the search direction. We conduct extensive experiments on 27 node and graph classification tasks from various application domains, and an image classification task on the CIFAR-10 dataset. We empirically show that FALCON can efficiently obtain the well-performing designs for each task using only 30 explored nodes. Specifically, FALCON has a comparable time cost with the one-shot approaches while achieving an average improvement of 3.3% compared with the best baselines.

Under review as a conference paper at ICLR 2023
EFFICIENT AUTOMATIC GRAPH LEARNING VIA DESIGN RELATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Despite the success of automated machine learning (AutoML), which aims to find the best design, including the architecture of neural networks and hyper-parameters, conventional AutoML methods are computationally expensive and hardly provide insights into the relations of different model design choices. This work focus on the scope of AutoML on graph tasks. To tackle the challenges, we propose FALCON, an efficient sample-based method to search for the optimal model design on graph tasks. Our key insight is to model the design space of possible model designs as a design graph, where the nodes represent design choices, and the edges denote design similarities. FALCON features 1) a task-agnostic module, which performs message passing the design graph via Graph Neural Network (GNN), and 2) a task-specific module, which conducts label propagation of the known model performance information the design graph. Both modules are combined predict the design performances in the design space, navigating the search direction. We conduct extensive experiments on 27 node and graph classification tasks from various application domains. We empirically show that FALCON can efficiently obtain the well perfoornn designs for each task using only 30 explored nodes. Specifically, FALCON has comparable time cost with the one-shot approaches while achieving an average improvement of 3.3% compared with the best baselines.
INTRODUCTION
Automated machine learning (AutoMI.) (Liu et al., 2019; Pham et al., 2018; Bender et al., 2018; Real et al., 2019; Zoph & Le, 2017: Cai et al., 2019; 2021: Gao et al., 2019; You et al., 2020b; Zhang et al., 2021) has demonstrated great success in various domains including computer vision (Chu et al., 2020; Ghiasi et al., 2019; Chen al., 2019), language modeling (Zoph & Le, 2017; So et al., 2019), and recommender systems (Chen et al., 2022). It is an essential component for the state-of-the-ar deep learning models (Liu et al., 2018: Baker et al., 2017: Xu al., 2020; Chen et al., 2021).
Given a graph learning task, e.g., node/graph classification task on graphs, our goal of AutoML is search for model architecture and hyper-parameter setting from a design space that results in the best test performance on the task. Following previous works (You et al., 2020b), we define design as set of architecture and normalization, skip connection between consecutive layers), and define design space as the space of all possible designs for given task.
choices (e.g., 3 layer, 64 embedding dimensions, batch
However, AutoML is very computationally intensive. The design space of interest often involves millions of possible designs (Elsken et al.; You et al., 2020a). Sample-based AutoML (Zoph & Le, 2017: Gao et al., 2019; Bergstra et al., 2011; Liu et al., 2017; Luo et al., 201 8) has been used to perform search via sampling candidate designs from the design space to explore. One central challenge of existing sample-bar based AutoML solutions is its sample efficiency: it needs to train as few models as possible to identify the best rest-performing model in the vast design space. To improve the efficiency, existing research focuses on developing good search algorithms to navigate in the design space (White et al., 2021; Shi et al., 2020; Ma et al., 2019).
However, these methods do not consider modeling the effect of model design choices, which provides strong inductive biases in searching for the best- performing model. By "inductive bias", we refer to the patterns of multiple variables interacting together, which can happen in multiple parts of the

---

Under review as a conference paper at ICLR 2023
HR PENe:
po regoree
(elpert)
2:00
Inference Sample
0.90
0.82
Navigation
Ppre
#mp
#post
sloge
ogg
1.2
sklcconcat
min, cod
(a) Design graph example
(b) Search strategy of our method
Figure 1: Overview of FALCON. (a) Design graph example. We present a small design graph on TU-COX2 graph classification dataset. The design choices are shown in the table, #pre, #mp, #post denotes the numbers of pre-processing, message passing, and post-processing layers, respectively. The better design performance, the darker node colors. (b) FALCON search strategy. Red: Explored nodes. Green: Candidate nodes to be sampled from. Blue: The best node. Gray: Other nodes. Locally, FALCON extends the design subgraph via search strategy detailed in Section 3.3. Globally, FALCON approaches the optimal design navigated by the inductive bias of the design relations.
design space. Thus, an efficient search strategy should rapidly rule out a large subset of the design space with potentially bad performance leveraging such learned inductive bias.
Proposed approach. overcome the limitations, we propose FALCON, an AutoML framework on graph tasks that achieves state-of-the-art sample efficiency and performance by leveraging model design insights. Our key insight is to build design graph over the design space of architecture and hyper-parameter choices. FALCON extracts model design insights by learning meta-model that captures the relation between the design graph and model performance and uses it to inform a sample-efficient search strategy. FALCON consists of the following two novel components.
Design space as a graph. Previous works view the model design space as a nigh-dimmnsional space with isolated design choices (You et al., 2020b), which offer few insights regarding the relations between different design choices. For example, through trial runs if we find the models with more than layers do not work well without batch normalization, this knowledge can help us reduce the search space by excluding all model designs of more than layers with batch normalization set to false. While such insights are hardly obtained with existing AutoML algorithms (Liu al., 2019; Pham et al., 2018; Gao et al., 2019; Zoph & Le, 2017; Cai et al., 2019), FALCON achieves it via constructing graph representation, design graph, among all the design choices. Figure 1(a) shows a visualization of design graph, where each node represents candidate design, and edges denote the similarity between the designs. See Section 3.1 for details on the similarity and graph construction.
Search by navigating on the design graph. Given the design graph, FALCON deploys a Graph Neural Network predictor, short for meta-GNN. which is supervised by the explored nodes' performances and learns predict the performance of specific design given the corresponding node in the design graph. The meta-GNN is designed with 1) a task-agnostic module, which performs message passing on the design graph, and 2) task-specific module, which conducts label propagation of the known model performance information on the design graph. Furthermore, we propose a search strategy that uses meta-GNN predictions navigate the search in the design graph efficiently.
Experiments. We conduct extensive experiments on 27 graph datasets, covering node- and graph- level tasks with distinct distributions. Moreover, demonstrate FAL.CON' potential applicability image datasets by conducting experiments on the CIFAR-10 image dataset. Our code is available at https: h/tp////nnoomuu../////// 4open. wcwenccee//alllee
RELATED WORK
Automatic Machine Learning (AutoML) is the cornerstone of discovering state-of toatee-of-hhe-eae the-thhettt art model designs without costing massive human efforts. We introduce four types of related works below.
2

---

Under review as a conference paper at ICLR 2023
Sample-based AutoML methods. Existing sample-based approaches explore the search space via sampling candidate designs, which includes heuristic search algorithms, e.g. Simulated Annealing, Bayesian Optimization approaches (Bergstra et al., 2011; White et al., 2021; Ma et al., 2019), evolutionary- (Xie & Yuille, 2017; Real et al., 2017) and reinforcement-basee methods (Zoph & Le, 2017; Zhou al., 2019; Gao al., 2019). However, they tend to train thousands of models from scratch, which results in the low sample efficiency. For example, (Zoph & Le, 2017: Gao et al., 2019) usually involve training hundreds of GPUs for several days, hindering the development of AutoML in real-world applications (Bender et al., 2018). Some syper-parameter search methods aim to reduce the computational cost. For example, Successive Halving (Karnin et al., 2013) allocates the training resources to more potentially valuable models based on the early-stage training information. Li et al. (2017) further extend it using different budgets to find the best configurations to avoid the trade-off between selecting the configuration number and allocating the budget. Jaderberg et al. (2017) combine parallel search and sequential optimisation methods to conduct fast search. However, their selective mechanisms are only based on the model performance and lack of deep knowledge, which draws less insight into the relation of design variables and limits the sample efficiency.
One-shot AutoML methods. The one-shot approaches (Liu et al., 2019; Pham et al., 2018; Xie et al., 2019; Bender et al., 2018; Qin et al., 2021) have been popular for the high search efficiency. Specifically,, they involve training a super-net representing the design space, i.e,, containing every candidate design, and shares the weights for the same computational cell. Nevertheless, weight sharing degrades the reliability of design ranking, as fails to reflect the true performance of the candidate designs (Yu et al., 2020).
Graph-based AutoML methods. The key insight of our work to construct the design space as a design graph, where nodes are candidate designs and edges denote design similarities, and deploy a Graph Neural Network, i.e.. meta- Neta-GNN, to predict the design performance, Graph Hy- perNetwork (Zhang et al., 2019a) directly generates weights for each node in a computation graph representation. You et al. (2020a) study network generators that output relational graphs and analyze the link between their predictive performance and the graph structure. Recently, Zhao et al. (2020) considers both the micro- (i.e., a single block) and macro-architecture (i.e., block connections) of each design in graph domain. AutoGML (Park et al., 2022) designs meta-graph to capture the relations among models and graphs and take a meta-learning approach to estimate the relevance of models to different graphs. Notably, none of these works model the search space design graph.
Design performance predictor. Previous works predict the performance of a design using the learning curves (Baker al., 2018), layer-wise features (Deng et al., 2017), computational graph structure (Zhang et al., 2019a; White et al., 2021; Shi et al., 2019; Ma et al., 2019; Zhang et al., 2019b; Lee al., 2021a), combining dataset information (Lee al., 2021a) via dataset encoder. To highlight, FALCON explicitly models the relations among model designs. Moreover, it leverages the performance information on training instances to provide task-specific information besides the design features, which is differently motivated compared with Lee et al. (2021b) that employs meta- learnig techniques and incorporate hardware features to rapidly adapt to unseen devices. Besides, meta- GNN is applicable for both images and graphs, compared with Lee et al. (2021a).
3 PROPOSED METHOD
This section introduces our proposed approach FALCON for sample- based AutoML. In Section 3.1, we introduce the construction of design graph, and formulate the AutoML goal as search on the design graph for the node with the best task performance. In Section 3.2, we introduce our novel neural predictor consisting of a task-agnostic module and a task-specific module, which predicts the performances of unknown designs. Finally, we detail our search strategy in Section 3.3. We refer the reader to Figure (b) for a high-level overview of FALCON.
3.1 DESIGN SPACE AS A GRAPH
Motivation. Previous works generally consider each design choice as isolated from other designs. However, it is often observed that some designs that share the same design features, e,g. graph neural networks (GNNs) that are more than 3 layers and have batch normalization layers, may have similar performances. Moreover, the inductive bias of the relations between design choices can provide valuable information for navigating the design space for the best design. For example, suppose we find that setting batch normalization of 3-layer GCN (Kipf & Welling, 2017) and 4-layer GIN (Xu
3

---

Under review as a conference paper at ICLR 2023
et al., 2019) to false both degrade the performance. Then we can reasonably infer that 3-layer GraphSAGE (Hamilton et al., 2017) with batch normalization outperforms the one without. We could leverage such knowledge and only search for the designs that are more likely to improve the task performance. To the best of our knowledge, FALCON is the first method to explicitly consider such relational information among model designs.
Design graph. We denote the design graph as ( E), where the nodes N include the candidate designs, and edges & denote the similarities between the candidate designs. Specifically, we use the notion of design distance to decide the graph connectivity, and we elaborate on them below.
Design distance. For each numerical design dimension, two design choices have distance if they are adjacent in the ordered list of design choices. For example, if the hidden dimension size can take values [16, 32, 64, 128), then the distance between 16 and 32 is 1, and the distance between 32 and 128 is 2. For each categorical design dimension, any two distinct design choices have distance 1. We then define the connectivity of the design graph in terms of the design distance:
Definition 1 (Design Graph Connectivity) The design graph can be expressed as g(N, E),, where the nodes N (di...., d are model designs, and (di, d,) E iff the design distance between d; and dj is 1.
Structure of the design graph. The definition of edges implies that the design graph is highly structured, with the following properties: (1) All designs that are the same except for one categorical design dimension form a clique subgraph. (2) All designs that are the same except numerical design dimensions form grid graph structure. Moreover,
special calculation for the design distance with a combination of design dimensions that have dependencies. For example, the design dimensions of pooling operations, pooling layers, and the number of layers can depend each other, thus the design graph structure becomes more complex. See the details in Appendix A.2.
Design subgraph. The design graph may contain millions of nodes. Therefore, directly applying the meta-model to the design graph is computationally intensive. Moreover, a reliable performance estimation for unknown node depends on its similarity between the nodes already explored by the search algorithm. Therefore, we focus on using a meta-model to predict performance for a dynamic subgraph, i.e., design subgraph, containing the explored nodes in the current search stage and the candidate nodes to be sampled in the next step. The candidate set can be constructed by selecting the multi-hop neighbors of explored nodes on the design graph. The design subgraph is defined as:
Definition 2 (Design Subgraph) During a search, suppose the explored node set is No and the candidate set is No. The design subgraph is formulated as 9, (N,. Ex), where N₂ No UNa are the nodes and E. {u,,))u E Ne, v No, (u, N are the edges.
Given the design subgraph, we formulate the AutoMI. problem as searching for the node, iee, design choice, with the best task performance.
3.2 META-GNE FOR PERFORMANCE PREDICTION
Here we introduce a meta-model, named meta-GNN, to predict the performance of model designs, i..., nodes of the design subgraph. The goal of meta-GNN is learning the inductive bias of design relations, which is used to navigate the search path on the design graph. As is illustrated in Figure 2, the meta-GNN comprises a task-agnostic module and a task-specific module, used capture the knowledge of model design and task performance, respectively.
Task-agnostic module. The task agg-sagnsticc module uses design encoder to encode the design features on nodes of the design subgraph, and relation encoder to capture the design similarities and differences on edges of the design subgraph. After that, performs message passing on the design subgraph. We introduce each component below:
Design encoder: it computes the node features of design subgraph by the concatenation of the feature encoding of each design dimension. For numerical design dimensions, we conduct min-max normalization on their values as the node features. For categorical design dimensions such as aggregation operator which takes of (SUM, MAX, MEAN), we encode it as one-hot feature.
Relation encoder: it captures the similarity relationships between the connecting designs. For each (di,d) E E, encode the design dimension where d, and dj differ by one-hot encoding.
4

---

Under review as a conference paper at ICLR 2023
Training Instances
Design feature
Task-agnostic Module
Relation encodar
Designe
Message passing
Design encoder
Estimated performance
0.00
Entropy
0d8 6:0
Task-agnostic ambedding
0.65
Sofmax
MLP
Mai 0.05
ABr asa
Parkemance Information
Task-speciflc embedding
Sample
Ywar
You
propagation
Linear layer
Task-Specific Module
Figure 2: Meta-GNN Framework: Task- aagostic module generates the embedding given the design variables and their graphical structures. Task fask-specific module leverages performance information and conducts label propagation to generate the task-specific embeddings. The two embeddings are concatenated and input into an MLP for predicting the design performance.
Message passing module: GNN model used take the design subgraph and the processed features to perform message passing and output node representations. This information will be combined with the task-specific module to predict the design's 's performance.
Task-specific module. The task-specific module takes into account the information of design performance on selected training instances and thus is specific for one dataset.
The challenge of including such task-specific performance is that it is only available very limited set of explored nodes. To overcome the challenge, we use label propagation to propagate the performance information of explored nodes the unexplored nodes. This is based our observation that models trained with similar designs typically make similar predictions on instances. We provide an example in Figure 2 to illustrate the spec-ic module.
The first step is to identify critical training instances that result in different performances across different designs. Here we use a set of explored designs (anchors) to provide the instance-wiss performances. Specifically, the (i, j) element of the top left matrix of Figure 2 represents whether the i-th design can correctly predict the label of j-th instance. Then, compute the entropy of each training instance's performance over the anchors. Then obtain the instance-wisg wise probability via Softmax on the entropy vector, from which we sample instances that result in high variation across designs. The high variation implies that these instances can distinguish good designs from bad ones in the design subgraph, which are informative.
Identifying critical instances:
Label propagation and linear projection: Based on the inductive bias smoothness, we per- form label propagation to make the task-specific information available to all candidate designs. Concretely, label propagation can be written as
(k+1) a 1/2 AD (12 + (1 (
(I)
where each row of Y is the performance vector of design (if explored) or a zero vector (for the unexplored designs). D E R!N, x|Val is the diagonal matrix of node degree, A E R [.INs is the adjacent matrix of the design subgraph, and is a hyper-parameter After label propagation, we use a linear layer to project the performance information to another high-dimensional space.
Finally, as shown in Figure 2, we concatenate the output embeddings of the task-specific and task- agnostic modules and use an MLP to produce the performance predictions.
Objective for Meta-GNN. The training of neural performance predictor is commonly formulated as regression using mean square error (MSE), in order to predict how good the candidate designs are for the current task. However, the number of explored designs is usually small for AutoML, especially during the early stage of the search process. Thus, the deep predictor tends to overfit, degrading the reliability of performance prediction. To solve this problem. we incorporate wair-wise rank loss (Burges et al., 2005; Hu et al., 2021) with the MSE objective, resulting in
efficient
5

---

Under review as a conference paper at ICLR 2023
Algorithm 1: Pseudocode of FALCON
Require: S: Design space. K: Exploration size. hę: Meta-GNN. V/V' Warm-up / Full epoch.
Learning rate. C: Number of start nodes.
// Initialize the exploration set // Construct candidate set from multi-hop neighbors 3: Yn = GET- DEET-VALIDATION-PEEE SAALIDATTONNPPRFORRANEECCCCC V) Explore the initial nodes (for epochs)
1: Q < SAMPLE-NODES C)
2: r HULLTI-HO--NEIGHEER
4: while | < K do
5: G. - (N₂, = QUI,, SIMILARITY(NG)
// (1) Update the design subgraph
6: while not converge do
7: / - A vLa,V,, ( )a)/00
I/ (2) Compute Eq. 2 and conduct optimization
8:
end while
9:
// (3) Sample candidate node with probability proportional to the meta-GNN's prediction (hg (he (G!"))))) )r))
10:
d(() = PROBLEELLIITPP Softmax
11:
Y,
BEE-VVALDDATTON-PEERRONAAMM
V) // (2) Explore the current selected node
12: Ω < QU (2 PU
13: end while
D SELECT-TOPK((Q; YJ size MIN([10% K), Models to be fully trained // Obtain the final performance // Obtain best model
15: Y
16: I ARGMAX(Y"")
17: return D1
a quadratic number of training pairs, thus reducing overfitting. Furthermore, predicting relative performance is more robust across datasets than predicting absolute performance. Overall, the objective is formulated as follows:
N
(( (Y: Y,)² + (Y, where
i=1
Y
Y
Crank (8,8)
Σ Σ (1)10)>)))....
where À is the trade-off hyper-parameteer 7 is the temperature controlling the minimal performance gap that will be highly penalized, and a is the Sigmoid function. Thus, the meta-GNT is trained to predict the node performance on the design subgraph supervised by the explored node performance.
3.3 SEARCH STRATEGY
Equipped with the meta-GNN, we propose a sequential search strategy search for the best design in the design graph. The core idea is to leverage meta-GNN to perform fast inference on the dynamic design subgraph, and decide what would be the next node explore, thus navigating the search, We summarize our search strategy in Algorithm 1. Concretely, our search strategy consists of the following three steps:
Initialization: As shown in Figure I (b), FALCON randomly samples multiple nodes on the design graph. The motivation of sampling multiple nodes in the initial step is to enlarge the receptive field on the design graph to avoid the local optima and bad performance, which is empirically verified in Appendix D. Then, FALCON explore the initialized nodes by training designs on the tasks and construct the instance mask for task-specific module.
Meta-GNN training: Following Figure meta-GNN predicts the performance of the explored nodes. The loss is computed via Equation 2 and baok-propagated to optimize the meta-GNN.
Exploration via inference: Meta-GNN is then used to make predictions for the performances of all candidate nodes. Then we apply Softmax on the predictions as the probability distribution of candidate designs, from which FALCON samples a new node and updates the design subgraph.
At every iteration, FALCON extends the design subgraph through the last two steps. After several iterations, it selects and retrains few designs in the search trajectory with top performances. Overall,

---

Under review as a conference paper at ICLR 2023
FALCON approaches the optimal design navigated by the relational inductive bias learned by meta- GNN, as shown in Figure I (b).
EXPERIMENTS
We conduct extensive experiments 27 graph datasets and image dataset. The goal is twofold: (1) to show FALCON's sample efficiency over the existing AutoML methods (ef. Section 4.2) and (2) to provide insights into how the inductive bias of design relartions navigate the search on design graph (cf. Section 4.3).
4.1 EXPERIMENTAL SETTINGS
We consider the following tasks in our evaluation and we leave the details including dataset split, evaluation metries, and
Appendix A.
Node classification. We use 6 benchmarks ranging from citation networks to product social networks: Cora, CiteSeer, PubMed (Sen et al., 2008), ogbn-arxiv (Hu et al., 2020), AmazonComput ers (Shchur et al., 2018), and Reddit (Zeng al., 2020).
Graph classification. We use 21 benchmark binary classification tasks in TUDataset (Morris et al., 2020), which are to predict certain properties for molecule datasets with various distribution. Image classification. We use CIFAR-10 (Krizhevsky, 2009). See details in Appendix C.
Baselines. We compare FALCON with three types of baselines:
Simple search strategies: Random, Simulated Annealing (SA), Bayesian Optimization (BO) (Bergstra et al., 2011).
AutoML approaches: DARTS (Liu et al., 2019). ENAS (Pham al., 2018), GraphNAS (Gao et al., 2019), Auto/Attend (Guan et al., 2021), GASSO (Qin et al., 2021), where the last three methods are specifically designed for graph tasks.
Ablation models: FALCON-G and FALCON-LP, where FALCON-G discards the design graph and predicts the design performance using an MLP, and FALCON-LP removes the task-specific module and predicts design performance using only the task-agnostic module.
We also include naive method, BRUTEFORCE, which trains 5% designs from scratch and returns the best design among them. The result of BRUTEFORCE regarded as the approximated ground truth performance. We compare FALCON and the simple search baselines under sample size controlled search, where we limit the number of explored designs. We set the exploration size as 30 by default.
Design Space. We use different design spaces on node- and graph-level tasks. Specifically, The design variables include common prpeerpprameters. e.g. dropout ratio, and architecture choices, e.g., layer connectivity and batch normalization. Moreover, we consider node pooling choices for the graph classification datasets, which is less studied in the previous works (Cai et al., 2021: Gao et al., 2019; Zhou et al., 2019). Besides, we follow You al. (2020b) and control the number of parameters for all the candidate designs to ensure fair comparison. See Appendix A.2 for the details.
4.2 MAIN RESULTS
Node classification tasks. Table summarizes the performance of FALCON and the baselines.
Notably, FALCON takes comparable search cost as the one- shot methods and is 15x less expensive than GraphNAS. Moreover, FALCON achieves the best performances over the baselines with sufficient margins in the most datasets, using only 30 explored designs. For example, FALCON outperforms ENAS by 1.8% in CiteSeer and GASSO by 1.6% in Ama- zonComputers. Also, the removal the design graph and task-specific module decreases the performance constantly, which validates their effectiveness. is worth mentioning that FALCON is competitive with BRUTEFORCE, demonstrating the excellence of FALCON in searching for globally best- performing designs.
0.65
0.60
0.55
0.50
0.45
NALCON
10
100
Number of Explored
Figure 3: Accuracy v.s. the number of explored nodes on ogbn-arxiv.
7

---

Under review as a conference paper at ICLR 2023
Table I: Search results on five node classification tasks, where Time stands for the search cost (GPU-hours). We conduct t-test to compute p-value on our method with the best AutoML baselines.
Cora
CiteSeer
Pubmed
AmazonComputer
Reddit
ACC
Time
ACC
Time
ACC
Time
ACC
Time
F1
Time
Random
80.811.
0.20
71.2108
0.22
86.0133
0.24
81.6130
0.16
94.310.1
0.97
BO
25.
0.28
72.6209
0.30
88.5+a1
0.31
82.3263
0.16
94.220.
0.94
SA
81.1108
0.24
74,7102
0.25
88,9101
0.29
81.2:69
0.23
94.310.
0.97
ENAS
85.810.
0.27
74.9102
0.39
88.6108
2.06
74,51
0.83
92.3110
1.98
DARTS
85 2
0.25
75.22n
0.25
19.1=a1
0.35
84.1=19
0.35
[OnM]
GraphNAS
82.2113.
3.12
74.910.
3.99
89.21003
5.37
88.5
19.11229
3.03
AuteAttend
84.6=02
1.23
73.9
1.25
84.4+a7
1.55
87.3=111
2.62
[OoM]
GASSO
86.8
0.38
75.3
0.33
86.3+a
0.41
89.8=a1
0.73
[OoM]
FALCON-G
84.5=0.
0.23
74.3+1.7
0.24
89.2+au
0.26
87.6=09
0.27
93.7+04
1.11
FALCON-LP
15
0.26
74.6+0.1
0.26
89.0=a2
0.29
90.7=aa
0.30
94.9
1.00
FALCON
86.4
0,26
76.210.
0,28
89,3
0.32
91.2=
0.30
95.2::03
1.15
BRUTEFORCE
87.0
52.5
76.0
59.7
90.0
63.0
91.4
81.5
95.5
2000
p-value
0.051
0.145
0.017
0002
trade-off of FALCON and other sample-based approaches in ogbn-arxiv, arxiv. We run several search trials under different sample sizes. As shown in Figure FALCON reaches the approximated ground truth result with very few explored nodes. In contrast, SA and Random require more samples to converge, while BO performs bad even with large number of explored nodes, potentially due to its inability in dealing with high-dimensional design features.
We further investigate the
Graph classification tasks. The graph classification datasets cover a wide range of graph distributions. In Table 2, we report the selected performance results for graph classification tasks and leave other results including the search costs in Appendix B. We highlight three observations:
On average, the state-of-the-ar AutoML baselines algorithms perform close to the simple search methods, indicating the potentially unreliable search, as similarly concluded by Yu et al. (2020).
FALCON surpasses the best AutoML baselines with an average improvement of 3.3%. The sufficient and consistent improvement greatly validates our sample efficiency under controlled sample size. where FALCON can explore the designs that are more likely to perform well through the relational inference based on the relations of previously explored designs and their performances.
In the second block, we attribute the high sample efficiency of FALCON to the exhibition of design relations and the performance information from the training instances. Specifically, FALCON outperforms FAL.CON-LP by 4.87% on average, indicating that the task-specific module provides more task information that aids the representation learning of model designs, enabling a fast adaption on a certain task. Moreover, FALCON gains an average improvement of 6.43% compared to FALCON-G, GG which justifies relational inductive bias and guide the search on the design graph.
motivation that the design relations promote the learning of
We also conduct experiments similar to Figure 3 to investigate how FALCON converges with the increasing sample size (ef. Appendix B.1) and report the best designs found by FALCON for each dataset (cf. Appendix B.2). Besides, provide sensitivity analysis on FALCON's Hyper-parameteed e.g., number random start nodes C (cf. Appendix D).
Image classification task. We demonstrate the potential of FALCON in image domain. Due to space limitation, we leave the results of CIFAR-10 to Appendix C. We found FALCON can search for designs that are best- terforerrrnigg. compared with the baselines. Specifically, it gains average improvements of 1.4% over the simple search baselines and 0.3% over the one-shot baselines on the architecture design space, with search cost comparable to the one-shot based baselines.
4.3 CASE STUDIES OF FALCON
We study FALCON in two dimensions: (1) Search process: we probe FALCON's inference process through the explanations of meta-GNN on a design graph, and (2) Design representations: we visualize the node representations output by the meta-GNN to examine the effect of design choices.
Search process. We use GNNExplainer (Ying et al., 2019) to explain the node prediction of meta- GNN and shed light on the working mechanism of FALCON. Here consider the importance of
8

---

Under review as a conference paper at ICLR 2023
Table 2: Selected results for the graph classification tasks. The average task performance (ROC-AUC) of the architectures searched by FALCON is 3.3% over the best AutoML baselines.
ER-MD
AIDS
OVCAR-8
MCF-7
SNI2C
NC1109
Tox21-AhR
Avg.
Random
1..
97.0+14
56,240.
58.2#03
57.4110
73.4109
75.7=2.
70.8
BO
77.6#35 96.1+10
63.640.7
60.7
54.8
73.6=12
75.5#1.1
71.7
SA
75.9+4.2 95,4+09
59.5#32
56.7+as
60.4+1.7
76.6#56
76.5#3.
71.6
ENAS
76,0122
97.1
56.0113
59.7108
66.4106
71.2110
73.610.9
71.4
DARTS
75.0.0.7
98.0100
56.810.3
60.2107
66.010.4
73.5
76.0
72.2
GraphNAS
76.9
95.9103
58.7
61.3a52
60.7
73.6129
70,6
71.1
AutoAttend
73.1
97.4+03
59.8
64.4
71.8
75.9
74.1
73.8
GASSO
73.2:04
95.2107
62.3203
62.5
70.9
73.9
70.243.
72.6
FALCON-G
78.3+30
96.3+11
56.4+1.1
62,3#45
69.8+22
70.3+64
72.5+28
72.3
FALCON-LP
76.7+224
96.0+02
61.5+49
59.5=57
70.3+18
73.1
76.5+25
73.3
FALCON
78.4:002
97.5+11
66.7=34
65.5=25
73.3+00
78.4:23
78.5
76.9
BRUTEFORCE
83,3
96.0
67.4
70.6
73.7
81.8
82.0
79.3
p-value
0.155
0.008
0.035
<0.001
0.096
0.018
each design dimension for each node's prediction. We demonstrate on a real case when searching on CIFAR-10 10 (cf. Table 12 for the design space). For conciseness, we focus on two design dimensions: (Weight Decay, Batch Size). Then, given a node of interest n' (0.9, 128), we observe the change in its predictions and dimension importance during the search process.
Explored node nT:
(0.99, 64)
(0.9, 64)
(0.99, 128)
Performance of nt: Prediction on n':
++
+
0.90
0.77
0.89
Dimension importance:
[0.5, 0.5]
[0.8, 0.2]
[0.6, 0.4]
Where + and indicate the relative performance of the explored nodes, / is the current search step. that the prediction ní' and the dimension importance evolve with the explored designs and their relations. For example, when weight decay changes from 0.99 to 0.9, there is huge drop the node performance, which affects the prediction of n' and increases the importance of Weight Decay as design performance seems to be sensitive to this dimension.
Interestingly,
Design representations. In Figure visualize the high-dimensional design representations via T-SNE (van der Maaten & Hinton, 2008) after training the meta-GNN on the Cora dataset.
In the left figure, the better the design performance, the darker the color. Generally, the points with small distance have similar colors or performances, indi- cating that meta- can distinguish "good" nodes from "bad" nodes. For the right figure, different colors represent different dropout ratios. The high discrimination indicates that the dropout ratio is an influential variable for learning the design representa- tion, which further affects design performance. This evidence validates the meta-GNNT expressiveness and capacity to learn the relational inductive bias between the design choices.
Figure 4: T-SNE visualization for the design representations on Cora dataset.
5 CONCLUSION, LIMITATION, AND FUTURE WORK
This work introduces FALCON, an efficient sample-based AutoML framework. We propose the concept of design graph that explicitly models and encodes relational information among model designs. On top of the design graph, we develop sample-efficient strategy to navigate the search on the design graph with a novel meta-model. One future direction is to better tackle the high average node degree on the design graphwhich could cause over-smoothing, especially when the design variables include many categorical variables. And a simple solution is to use edge dropout to randomly remove portion of edges at each training epoch. Another future direction is to better adapt FALCON on continuous design variables via developing a dynamic design graph that enable a more fine- grained search between the discretized values.

---

Under review as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
All of the datasets used in this work are public. For experimental setup, we state the detailed settings in Appendix A and Appendix C, including the graph pre-processing, dataset splits, hyper-parameters. Moreover, we include our code in an anonymous link for public access. For the results, we report the best models found by algorithm well their corresponding performances. Overall, we believe we have made great efforts to ensure reproducibility in this paper.
ETHICS STATEMENT
In this work, we propose a novel algorithm to search for the best model designs where no human subject related. This work could promote the discovery of more powerful and expressive models and provide insights into design relations. However, while best-performing models may be "experts" in fulfilling given tasks, they are not necessarily fair towards different user or entity groups. We believe this is general issue in the AutoML area and should be well addressed to ensure the ethics of models in real-world applications.
REFERENCES
Bowen Baker, Otkrist Gupta, Nikhil Naik, and Ramesh Raskar. Designing neural network architec- tures using reinforcement learning. In ICLR, 2017.
Bowen Baker, Otkrist Gupta, Ramesh Raskar, and Nikhil Naik. Accelerating neural architecture search using performance prediction. In ICLR, 2018.
Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, and Quoc V. Le. Understand- ing and simplifying one-shot architecture search. In ICML. 2018.
James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for prrampprreeeter optimization. In NeurlPS, 2011.
Filippo Maria Bianchi, Daniele Grattarola, Lorenzo Livi, and Cesare Alippi. Graph neural networks with convolutional ARMA filters. arXiv, 2019.
Christopher C. Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Gregory N. Hullender. Learning rank using gradient descent. In /CML, 2005.
Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search target task and hardware. In ICLR, 2019.
Shaofei Cai, Liang Li, Jincan Deng, Beichen Zhang, Zheng- Jun Zha, Li Su, and Qingming Huang. Rethinking graph neural architecture search from message-passing In CVPR, 2021.
Bo Chen, Xiangyu Zhao, Yejing Wang, Wenqi Fan, Huifeng Guo, and Ruiming Tang. Automated machine learning for deep recommender systems: A survey, 2022.
Xiangning Chen, Ruochen Wang. Minhao Cheng. Xiaocheng Tang, and Cho-Jui Hsich. Drnas: Dirichlet neural architecture search. In ICLR, 2021.
Yukang Chen, Tong Yang, Xiangyu Zhang, Gaofeng Meng, Chunhong Pan, and Jian Sun. Detnas: Neural architecture search object detection. 2019.
Xiangxiang Chu, Zhang, Hailong Ma, Ruijun Xu, and Qingyuan Li. Fast, accurate and lightweight super-resolution with neural architecture search. In ICPR. IEEE, 2020.
Boyang Deng, Junjie Yan, and Dahua Lin. Peephole: Predicting network performance before training. Arxiv, 2017.
Frederik Dichl. Edge contraction pooling for graph neural networks. CoRR, abs/1905.10090 2019.
Jian Du Shanghang Zhang, Guanhang Wu, José M. F. Moura, and Soummya Kar. Topology adaptive graph convolutional networks.
10

---

Under review as a conference paper at ICLR 2023
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. J. Mach. Learn. Res.
Hongyang Gao and Shuiwang Ji. Graph u-nets. In ICML, volume 97, pp. 2083-2092, 2019.
Yang Gao, Hong Yang, Peng Zhang, Chuan Zhou, and Yue Hu. Graphnas: Graph neural architecture search with reinforcement learning. arXiv, 1904.09981, 2019.
Golnaz Ghiasi, Tsung-Y Yi Lin, and Quoc V. Le. NAS-FPN: learning scalable feature pyramid architecture for object detection. In CVPR, 2019.
Chaoyu Guan, Xin Wang, and Wenwu Zhu. Autoattend: Automated attention representation search. In ICML, 2021.
William L. Hamilton, Zhitao Ying. and Jure Leskovec. Inductive representation learning on large graphs. In NeurIPS, pp. 1024-1034, 2017.
Chi Hu, Chenglong Wang, Xiangnan Ma, Xia Meng, Yinqiao Li, Tong Xiao, Jingbo Zhu, and Changliang Li. Ranknas: Efficient neural architecture search by pairwise ranking, In EMNLP, 2021.
Weihua Hu, Matthias Fey, Marinka Zitmik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. arXiv preprint arXiv:2005.00687, 2020.
Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojeiech M. Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, lain Dunning, Karen Simonyan, Chrisantha Fernando, and Koray Kavukcuoglu. Population based training of neural networks. CoRR, 2017.
Zohar Shay Karnin, Tomer Koren, and Oren Somekh. Almost optimal exploration in multi-armed bandits. In ICML, 2013.
Thomas N. Kipf and Max Welling, Semi-supervised classification with graph convolutional networks. In ICLR, 2017.
Alex Krizhevsky. Learning multiple layers of features from tiny images. In Technical report, 2009.
Hayeon Lee, Eunyoung Hyung, and Sung Ju Hwang. Rapid neural architecture search by learning to generate graphs from datasets. In ICLR, 2021a.
Hayeon Lee, Sewoong Lee, Song Chong, and Sung Ju Hwang. Hardware-adaptive efficient latency prediction for NAS via meta-learning. In NeurlPS, 2021b.
Junhyun Lee, Inyeop Lee, and Jaewoo Kang. Self-attention graph pooling. In ICML, 2019.
Lisha Li, Kevin G. Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hy- perband: A novel bandit-based approach to hyperparameter optimization. J. Mach. Learn, Res., 2017.
Chenxi Liu, Barret Zoph, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan L. Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. 1712.00559, 2017.
Hanxiao Liu, Karen Simonyan, Oriol Vinyals, Chrisantha Fernando, and Koray Kavukcuoglu. Hierar- chical representations for efficient architecture search. In ICLR, 2018.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: differentiable architecture search. In ICLR, 2019.
Renqian Luo, Fei Tian, Tao Qin, Enhong Chen, and Tie- Liu. Neural architecture optimization. In NewrlPS, 2018.
Lizheng Ma. Jiaxu Cui, and Bo Yang. Deep neural architecture search with deep graph bayesian optimization. In WI. ACM, 2019.
11

---

Under review as a conference paper at ICLR 2023
Zheng Ma, Junyu Xuan, Yu Guang Wang, Ming Li, and Pietro Liò. Path integral based convolution and pooling for graph neural networks. 2020.
Christopher Morris, Martin Ritzert, Matthias Fey, William L. Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe. Weisfeiler and leman go neural: Higher-order graph neural networks. In AAAI, pp. 4602-4609, 2019.
Christopher Morris, Nils M. Kriege, Franka Bause, Kristian Kensting, Petra Mutzel, and Marion Neumann. Tudataset: A collection of benchmark datasets for learning with graphs. 2020.
Namyong Park, Ryan A. Rossi, Nesreen K. Ahmed, and Christos Faloutsos. Autogml: Fast automatic model selection for graph machine learning. CoRR, 2022.
Hieu Pham, Melody Y. Guan, Barret Zoph, Quoc V. Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. In ICML, 2018.
Yijian Qin, Xin Wang, Zeyang Zhang, and Wenwu Zhu. Graph differentiable architecture search with structure learning. In NeurlPS, 2021.
Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoe Le, and Alexey Kurakin. Large-scale evolution of image classifiers. In ICML, 2017.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V. Le. Regularized evolution for image classifier architecture search. AAAI Press, 2019.
Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Gallagher, and Tina Ritassi-Rad Collective classification in network data. Al Mag., 2008.
Oleksandr Shchur, Maximilian Mumme, Aleksandar Bojchevski, and Stephan Günnemann. Pitfalls of graph neural network evaluation. 2018
Han Shi, Renjie Pi, Hang Xu, Zhenguo Li, James T. Kwok, and Tong Zhang. Multi-objective neural architecture search via predictive network performance optimization. 1911.0933, 2019.
Han Shi, Renjie Pi, Hang Xu, Zhenguo Li, James T. Kwok, and Tong Zhang. Bridging the gap between sample-based and one-shot neural architecture search with BONAS. In NeurIPS, 2020.
David So, Quoc V. Le, and Chen Liang. The evolved transformer. In ICML, 2019.
Laurens van der Maaten and Geoffrey Hinton. Visualizing high-dimensional data using t-sne. Journal of Machine Learning Research, 2008
Colin White, Willie Neiswanger, and Yash Savani. BANANAS: bayesian optimization with neural architectures for neural architecture search. In AAAI, 2021.
Lingxi Xie and Alan L. Yuille. Genetic CNN. In JEEE, 2017.
Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. SNAS: stochastic neural architecture search. In ICLR, 2019.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In ICLR, 2019.
Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Guo-Jun Qi, Qi Tian, and Hongkai Xiong. PC-DARTS: partial channel connections for memory-efficient architecture search. In ICLR, 2020.
Zhitao Ying, Dylan Bourgeois, Jiaxuan You, Marinka Zitikk, and Jure Leskovec. Gnnexplainer: Generating explanations for graph neural networks. NeurIPS, pp. 9240-9251, 2019.
Jiaxuan You, Jure Leskovec, Kaiming He, and Saining Xie. Graph structure of neural networks. In ICML, 2020a.
Jiaxuan You, Zhitao Ying, and Jure Leskovec. Design space for graph neural networks. In NeurIPS, 2020b.
12

---

Under review as a conference paper at ICLR 2023
Kaicheng Yu, Christian Sciuto, Martin Jaggi, Claudiu Musat, and Mathieu Salzmann. Evaluating the search phase of neural architecture search. In ICLR, 2020.
Hanqing Zeng, Hongkuan Zhou, Ajitesh Srivastava, Rajgopal Kannan, and Viktor K. Prasanna. Graphsaint: Graph sampling based inductive learning method. In ICLR, 2020.
Chris Zhang. Mengye Ren, and Raquel Urtasun. Graph hypernetworks for neural architecture search. In ICLR, 2019a.
Muhan Zhang, Shali Jiang, Zhicheng Cui, Roman Gamett, and Yixin Chen. D-VAE: A variational autoencoder for directed acyclic graphs. In NeurIPS, 2019b.
Ziwei Zhang, Xin Wang, and Wenwu Zhu, Automated machine learning on graphs: A survey. In IJCAI, 2021.
Yiren Zhao, Duo Wang. Xitong Gao, Robert Mullins, Pietro Liò, and Mateja Jamnik. Probabilistic dual network architecture search on graphs. 2020.
Kaixiong Zhou, Qingquan Song, Xiao Huang, and Xia Hu. Auto-gnm: Neural architecture search of graph neural networks. 2019.
Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. In ICLR, 2017.
13

---

Under review as a conference paper at ICLR 2023
A EXPERIMENT DETAILS
A. SETTINGS
Graph classification datasets. The graph classification datasets used in this work are summarized in Table 3. And the detailed dataset statics can be referred from https: //chssrrrs thub. io blatasets/docs /at.ssss///ttss//.../
Table 3: List the graph classification datasets used in this work.
AIDS, BZR-MD, COX2-MD, DHFR-MD, Mutagenicity, NCII, NC1109, PTC-MM, PTC-MR
Small Scale
Tox21-ARR, MCF-7, MOLT-4, UACC257, Yeast, NCI-H23, OVCAR-8, P388, PC-3, SF-295, SNI2C, SW-620
Medium/Large Scale
Specifically, all datasets are binary classification tasks that predict certain properties for small molecules. For example, the labels Tox21-AhR represent toxicity/non-txxiiity. while the graphs in Mutagenicity classified into two classes based on their mutagenic effect bacterium (Morris et al., 2020). Consequently, we use atom types as the node features and bond types as edge features.
Evaluation metrics. For Reddit, previous work (Zeng et al., 2020). For other node classification tasks and image dataset, we use classification accuracy as the evaluation metric. For the graph classification tasks, we ROC-AUC as the evaluation metric.
F1 score (micro) the evaluation metric following the
the standardized dataset split. For other node classification datasets, we split the nodes in each graph into 70%, 10%, 20% in training, validation, and test sets, respectively. For graph classification tasks, we split the graphs into 80%, 10%, 10% for training, validation, and test sets, respectively.
Dataset splits. For ogbn-arxiv and Reddit,
Hyper-Parameter We tuned the syper-parametert of the baselines based the default setting in their public codes. For FALCON, we construct the candidate set as the 3-hop neighbors of the explored nodes and the number of start nodes as min([10% K7, 10), where denotes the exploration size. The meta- Neta-GNN/ is constitute of 3 message-passing layers and 3 label propagation layers. All the experiments are repeated least 3 times.
A.2 DESIGN SPACES
A.2.1 DESIGN SPACES FOR THE SAMPLE-BASED METHODS
In this work, we use different design spaces for the datasets depending on the task types, i.e., node or graph level. We summarize the design variables and choices in Table 4 and Table 5. For the design space of Reddit, we replace "Aggregation" in Table with "Convolutional layer type", which takes values from (GCNConv (Kipf & Welling, 2017), SAGECony (Hamilton al., 2017). GraphConv (Morris et al.. 2019), GINConv (Xu et al., 2019), ARMAConv (Bianchi et al., 2019), TAGConv (Du et al.)).
14

---

Under review as a conference paper at ICLR 2023
Table 4: Design Space for the node-level tasks (except for Reddit). 5,832 candidates in total.
Type
Variable
Candidate Values
Hyper-panee Saper--arameters
Dropout ratio
[0.0, 0.3, 0.6]
Pre-process layers
[1, 2, 3]
# Message passing layers
[2. 4, 6, 8]
Post-precess layers
[1, 2, 3]
Architecture
Layer connectivity Activation Batch norm Aggregation
STACK, SUM, CAT
ReLU, Swish, Prelu
True, False
Mean, Max, SUM
Table 5: Design Space for the graph-level tasks. 58,320 candidates in total.
Type
Variahle
Candidate Values
Hyper-parameters
Dropout ratio
[0.0, 0.3, 0.6]
# Pre-process layers
[1, 3]
Message passing layers # Post-precess precess layers Layer connectivity Activation Batch norm Aggregation
[2. 4. 6, 8]
[1, 2, 31
STACK, SUM, CAT
ReLUU, Swish, Prelu
Architecture
Truc, False SUM
Mean, True, False
Node pooling flag (Use node pooling)
TopkPool (Gao & Ji. 2019). SAGPool (Lee al., 2019). PANPool (Ma al., 2020), EdgePool (Dieh), 2019) 4,
Node pooling type (if applicable)
Node pooling loop
Specifically, The STACK design choice means directly stacking multiple GNN layers, i.e., without skip-connections. We also support node pooling operations for graph classification tasks, where the pooling loop stands for the number of message passing layers between each pooling operation. If the number of message passing layers is m and the node pooling loop is I, there will be node pooling layer after the ith message passing layer in the design model (hierarchical pooling), where i e {1 + k I k = ...... [m 1)/( 1}. Moreover, to avoid duplicated and invalid designs, some design variables are required satisfy specific dependency rules, and we take two examples to elaborate on this point.
If the node pooling flag of a design is False, then the design does not have any value on node pooling type and node pooling loop, and vice versa.
For example, denote node pooling flag f. node pooling type and as any design choice, then(/=False, (=+) ) or (/=False, I=*) will both be invalid.
The node pooling loop should not exceed the number of message passing layers.
For example, design A (m=4, (=4) and design B (m=4, [=6) that take the same values on other design variables are duplicated.
Thus, the design graph constructed under dependency rules is more complex. Without loss of generality, we define that the distance (f=False) and (f=True, I=MIN((i L))) as 1, where L represents the design choice of node pooling loop. Thus, the design graph is a connected graph that enables the exploration of any node with random initialization. It is also worth mentioning that the search strategy of FALCON is modularized given the design graph. In contrast, the dependency rules constrain the action space of reinforcement learning methods, e.g., (f=Ture - False) is inapplicable, which requires special operation inside the controller.
15

---

Under review as a conference paper at ICLR 2023
Table 7: Design space for the one-shot baselines on node and graph classification tasks.
Variable
Candidate Values
Dropout ratio
[0.0, 0.3, 0.6] STACK, SUM
Layer connectivity
Pre-process layers
[1. 2, 3]
# Message passing layers
[2, 4, 6, 8]
Post-precess layers
[1. 2, 3]
Activation
ReLU, Swish, Prelu
Batch norm
True, False
Aggregation
Mean, Max, SUM
Table 6: Statistics and the construction time of the design graphs.
#Nodes
#Edges (Undirected)
Ave. Degree
Diameter
construction time (s)
DG-1
5,832
78.732
13.5
13
13
DG-2
58,320
1,070,172
18.4
17
147
We further summarize the statistics and construction time of the design graphs in Table 6, where DG-1 and DG-2 denote the design graphs for node-level and graph-level tasks, respectively. We use multi-processing programing
50 CPUs (Intel Xeon Gold 5118 CPU @ 2.30GHz) graph construction. Note that we don't have to construct the entire design graph in the pre-processing step, since we only extend the small portion of the design graph, i.e., the design subgraph, during the search process. Thus, the total time cost of constructing the design subgraph will be O(E') where E' is the number of edges in the design subgraph, which largely lowers the time costs.
conduct the
A.2.2 DESIGN SPACES FOR THE ONE-SHOT BASELINES
The one-shot models (Liu al., 2019; Pham et al., 2018) is built upon super-model that is required to contain all of the architecture choices. We build the macro search space over entire models for both node classification and graph classification datasets with constraints. Firstly, do not consider CAT (skip-concatenate) a layer connectivity choice, and we also remove design variables for node pooling. The reason is that CAT and node pooling operations change the input shape and make the output embeddings inapplicable for the subsequent weight-sharing modules in our settings. Secondly, the layer connectivity customized for each layer following the previous works (Liu al., 2019; Pham et al., 201 8), instead of setting as global value for every layer. Overall, we summarize the design space in Table 7.
To enable a fair comparison, we fine-tune the hyper-parameters and report the best results of the architectures found by the one-shot methods according to their performance in the validation sets.
B
MORE EXPERIMENTAL RESULTS ON GRAPH TASKS
B. GRAPH CLASSIFICATION TASKS
Task performance. Here we provide more results task performance on the graph classification dataset. We repeat each experiment at least 3 times and report the average performances and the standard errors. The results are summarized in Table The results well demonstrate the preeminence of FALCON in searching for good designs under different data distributions.
Search cost. As shown Figure 5, we report the search cost of Random, DARTS, ENAS, GraphNAS, and FALCON on the selected datasets. The time measurements conducted single NVIDIA GeForce 3070 GPU (24G). We see FALCON has a comparable time cost with Random and DARTS, which empirically proves the efficiency of FALCON.
However, as FALCON still needs to sample designs and train them from seratch (i.e., the search cost of FALCON is bounded by the search cost of Random), the computational cost is relatively high in
16

---

Under review as a conference paper at ICLR 2023
Table 8: Test performance (ROC-AUC) on the graph classification datasets.
DHFR-MI MD
COX2-MD
Mutagenicity
MOLT-4
NCI-H23
PTC-MR
P388
Random
59.0.5.2
63.2
17.1
58.6107
58.4
59.9
63.9
BO
20.1
71.6255
78.3+07
58.1
63.6200
58.8117
68.9200
SA
56.0#7.1
67. 33.1
81.1+03
54.8+111
56.7+33
59.4=6.2
74.4
ENAS
53,5=37
57 1.77
75.0+16
61.5+01
61.22111
59.8
68. 18.3
DARTS
55.8=6.3
70.4
74.4+07
61.4+07
63.5
59.3+
70,8
GraphNAS
61.6144.3
63.9
80.2+15
62.1+10
62,4
58.616.7
68 28.2.2255
AutoAttend
63.3400
68.8+0.
79.6+a.
59.5+02
61.8=0.2
57.8+006
74.940.
GASSO
60.9
68.5
75.1
$7.4109
64.7
51.9
71.3
FALCON-G
58,5
67.3
79.8
62.8433
62.2
55.646.9
74.2
FALCON-LP
61,421.5
66.8 43.6
80.2408
58.5 =**
63.724.
55.6220
75.1
FALCON
63.
67.3
81.1
64.4
66.6
60.0
77.0
(cont) PTC-MM
PC-3
SF-295
NCI1
SW-620
UACC257
Yeast
Avg.
$2.5
60.4
55.3
77.5
57.5
61.1
53.3
61.3
61.1
59.7.0.1
60.6
77.3400
63.8409
60.82n0
55.0=00
63.7
1:4.6
69.0#2.
55.3
79.6453
58.2+2.1
64.2.a0.1
53.8
63.4
52,4=29
62.2=211
60.9
77.2+12
64.8222
64.7=20
63.0=1.1
63.0
52.6#t4.1
61.6#00.
62.2
66.3+33.
66.0403
65.7=0.2
61.4#14
63.7
54.8
68.6226
65. 03.0
78.1
61.8+444
61.5=5.1
57.221.
64.6
6..7=112
66.2+0.3
64.2+00
7..3+16
65.5+04
57.1
59.6#08
65.9
63.2
66.0:1188
65.8:004
76.6106
64.7
62.7.002
60.1
64,9
55,2+24
65.1
64.3
80.6108
62.013.0
61.0
57.5
64.7
56.8261
68.1
63.5234
80.4105
65.2
54.5
56.6217
64.7
57.1
71.0
64.4
80.9
66.64221
66.7=2.1
58.1
67.5"
Randem
DARTS
Search Cost (GPU hours)
ENAS GraphMAS
30*
ER-ND
AIDS
OVCAR-8
MCF-7
SM12C
NC1109
Toka1-AhR
Figure 5: Search cost on the selected datasets.
large datasets, e.g., OVCAR-8 and MCF-7. We can potentially alleviate this limitation via integrating dataset sampling to reduce time costs.
ROC-AUC exploration size. Here report the change in task performance graph classifica- tion datasets with the number of explored nodes. In Figure 6, we visualize the results on two graph classification datasets. We that FALCON can approach the best-performing designs quickly as the explored size grows.
B.2 BEST DESIGNS
In Table 9 and Table 10, we summarize the best designs found by FALCON and BRUTEFORCE in each dataset, where the average number of parameters is 137.5k for all the graph classification datasets. Note that we select the best designs according to their performance cases where FALCON surpasses BRUTEFORCE. We highlight the design variables that are different between FALCON and BRUTEFORCE for comparison.
validation sets; thus, there
17

---

Under review as a conference paper at ICLR 2023
NC11
PTC MR
0.675
0.85
0.650
0.80
0.625
ROC-AUC
0.600
0.75
0.575
0.550
0.70
0.525
Bruteforce
0.500
Bruteforce
Number Explored Nodes
Number Explored Nodes
(a)
(b)
Figure 6: Accuracy v.s. the number of explored nodes on two graph classification datasets.
Table Average parameters Best designs in the node classification datasets.
Average Param (k)
Test perfor- mance (%)
Dataset
Best design
FALCON
(0.0, 4, STACK, Swish, True, Mean)
70.36
ogbn-arxiv
44.5
BRUTEFORCE
(0.3, J, 4, 2, SUM, Relu, True, Mean)
70.51
FALCON
(0.0) 1, 6, 1, SUM, Swish, False, Meanj
87.18
Cora
77.8
BRUTEFORCE
(0.0, 1, 4, 1, STACK, Swish, False, Mean)
86.99
FALCON
(0.3, 1, 1, SUM, Prelu, True, Mean)
76.19
Citeseer
289.0
BRUTEFORCE
1, SUM. Prelu. False, Mean)
75.99
FALCON
(0.3, J, 8, I, SUM. Relu, True, Add)
90.04
Pubmed
57.8
BRUTEFORCE
(0.3, J, 8, I, SUM, Relu, True, Add)
90.04
FALCON
(0.0) 4, 1, STACK, Swish, True, Mean)
91.64
AmazonComputerrs
46.2
BRUTEFORCE
(0.0, 4, 2, STACK, Prelu, True, Mean)
91.35
FALCON
(0.0, 3, 4, 2, STACK, Prelu, True, ARMACon))
95,46
Reddit
47.2
BRUTEFORCE
(0.0, 3, 4. 2, STACK, Prelu, True, ARMACon.)
95.46
B.3 BRUTEFORCE'S CONFIDENCE INTERVAL AND VARIANT
To estimate the uncertainty of Bruteforce, we compute the 95% confidence interval Bruteforce using bootstrapping. Moreover, we consider variant of Bruteforce baseline to compare with Bruteforce. Specifically, we train all the designs in the design space for 30 epochs, select the top 10% design, and resume the training until 50 epochs. After that, we choose the top 50% designs to be fully trained and return the best fully trained design based on validation performance. We run Bruteforce-bootstrap on four datasets as demonstrations, We summarize the results in Table 11.
Table 11: Test performances of Bruteforce and its variant.
Cora
CiteSeer
ER-MD
AIDS
Bruteforce (max)
87.0
76.0
83.3
96.0
Confidence interval length
0.2
0.1
0.6
0.0
Bruteforce-bootstrap
87.0
76.4
83.8
95.7
Surprisingly, we found that the performance of Bruteforce and seteeforre-boosstraa are very close. This indicates that Bruteforce (fully trained 5% design) is good surrogate of Bruteforce- bootstrapping (fully trained 5% design, but using bootstrapping selection), and could also well approximate the ground truth performance of the best design.
18

---

Under review as a conference paper at ICLR 2023
Table 10: Best designs in the graph classification datasets.
Test perfor-
Dataset
Best design
mance (%)
FALCON
(0.3, I, 4, 2. SUM. Prelu, True, Add, NoPool)
99.02
AIDS
BRUTEFORCE
(0.0, STACK, Relu, True, Add, SAGPool, 2)
95.97
FALCON
(0.0, 2, 6, SUM, Swish, True, Add, EdgePool, 2)
69.87
COX2-MD
BRUTEFORCE
(0.0, I, 4, STACK, Prelu, True, Max, TopkPool. 2)
65.39
FALCON
(0.3, 2, 4, 2, SUM, Swish, True, Mean, PANPool, 2)
71.33
DHER-MD
BRUTEFORCE
(0.0, 1, 4, CAT, Prelu, True, Mean, SAGPool, 4)
56.22
FALCON
(0.3, 4, 2, CAT, Relu, True, Add, PANPool, 2)
81.67
ER-MD
BRUTEFORCE
(0.0, 3, 5, CAT, Prelu, True, Max, PANPool, 6)
83.33
FALCON
(0.6,
CAT, Swish, True, Add, NoPool)
67.85
MCF-7
BRUTEFORCE
(0.0, 2, 6, SUM, Prelu, True, Add, NoPool)
70.61
FALCON
(0.0, 6, 2, SUM, Prelu, True, Max, NoPool)
69.03
MOLT-4
BRUTEFORCE
(0.0, SUM, Prelu, True, Add, NoPool)
70.30
FALCON
(0.0, 2, 6, CAT, Relu, True, Add, NoPool)
81.73
Mutagenicity
BRUTEFORCE
(0.0,
CAT, Relu, True, Add, PANPool, 2)
81.17
FALCON
(0.0, 2. 4, 2, SUM. Swish. True, Max, EdgePool, 2)
80.13
NCT1
BRUTEFORCE
(0.0. 4, 2. STACK, Prelu, True, Add, EdgePool, 4)
82.81
FALCON
(0.0, 2, 8, 2, STACK, Prelu, True, Max, NoPool)
79.98
NCT109
BRUTEFORCE
(0.0, 3, 6, 2, STACK, Prelu, True, Add, EdgePool, 6)
81.77
FALCON
(0.0, 2, 6, CAT, Swish, True, Add, NoPool)
71.14
NCI-H23
BRUTEFORCE
(0.0, 2, 6, 1, CAT, Swish, True, Add, NoPool)
71.14
FALCON
(0.6, 6, 2, CAT, Swish, True, Add, NoPool)
68.21
OVCAR-8
BRUTEFORCE
(0.6. 1, 2, 3. CAT, Swish, True, Add, NoPool)
67.40
FALCON
(0.0,
CAT, Prelu, True, Add, NoPool)
78.75
P388
BRUTEFORCE
(0.0, 2, 6, 2, CAT, Prelu, True, Add, NoPool)
78.75
FALCON
(0.0, 6, SUM, Relu, True, Max, NoPool)
73.28
PC-3
BRUTEFORCE
(0.0,
SUM, Relu, True, Max, NoPool)
73.28
FALCON
(0.0, 2, 2, STACK, Swish, True, Max, EdgePool, 2)
56.96
PTC-MM
BRUTEFORCE
(0.0,
SUM, Swish, True, Mean, EdgePool, 2)
52.93
FALCON
(0.0, 2, 4, 2. SUM. Swish, True, Mean, TopkPool, 2)
60.56
PTC-MR
BRUTEFORCE
(0.3,
SUM. Relu, True, Add, NoPool)
63.06
FALCON
(0.6, 2, 6, 2, CAT, Swish, True, Add, NoPool)
64.75
SF-295
BRUTEFORCE
(0.0, 3, 8, 3, SUM, Prelu, True, Max, PANPool, 4)
66.47
FALCON
(0.0, 8, 1, CAT, Swish, True, Add, NoPool)
73.34
SNI2C
BRUTEFORCE
(0.0, 1, 8, 3, CAT, Prelu, True, Add, EdgePool, 6)
73.73
FALCON
(0.0, 2, STACK, Prelu, True, Max, NoPool)
SW-620
BRUTEFORCE
(0.0, 2, 4, 2, STACK, Prelu, True, Max, NoPool)
69.26
FALCON
(0.0,
SUM, Prelu, True, Add, EdgePool,
79.10
Tox21-AhR
BRUTEFORCE
(0.0, 6, 2, SUM, Prelu, True, Add, NoPool)
82.02
FALCON
(0.3, 6, CAT, Swish, True, Max, NoPool)
67.94
UACC257
BRUTEFORCE
(0.0, 6, SUM, Prelu, True, Max, NoPool)
70.24
FALCON
(0.6, 8, CAT, Prelu, True, Add, NoPool)
59.60
Yeast
BRUTEFORCE
(0.0, SUM, Swish, True, Add, EdgePool, 2)
60.41
19

---

Under review as a conference paper at ICLR 2023
C EXPERIMENTAL RESULTS ON THE IMAGE TASK
C.I DATASET PRE-PROCESSING
We use the CIFAR-10 (Krizhevsky, 2009) image dataset to show FALCON can work well on other machine learning domains. This dataset consists of 50,000 training images and 10,000 test images. We randomly crop them size 32 32, and conduct random flipping and normalization.
C.2 DESIGN SPACES
Here we use two different design spaces to demonstrate FALCON's ability in searching for both hyper-parameters and architectures on image dataset.
Hyper-parameter Design Space. We consider a broad space of hyper-parameter search, including common Syper-parameters like Batch Size. We train each design using SGD optimizer, which requires weight decay and momentum as hyper-paramettrs. We also use a learning rate (LR) scheduler, which reduces the learning rate when validation performance has stopped improving. Specifically, the scheduler will reduce the learning rate by factor if no improvement is seen for 'patience' number of epochs. also worth mentioning that FALCON is flexible for other sets of pyper-parameter choices determined by the user end.
Table 12: prrper-prramttr design space for image tasks.
Variable
Candidate Values
Momentum (SGD)
[0.5, 0.9, 0.99]
Weight decay
[le-4, 5e-4, le-3, 5e-3]
Batch size
[32, 64, 128, 256]
LR decay factor
[0.1, 0.2, 0.5, 0.8]
LR decay patience
[3, 5, 10]
Architecture Design Space. We construct micro design space for the computational cells. Each cell constitutes of two branches, where enable five selections: separable convolution with kernel size x and 5 x 5, average pooling and max pooling with kernel size 3x3, and identity. In each branch, we have dropout layer, where the dropout ratio is one of {0.0, 0.3, 0.6). and one batch normalization layer. We also use one identity and skip-sum as skip-connection within each branch. After the input data is separately computed in each branch, the outputs are added the final cell output, which is different with the original ENAS paper that searches the computational DAG on the defined nodes.
C.3 EXPERIMENTAL RESULTS
Here we set the exploration size 20 for all the sample-based methods. For hyper-parameter design space, we compare FALCON with baselines that are available for hyper-parameter tuning. Moreover, to accelerate the search process, FALCON explores an unknown design by fine-tuning pretrained model for several epochs based on the selected hyper parameters, instead of training each candidate design from scratch. The results summarized in Table 13.
Table 13: Search results on the hyper- parameeerr design space for CIFAR-11 dataset.
Average Error (%)
Random
4.13
BO
4.95
SA
4.04
FALCON
3.80
20

---

Under review as a conference paper at ICLR 2023
CiteSeer
Tox21-AhR
5
E3313
3-
4-
134
5-
Random Start Nodes
Random Start Nodes
(a)
(h)
Figure 7: Sensitivity of FALCON's paper-paramtters.
For the architecture design space. For ENAS and DARTS, the learning rate is 0.01, and the maximum training epoch is 300. We repeat each experiment three times and summarize the results in Table 14.
Table 14: Search results on the architecture design space for CIFAR-10 dataset.
Average Error (%)
Search Cost (GPU days)
Random
10.43
0.64
BO
9.83
0.67
SA
10.16
0.62
ENAS-micrs
9.20
0.77
DARTS-micro
8.97
0.89
FALCON
8.87
0.81
D SENSITIVITY ANALYSIS
We analyze the sensitivity of the classification task, CiteSeer, and a graph classification task, ThRR.-AhR. Specifically, we study the influence of the number of random starting nodes and the candidate scale resulting from an explored node (i.e., how many hop neighbors of an explored node are to be included in the candidate set).
in the search strategy of FALCON, using node
As shown in Figure 7, find that FALCON outperforms the best AutoML baselines with large range of syrem--arameters. Specifically, 73% and 97% hyper-parameter combinations of FALCON rank best among the baselines in CiteSeer and
respectively.
Moreover, we discover an interesting insight about the size of receptive field, i.e., the number of design candidates, during the search process of FALCON. According the construction of design subgraph, the receptive field size is O(rh²), where is the number of random start nodes, the number of neighbor hops, and d is the average node degree. We find that the performance of design searched by FALCON increases with the receptive field's size until it reaches a certain scale.
Such patterns have been widely observed in multiple datasets. While the receptive field the design subgraph should contain sufficient candidates for sampling good ones, it should also prune inferior design space, which doesn't provide insights navigating the best-performing designs. Thus, the size of the receptive field may be crucial factor influencing the search quality of FALCON.
21


Reviews:
[{"review_id": "y6mvm-tZHXT", "reviewer": "Reviewer_1F3M", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, a new method of sample-based NAS is proposed, Falcon. The key idea is to represent the model design space as a graph, where nodes correspond to separate designs and an edge corresponds to a minimal change in one of the design parameters. The problem of architecture search is thus translated to a black-box optimization task over the design graph. Falcon builds a GNN meta-model, which approximates the values in the nodes and use it in a search strategy.  ", "summary_of_the_review": "I think, this paper does not appropriately place itself in the context of state-of-the-art and classical algorithms relevant to the problem, and needs a major revision.", "strength_and_weaknesses": "+ The idea of representing the design space as a graph looks novel for NAS\n\n- Related work seems incomplete. In particular, the claim “Previous works generally consider each design choice as isolated from other designs” seems too strong. I think, there should be an overview of previous papers related to meta-models over design spaces. Apart from NER, there should be also literature on design spaces in the context of classical ML. In this context, I am not convinced that the scientific problem is well-formulated. When we look at the empirical setting of the work, we see that the design parameters enable a simpler approach without any design graphs: each parameter can be quantified as a separate dimension of a joint design space, and the problem becomes a black-box optimization on the obtained grid.\n- The motivation behind the proposed Falcon is questionable. Why not using Bayesian approaches? Gaussian processes?\n- Some parts of the paper lacks of details. In particular, GNN meta-model and its training pipeline is not fully described\n- The pipeline of the empirical part is not reproducible based solely on the text of the paper. The whole system is complex, it has many hyperparameters and itself design choices. This choice is not transparent, and I cannot be sure that the obtained empirical improvements are not an attribute of these choices.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "nEbi656Qhdq", "reviewer": "Reviewer_ZhyY", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper describes a new method for AutoML (FALCON) that comprises a online search in the space of designs that is guided by a GNN model (\"meta-GNN\") that is trained along the way. The GNN comprises a task-agnostic component that tries to capture similarity between designs using various features and relations, as well as a task-specific component that performs label propagation based on a set of evaluated designs, and is trained using a pairwise rank loss. Experiments are presented on a variety of graph-based AutoML tasks, and results on an image classification task are included in the Appendix. ", "summary_of_the_review": "This paper contains some interesting work and ideas and the results are promising, but needs a rewrite to clean up some of the terminology used and present the details of the algorithm in a clearer way. ", "strength_and_weaknesses": "Strengths:\n- The proposed approach is novel to the best of my knowledge. \n- The experimental results show that FALCON out-performs simple baselines like random search, simulated annealing and Bayesian optimization, as well as a variety of generic AutoML approaches (DARTS, ENAS) as well as AutoML methods designed specifically for graph tasks (AutoAttend, GraphNAS, GASSO).\n- The ablation studies show that (a) the search is significantly improved by using the meta-GNN and (b) the meta-GNN is significantly improved by using the task-specific component. \n\nWeaknesses:\n- There is some confusing terminology in this paper. It is stated that FALCON \"learns the inductive biases\" that are present in the problem using a \"meta-model\". The examples given for the inductive biases seem to be simply relations that exist between various hyper-parameters and design choices in the optimization domain. For instance, the example given \"using a large number of layers without batch normalization degrades the performance” seems like a simple relation that can be learned by any model-based AutoML framework (even BO can in principle learn this). Why is the meta-GNN a \"meta\" model - does it learn anything across different tasks or datasets? The task agnostic component is not really learned across different tasks as far as I can tell. It is more like a component that captures some domain-specific knowledge about the design space. The meta-learning terminology is confusing to the reader. \n- It is stated that other sample-based AutoML methods require \"thousands of models from scratch\". This may be somewhat true, but there also exist efficient methods like Successive Halving and Hyperband which train many models but using a small fraction of the training data in order to eliminate designs that are not promising very quickly. Why were these methods not discussed or used as baselines?\n- The main algorithm is not described in sufficient detail to allow the reader to follow what is going on. Algorithm 1 is presented with little-to-no discussion of any of the subroutines that are present. Figure 1(b) is not sufficiently explained in the text. Important details like the design distance are offloaded to the appendix with little discussion. \n\nOther questions:\n- Section 4.2: what are the \"one-shot methods\"?\n- Table 1 and Table 2: what statistical test was used to compute p-values + why are the p-values missing from some columns?\n- Section 3.2: can you please elaborate on how the \"instance-wise performances\" are measured? In Figure 2 they seem to be binary vectors. What exactly do these vectors represent?\n- Section 2: what is a \"computational cell\"?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "arjvmDty4s", "reviewer": "Reviewer_Wvmw", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed an efficient automated model design algorithm called FALCON. FALCON views the overall design space as a design graph. Each node in the design graph represents an individual modeling choice, which includes the model architecture and the optimization hyper-parameters such as learning rate and weight decay. The nodes are connected following the predefined rules. The key intuition of FALCON is that the connected nodes tend to have the same performance. Thus, FALCON trains a performance model by applying GCN on the design graph, and use the performance model to explore new modeling choices and predict their performances. Results show that FALCON achieves better performance than previous auto-search algorithms, and the overall GPU-cost is not very huge. The author also conducted ablation analysis to show that modeling design relations is important for performance improvement.\n", "summary_of_the_review": "I voted for weak reject due to concerns that the author is overselling their method. In addition, the smoothness inductive bias has not been discussed in depth.", "strength_and_weaknesses": "Strength\n\n- The performance of FALCON on GNN tasks is strong.\n- It is reasonable to view the design space as a graph. However, the final performance might be sensitive to the rules that FALCON pick for constructing the graphs.\n\n\nWeaknesses\n\n- The author only conducted experiments on GNN tasks. Experiments on Image tasks are just conducted on a CIFAR-10 while previous papers usually report the performance on NAS-Bench. Actually, the author claimed in the end of Introduction (Page 2) that \"Without loss of generality, we mainly focus on AutoML for graph representation learning in this paper.\". I do not think that the author can claim \"Without loss of generality\" by just conducting experiments on GNNs. The method is very general and the author claimed it to be an AutoML framework. Experiments on a more diverse collection of tasks (such as more NAS-Bench, or NLP tasks) is necessary.\n- The method is largely based on inductive bias of smoothness. However, I think smoothness won't hold in the general scenario (e.g., certain learning rates are preferred for certain network architectures). Thus, the author may need to elaborate more on the smoothness hypothesis.\n- I think the method is also related to Population-based Training (PBT), and the author needs to mention the connection / difference between FALCON and PBT.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1gp5PboKxJ7", "reviewer": "Reviewer_xdfg", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a sample-based method named FALCON, to search for the optimal model design. The key idea is to build a design graph over the design space of the architectures and hyperparameter choices. A meta-model is built to capture the relation between the design graph and model performance. The method uses GNN to learn and predict the performance of a specific design giving the corresponding nodes in the design graph. The GNN model includes both a task-specific module and a task-agnostic module.", "summary_of_the_review": "This work proposes a sample-based method named FALCON, to search for the optimal model design. The key idea is to build a design graph over the design space of the architectures and hyperparameter choices. A meta-model is built to capture the relation between the design graph and model performance. The method uses GNN to learn and predict the performance of a specific design giving the corresponding nodes in the design graph. The GNN model includes both a task-specific module and a task-agnostic module.\n\nThe idea of design of modeling the search space as a design graph is quite novel. The method is also carefully designed with many thoughtful insights. For example, the construction of the design graph, the design distance, and the design of the subgraph. The task-agnostic and task-specific modules are also designed with caution. The empirical evaluation is quite comprehensive and solid, showing improved performance compared to state-of-the-art baselines.\n\nBut I do have concerns about the applicability of the proposed method to more general search spaces (see concerns and questions in the Strength And Weaknesses section)", "strength_and_weaknesses": "**Strength**\n1. The idea of design of modeling the search space as a design graph is quite novel. The method is also carefully designed with many thoughtful insights. For example, the construction of the design graph, the design distance, and the design of the subgraph. The task-agnostic and task-specific modules are also designed with caution.\n\n2. The empirical evaluation is quite comprehensive and solid, showing improved performance compared to state-of-the-art baselines.\n\n**Concerns and questions** \n\nThe scope of applicability of the proposed method is not clearly stated. It seems to me this method only works for cases where the choices are all discrete. What if there are hyperparameters with continuous domains? For example, the dropout ratio, momentum, and weight decay are all hyperparameters with continuous values. Is FALCON directly applicable to handling a search space with those continuous hyperparameters? The current experiment only takes several discrete values. Does it mean one has to discretize the continuous hyperparameters? If so, how to discrete them? In Table 12, the candidate values for the momentum of SGD are 0.5, 0.9, 0.99; the LR decay factor is 0.1, 0.2, 0.5, 0.8. I do not see any pattern in this discretization step. This specialized design space raises my doubts about how general this method can be. If there is a newly developed model needs to be tuned, how should one correctly configure those carefully designed components?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject