Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this paper, a new method of sample-based NAS is proposed, Falcon. The key idea is to represent the model design space as a graph, where nodes correspond to separate designs and an edge corresponds to a minimal change in one of the design parameters. The problem of architecture search is thus translated to a black-box optimization task over the design graph. Falcon builds a GNN meta-model, which approximates the values in the nodes and use it in a search strategy.  
strength_and_weaknesses: + The idea of representing the design space as a graph looks novel for NAS

- Related work seems incomplete. In particular, the claim “Previous works generally consider each design choice as isolated from other designs” seems too strong. I think, there should be an overview of previous papers related to meta-models over design spaces. Apart from NER, there should be also literature on design spaces in the context of classical ML. In this context, I am not convinced that the scientific problem is well-formulated. When we look at the empirical setting of the work, we see that the design parameters enable a simpler approach without any design graphs: each parameter can be quantified as a separate dimension of a joint design space, and the problem becomes a black-box optimization on the obtained grid.
- The motivation behind the proposed Falcon is questionable. Why not using Bayesian approaches? Gaussian processes?
- Some parts of the paper lacks of details. In particular, GNN meta-model and its training pipeline is not fully described
- The pipeline of the empirical part is not reproducible based solely on the text of the paper. The whole system is complex, it has many hyperparameters and itself design choices. This choice is not transparent, and I cannot be sure that the obtained empirical improvements are not an attribute of these choices.
clarity,_quality,_novelty_and_reproducibility: The novelty of the main idea seems questionable. When we look at the empirical setting of the work, we see that the design parameters enable a simple approach without any design graphs: each parameter can be quantified as a separate dimension of a joint design space, and the problem becomes a black-box optimization on the obtained grid. Why this scenario is not addressed in the paper?

Some parts of the paper lacks of details. In particular, GNN meta-model and its training pipeline is not fully described The pipeline of the empirical part is not reproducible based solely on the text of the paper. The whole system is complex, it has many hyperparameters and itself design choices. This choice is not transparent, and I cannot be sure that the obtained empirical improvements are not an attribute of these choices.
summary_of_the_review: I think, this paper does not appropriately place itself in the context of state-of-the-art and classical algorithms relevant to the problem, and needs a major revision.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work proposes a sample-based method named FALCON, to search for the optimal model design. The key idea is to build a design graph over the design space of the architectures and hyperparameter choices. A meta-model is built to capture the relation between the design graph and model performance. The method uses GNN to learn and predict the performance of a specific design giving the corresponding nodes in the design graph. The GNN model includes both a task-specific module and a task-agnostic module.
strength_and_weaknesses: **Strength**
1. The idea of design of modeling the search space as a design graph is quite novel. The method is also carefully designed with many thoughtful insights. For example, the construction of the design graph, the design distance, and the design of the subgraph. The task-agnostic and task-specific modules are also designed with caution.

2. The empirical evaluation is quite comprehensive and solid, showing improved performance compared to state-of-the-art baselines.

**Concerns and questions** 

The scope of applicability of the proposed method is not clearly stated. It seems to me this method only works for cases where the choices are all discrete. What if there are hyperparameters with continuous domains? For example, the dropout ratio, momentum, and weight decay are all hyperparameters with continuous values. Is FALCON directly applicable to handling a search space with those continuous hyperparameters? The current experiment only takes several discrete values. Does it mean one has to discretize the continuous hyperparameters? If so, how to discrete them? In Table 12, the candidate values for the momentum of SGD are 0.5, 0.9, 0.99; the LR decay factor is 0.1, 0.2, 0.5, 0.8. I do not see any pattern in this discretization step. This specialized design space raises my doubts about how general this method can be. If there is a newly developed model needs to be tuned, how should one correctly configure those carefully designed components?
clarity,_quality,_novelty_and_reproducibility: The paper is very well-written. The overall quality of this work is very high.  
summary_of_the_review: This work proposes a sample-based method named FALCON, to search for the optimal model design. The key idea is to build a design graph over the design space of the architectures and hyperparameter choices. A meta-model is built to capture the relation between the design graph and model performance. The method uses GNN to learn and predict the performance of a specific design giving the corresponding nodes in the design graph. The GNN model includes both a task-specific module and a task-agnostic module.

The idea of design of modeling the search space as a design graph is quite novel. The method is also carefully designed with many thoughtful insights. For example, the construction of the design graph, the design distance, and the design of the subgraph. The task-agnostic and task-specific modules are also designed with caution. The empirical evaluation is quite comprehensive and solid, showing improved performance compared to state-of-the-art baselines.

But I do have concerns about the applicability of the proposed method to more general search spaces (see concerns and questions in the Strength And Weaknesses section)
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposed an efficient automated model design algorithm called FALCON. FALCON views the overall design space as a design graph. Each node in the design graph represents an individual modeling choice, which includes the model architecture and the optimization hyper-parameters such as learning rate and weight decay. The nodes are connected following the predefined rules. The key intuition of FALCON is that the connected nodes tend to have the same performance. Thus, FALCON trains a performance model by applying GCN on the design graph, and use the performance model to explore new modeling choices and predict their performances. Results show that FALCON achieves better performance than previous auto-search algorithms, and the overall GPU-cost is not very huge. The author also conducted ablation analysis to show that modeling design relations is important for performance improvement.

strength_and_weaknesses: Strength

- The performance of FALCON on GNN tasks is strong.
- It is reasonable to view the design space as a graph. However, the final performance might be sensitive to the rules that FALCON pick for constructing the graphs.


Weaknesses

- The author only conducted experiments on GNN tasks. Experiments on Image tasks are just conducted on a CIFAR-10 while previous papers usually report the performance on NAS-Bench. Actually, the author claimed in the end of Introduction (Page 2) that "Without loss of generality, we mainly focus on AutoML for graph representation learning in this paper.". I do not think that the author can claim "Without loss of generality" by just conducting experiments on GNNs. The method is very general and the author claimed it to be an AutoML framework. Experiments on a more diverse collection of tasks (such as more NAS-Bench, or NLP tasks) is necessary.
- The method is largely based on inductive bias of smoothness. However, I think smoothness won't hold in the general scenario (e.g., certain learning rates are preferred for certain network architectures). Thus, the author may need to elaborate more on the smoothness hypothesis.
- I think the method is also related to Population-based Training (PBT), and the author needs to mention the connection / difference between FALCON and PBT.

clarity,_quality,_novelty_and_reproducibility: - Clarity: The paper is clearly written.

- Quality: The technical quality is good and the author also attached the source code.

- Novelty: The novelty of the ranking loss (equation 2) is marginal because using paired data for training a ranking model has been studied in "[NeurIPS2020] BRP-NAS: Prediction-based NAS using GCNs". However, I think using a design graph to accelerate the model choice search for GNN is novel.

- Reproducibility: The author has attached the source code in https://anonymous.4open.science/r/Falcon . The reviewer hasn't run the source code but believe it should be largely reproducible.

summary_of_the_review: I voted for weak reject due to concerns that the author is overselling their method. In addition, the smoothness inductive bias has not been discussed in depth.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper describes a new method for AutoML (FALCON) that comprises a online search in the space of designs that is guided by a GNN model ("meta-GNN") that is trained along the way. The GNN comprises a task-agnostic component that tries to capture similarity between designs using various features and relations, as well as a task-specific component that performs label propagation based on a set of evaluated designs, and is trained using a pairwise rank loss. Experiments are presented on a variety of graph-based AutoML tasks, and results on an image classification task are included in the Appendix. 
strength_and_weaknesses: Strengths:
- The proposed approach is novel to the best of my knowledge. 
- The experimental results show that FALCON out-performs simple baselines like random search, simulated annealing and Bayesian optimization, as well as a variety of generic AutoML approaches (DARTS, ENAS) as well as AutoML methods designed specifically for graph tasks (AutoAttend, GraphNAS, GASSO).
- The ablation studies show that (a) the search is significantly improved by using the meta-GNN and (b) the meta-GNN is significantly improved by using the task-specific component. 

Weaknesses:
- There is some confusing terminology in this paper. It is stated that FALCON "learns the inductive biases" that are present in the problem using a "meta-model". The examples given for the inductive biases seem to be simply relations that exist between various hyper-parameters and design choices in the optimization domain. For instance, the example given "using a large number of layers without batch normalization degrades the performance” seems like a simple relation that can be learned by any model-based AutoML framework (even BO can in principle learn this). Why is the meta-GNN a "meta" model - does it learn anything across different tasks or datasets? The task agnostic component is not really learned across different tasks as far as I can tell. It is more like a component that captures some domain-specific knowledge about the design space. The meta-learning terminology is confusing to the reader. 
- It is stated that other sample-based AutoML methods require "thousands of models from scratch". This may be somewhat true, but there also exist efficient methods like Successive Halving and Hyperband which train many models but using a small fraction of the training data in order to eliminate designs that are not promising very quickly. Why were these methods not discussed or used as baselines?
- The main algorithm is not described in sufficient detail to allow the reader to follow what is going on. Algorithm 1 is presented with little-to-no discussion of any of the subroutines that are present. Figure 1(b) is not sufficiently explained in the text. Important details like the design distance are offloaded to the appendix with little discussion. 

Other questions:
- Section 4.2: what are the "one-shot methods"?
- Table 1 and Table 2: what statistical test was used to compute p-values + why are the p-values missing from some columns?
- Section 3.2: can you please elaborate on how the "instance-wise performances" are measured? In Figure 2 they seem to be binary vectors. What exactly do these vectors represent?
- Section 2: what is a "computational cell"?

clarity,_quality,_novelty_and_reproducibility: The work in this paper seems novel and code has been provided to enable the reader to reproduce the results (although I have not tried to run it). However, the paper is not written to a high standard and many details are unclear. 
summary_of_the_review: This paper contains some interesting work and ideas and the results are promising, but needs a rewrite to clean up some of the terminology used and present the details of the algorithm in a clearer way.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents an approach for AutoML in the domain of optimizing hyperparameters for graph neural networks.  The main idea is to use a graph structure to represent the space of hyperparameters such that a GNN model can be used to exploit the structure of this space and make the search more efficient.

This paper is a bit below the bar of acceptance for a few reasons, but mostly:
1. The applicability and the empirical results presented in this paper is a bit limited, it only considers hyperparameters with small numbers of discrete choices, and with a heavy focus on the narrower GNN domain.  This restriction hurts the significance of this paper. The approach itself is general, so there’s no real reason for the authors to not do more in the more general setting, otherwise the audience needs to be convinced that the graph setting is somewhat special and people should care about this speciality.
2. Clarity of the approach can be improved, with multiple reviewers confused about the presentation.
justification_for_why_not_higher_score: Limited applicability of the approach, thus limited significance.

Clarity issues in the presentation.
justification_for_why_not_lower_score: Can't be lower.
Decision: Reject