Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper propose a NAS algorithm named GraphPNAS, which uses a graph generator instead of classical RNN generator to sample architectures from the space. RL algorithm is used to update the generator. Experiments on several different search space show the effectiveness of the method.
strength_and_weaknesses: Pros:
1.	The paper is clear written and easy to follow.
2.	The Figure3 show the architecture space model by the proposed generator is better than classical RNN controller.
3.	The authors use several different search spaces on different datasets to demonstrate model effectiveness.

Cons:
1.	The literature review is not sufficient. There are lots of papers use graph generators for NAS[1~3]. So this point is not new. The authors should explain how their model is more brilliant than these methods.
2.	The graph generator model designs are borrowed from previous works. There are few contributions in this part.
3.	More NAS baselines should be compared in the experiment part. The authors should also show why the architecture space model by the proposed graph generator is better than previous works.


[1]Zhang, Miao, et al. "Differentiable neural architecture search in equivalent space with exploration enhancement." Advances in Neural Information Processing Systems 33 (2020): 13341-13351.
[2]Shi, Han, et al. "Bridging the gap between sample-based and one-shot neural architecture search with bonas." Advances in Neural Information Processing Systems 33 (2020): 1808-1819.
[3]Ru, Robin, Pedro Esperanca, and Fabio Maria Carlucci. "Neural architecture generator optimization." Advances in Neural Information Processing Systems 33 (2020): 12057-12069.

clarity,_quality,_novelty_and_reproducibility: Clarity: good
Quality: fair
Novelty: low
Reproducibility: good

summary_of_the_review: This paper propose a NAS algorithm named GraphPNAS, which uses a graph generator instead of classical RNN generator to sample architectures from the space. However, the contributions are limited on model design. Furthermore, extend explanations and experiments are needed to support the method.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a GNN-based neural architecture method that can model the distribution of well-performing architectures. With a reinforcement learning formulation, the paper claims that the learned GNN-generator is more flexible and efficient than existing work.
strength_and_weaknesses: ### Strengths
1. The paper proposes a GNN-based graph generator for NAS, which is an interesting exploration.
2. The paper applies the method to extensive experiments and shows state-of-the-art results on multiple benchmarks.
3. The paper also introduces a larger search space and proves the efficiency of the method against the existing work.


### Weaknesses
1. More discussions of how the neural architecture is encoded in a GNN model would be appreciated. For example, figure 4 in the appendix should be in the main paper and illustrated.
2. The evaluation of the method is applied to smaller datasets (Tiny-Imagenet, which is not common in the previous work. It would be more interesting to see how the method performs on large-scale real applications.
3. It would be interesting to see different constraints for the NAS method, such as latency constraints of the neural architecture.
clarity,_quality,_novelty_and_reproducibility: The writing is clear, the quality is good, and it is an interesting application of GNN.
summary_of_the_review: Please refer to above.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposed to use GNN as the architecture controller in reinforcement learning based NAS.
strength_and_weaknesses: Strength:
1. Some novelty as the first paper to propose using GNN-based generative modeling for generating architectures in NAS.
2. Improved results compared to an RNN-based generator and several baselines that are quite old though.

Weakness:
1. I feel the cited papers and compared baselines are quite out-of-date. Cited works are mostly before 2020. While I am not well literatured in NAS, I do know there were multiple NAS methods proposed in 2021/2022. Maybe most of them are differentiable NAS methods but some comparisons should be made.
2. Can evaluation be performed on the larger ImageNet and the newer NAS-bench-201 datasets? I feel results on the smaller and older datasets, without many more recent baselines, are less convincing.
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written. Not reproducible as code is not provided and it is not trivial to implement the proposed method.
summary_of_the_review: An interesting paper that first proposed using GNN as architecture controller, but need to be benchmarked with more recent methods and on more datasets.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a probabilistic graph generator that models the distribution over good architectures using GNNs. This converts the neural architectures to a learnable computational graph. Extensive experiments indicate the effectiveness and efficiency of the proposed method. This paper is generally well-written, and the idea is interesting. 
strength_and_weaknesses: Strength:
1. The idea of considering the NAS as GNN is interesting.
2. The paper is well-organized and easy to follow. 


Weaknesses:
1. The experiment does not show the efficiency of the method. Would you consider adding computational complexity analysis or time consumption?
2. I don't quite understand why this method is less effective than other methods when the number of architectures evaluation is small. It suddenly got better as the number increased （Figure 2). Can you explain?
3. In addition, the authors do not conduct experiments on the most common DARTS space, making it hard to conduct comparisons with most existing SOTA NAS methods. I understand that the DARTS space is an old benchmark and not easy to get significant improvement, while I am still interested in the performance of the proposed method in this space.
clarity,_quality,_novelty_and_reproducibility: This paper proposed a clear framework, and the idea is easy to follow. The extensive experimental results on several benchmark datasets demonstrated the effectiveness of the proposed method. However, the lacking of open-source codes may limit reproducibility. 
summary_of_the_review: This paper is generally well-written, and the idea is novel. I'd like to increase the score if the authors can resolve the above concerns.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposed a NAS algorithm named GraphPNAS, which uses a probabilistic graph generator as the architecture controller in reinforcement learning based NAS. RL algorithm is used to update the generator. Experiments on several different search space show the effectiveness of the method. The paper claims that the learned GNN-generator is more flexible and efficient than existing work. The reviewers raised concerns on the added values of the proposed framework over existing graph generators for NAS, differentiations over similar graph generator model designs, more up-to-date NAS baselines, larger experimental datasets. The authors provided point to point answers and added experiment results as permitted by the time of the response period. However, there still seems to be a lack of enough enthusiasm among the reviewers after the author responses.
justification_for_why_not_higher_score: The authors provided point to point answers and added experiment results as permitted by the time of the response period. However, there still seems to be a lack of enough enthusiasm among the reviewers after the author responses.
Decision: Reject