Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposed a predictor based on the Transformer architecture to predict the performance of neural architectures in NAS.
It proposed to use Laplacian matrix based positional encoding strategy to better represent the topology information to model the neural architectures.
It also proposed a model-agnostic self-evolution framework that can fully utilize temporal information as guidance.
main_review: Predictor plays an important role in NAS and predictor-based NAS is straight and simple for deploying in practice.
Besides previous LSTM/GNN based predictors, the authors proposed to use the advanced Transformer architecture for modeling the neural network architectures to predict the performance (e.g., accuracy).
Given the experiment results and the announced "... 2nd among all teams in one recent international NAS challenge", the methods demonstrate its effectiveness.

Strengths:
1. The proposed positional encoding seems to bring the benefit, compared to [1] where the authors showed that directly applying Transformer to the task gets bad performance. Does the gain come from the positional encoding?

Weaknesses and questions:
1. There lacks an introduction of Laplacian matrix but directly uses it. A paper should be self-contained.
2. The motivation is not strong. The authors stated that "... the transformer architecture ... outperformed many SOTA models ... motivates us". This sounds like "A tool is powerful, then I try the tool on my task, and it works well! Then I publish a paper". However, it lacks of analysis why Transformer works well on this task, which would bring more insights to the community.
3. Section 3.3 needs to be polished more on writing.\
1 In Eqn. (8), The e^{(t)} is proposed without further explanation. What is it? Why is it needed? What is the motivation of proposing it?\
2 In Eqn. (8), are f_\theta(v_j) and \hat{y}_j^{(t)} the same thing since they both are the predictions of v_j by the predictor.\
3 In line 177-178, if the y_j^{true} is "user-unkonwn", then how do you compute e^{(t)} in Enq. (8)?\
4 Why this SE framework can help to improve, how does it help? Similar to 2, please DO NOT just show me what you have done and achieved, but also show me why and how you manage to do these. 

I would consider increasing the rating based on the authors' response.

Reference:
[1] Luo, et al. "Neural architecture search with gbdt." arXiv preprint arXiv:2007.04785 (2020). https://arxiv.org/abs/2007.04785
limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposed a transformer-based NAS performance predictor, accompanying with a Laplacian matrix based positional encoding to represent spatial topology information and a self-evolution mechanism to take advantage of the temporal information such as historical evaluations.
main_review: This paper is well-organized with solid experiments in both the main paper and supplementary material.

Strength:
+ Using Laplacian matrix  to model spatial topology information is somewhat novel 
+ Good performance on NAS-Bench 101, 201

Weakness: 
There are some issues needed to be addressed before accepted.
- The claimed contributions (positional embedding and self-evolution) are closely related to BONAS from NIPS 2020. What is the difference and advantage over BONAS? 
- Is transformer necessary? How about replacing transformer with GCN or MLP?
- In the ImageNet experiment, the performance gap between TNASP-A,B,C is big while the flops difference is small. What is the reason?  What is the variance between different runs?
- Potential double blind violation: "Our proposed method ranked 2nd among all teams in one recent international NAS challenge." This is a near miss. I will warn the author not to play this trick next time, if not deck rejected this time.
limitations_and_societal_impact: Please justify the weakness to improve the paper.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes a transformer-based NAS performance predictor to better encoding the spatial topology information and introduces a self-evolution framework to further improve the performance of the predictor. The proposed method achieves SOTA results on three benchmarks. 
main_review: Pros:

(1)	A transformer-based predictor and a self-evolution framework are proposed, and the experiments on different benchmarks verify the effectiveness of the proposed method. 

(2)	The self-evolution framework makes full use of the dataset by leveraging the validation loss. When the size of the training dataset is small, it improves the generalization of the model.

Cons:

(1)	Though the author said that the self-evolution framework is practical in machine learning competitions, the test dataset in competitions is usually not provided to users to store the predictions for validation loss, which limits the usage of the proposed framework. 

(2)	The detailed experiment setting of the transformer-based predictor is not reported. 

(3)	The author said the core factor of the proposed predictor is the better encoding ability for spatial topology information. However, it is not convincing that Laplacian matrix-based positional encoding does represent topology information. As a comparison, what is the performance of transformer-based predictor associated with random initialized positional embedding? Similarly, the Laplacian matrix is also used by GCN-based predictor which strictly limits the direction of information aggregation by Laplacian matrix. It would be better to provide the experiment results of GCN-based methods.

(4)	The main idea of self-evolution framework is similar to the pseudo label technique that SemiNAS adopts. What is the performance of SemiNAS on these benchmarks? 

(5)	Why is lambda in Eq. (10) must be greater than or equal to 0?

(6)	How does the proportion of training set and test set affect the performance of self-evolution framework? 

(7)   There are some recent/concurrent works on NAS missing in the related works, such as "CATE: Computation-aware Neural Architecture Encoding with Transformers", "Vision Transformer Architecture Search" and "AutoFormer: Searching Transformers for Visual Recognition". It is suggested to add them into comparison and discussion, especially for CATE. Recent papers on NeurIPS'20 are also not included in the comparison.

Typos:
(1)	In line 127, acrylic may be acyclic.
(2)	In line 128, N may denote the number of nodes, not the connections (edges).
(3)	In line 4 of Algorithm 1, the new constraint may be added into Eq. (8), not Eq. (10)

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work leverages a transformer-based predictor network to approximate the accuracies of unseen architectures from a search space, for the purpose of NAS. The authors further proposed a ADMM-based method to accelerate the convergence of the predictor training. Experiments are conducted on NAS benchmarks and DARTS spaces.
main_review: 1. I think the idea or graph or transformer-based predictor is reasonable. However, I do not think it is novel enough, given there are already many graph-based predictor works for NAS. I further named a few below that are not cited and compared by the authors.
2. The description from Eq. 8 to Eq. 13 is clear to me, but the flow char in Fig 2 is a bit confusing and less informative.
3. The self-evolution (SE) method further introduces extra hyper-parameters, the three $\eta$s in Eq.11~13. From Table 1 it seems the gain from SE is quite marginal. I am concerned about the cost of extra hyper-param tuning vs. this performance gain.


[1] Graph HyperNetworks for Neural Architecture Search

[2] D-VAE: A Variational Autoencoder for Directed Acyclic Graphs

[3] Neural Graph Embedding for Neural Architecture Search

[4] Fitting the Search Space of Weight-sharing NAS with Graph Convolutional Networks
limitations_and_societal_impact: My main concern is the limited novelty of transformer-based NAS predictor, and the performance gain from self-evolution given the extra cost of hyper-parameter tuning.
review_title: null

Meta-review: The authors apply transformer with Laplacian matrix positional encoding as a predictor in predictor based neural architecture search. They also introduce a self-evolution framework to incorporate new data points to improve the performance. They test the model on standard NAS and DARTS benchmarks (collected set of architectures with their performances) and achieve some of the best results. 

Overall the paper is sound. The main issue from the reviewers was if the paper is sufficiently novel to be accepted to neurips and also partially the small size of the NAS and DARTS benchmarks as an evaluation for the neural architecture search. There were also a lot of question to relations to other works and reference that were missing in the paper, however the authors have clarified a lot of the differences from the previous works in the rebuttal.
Decision: Accept (Poster)