Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The authors consider encoding schemes for neural architecture search (NAS). While current encodings model the data flow through the architecture operations by encoding  each operation, the current authors propose a new encoding based on the idea that different operations need different encodings, since each operation has its own separate weights that are learned throughout architecture training. Specifically, they perform multiple forward and backward passes on the architecture and output the trained embeddings as the encoding. They also give symmetry-breaking techniques (similar idea as random initialization). They test their technique on NAS-Bench-101, 201, 301, and ENAS.
strengths_and_weaknesses: ### Strengths

Intuitive and well-performing. The TA-GATES intuition, i.e., operations should not be encoded the same way because they have trainable parameters, makes sense, and TA-GATES is an intuitive solution to this. The symmetry-breaking technique also makes sense, and overall these techniques are shown to achieve strong performance on four benchmarks.

They released code with a detailed README, so reproducibility is high.

Good ablations: ablation for number of time steps of TA-GATES, and ablation for symmetry-breaking technique.

The anytime performance prediction is a very interesting application of TA-GATES.

### Weaknesses

No NAS experiments. All of the experiments compute rank correlation or precision for performance prediction. To show the usefulness of TA-GATES in NAS, the authors should run full NAS experiments. 

Similarly, the method is only compared to four other performance prediction models: MLP, LSTM, GCN, and GATES. But, the authors could include even more. For example, there are other encoding-based works such as
“NASBOWL”, “Contrastive Embeddings for Neural Architectures”, “Does Unsupervised Architecture Representation Learning Help Neural Architecture Search?”, “CATE”, and. Some of these works learn architecture embeddings, which is similar in spirit to TA-GATES, so it is important to discuss and compare to them.

The specific TA-GATES approach seems to be specific for each search space (based on all of the details in Section B.3) so it would need to be tuned for each new search space.

Repeatedly refer to GATES and D-VAE as state-of-the-art techniques. But these have been out for more than two years. There are other methods now, such as the ones listed in the second point above.

The runtime of their method should be included in the main body, as well as total CPU and/or GPU usage.

No discussion on limitations or societal impact (see below).
questions: Can the authors show the usefulness of TA-GATES in NAS by running full NAS experiments? Can the authors compare to more encoding-based methods such as the examples above?

Can the authors include runtimes, CPU/GPU information, limitations/societal impact?
limitations: The authors did not discuss societal impact of their work, and I also did not see a discussion on limitations. Remember that the NeurIPS page limit was increased from 8 to 9 pages to give space for the questions in the author checklist, so these sections should be taken more seriously.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper recommends a new GCN encoder based performance predictor built on top of the GATES encoder/predictor. While most encoders assign the same embedding for a given operation irrespective of their position in the architecture, TA-GATES proposes a predictor that captures the training process of the architecture and assigns different embeddings for a given operation.

Given an adjacency matrix representation of the architecture $a$ and the input information E, for each time step, it computes the information flow of $a$ through a GCN similar to the GATES paper. This simulates the forward propagation of NN training. Then this information is transformed by passing through an MLP and  the resulting output along with $a^{T}$ is in turn passed through the GCN with different weight parameters for information propagation. To simulate backpropagation of the training, they compute derivates and use chaining rule in their second information propagation. The output of the backward propagation from the final timestep is the architecture embedding. Similar to GATES, they have a pairwise ranking loss and MSE loss. But rather than predicting the final accuracy, TA-GATES predicts the accuracy for every half epoch and full epoch time steps. This is seamlessly learnt because TA-GATES is trained for every time-step.

Further, to avoid the operations from having the same embeddings initially, they use some symmetric breaking techniques such as (1) adding random noise (2) add an vector comprised of zero-cost metrics such as grad_norm, snip etc. only initially or at every time step


strengths_and_weaknesses: Strengths:
TA-GATES is able to predict the accuracy for every epoch and not just the final accuracy. So in addition to using it as standalone for NAS, this would be useful to select models that perform well by training for fewer epochs where we have limited training budget.
TA-GATES is able to out perform all the other baselines when trained on limited data.

Weakness:
Despite their new training strategy and improved correlation metrics, TA-GATES is not able to rate the top-k architectures significantly better than GATES as indicated by Precision @ K.
The range of accuracies of architectures found in NDS-Darts is a lot lesser than NAS-Bench 201. So in general, it would be good to demonstrate that the surrogate models work effectively there. NAO was also initially developed on the DARTS search space. 

questions: 1) For table 1 and Table 3, Please compare against more recent performance predictors for some search spaces at least. [1] has open sourced several performance predictors. XGBoost seems to be competitive.

2) For Table1, please include the kendall Tau metrics for 50% and 100% for NB201 and 100% for NB301. 

3) Similarly, please include Precision @ K for all the predictors when predicted on 50% and 100% of the training data.

4) For Table3 also, it would be good to know how Multi-Gates fares against TA-Gates when trained on 100% of the training data.

5) For Table A5, please list the accuracies of the architectures found when the predictors are trained on 100% of the training data.

[1]How Powerful are Performance Predictors in Neural Architecture Search?, White et al.
limitations: See the weakness and questions section.

While the idea of training the surrogate model that captures the training dynamics of the model is an interesting idea, it would be good to understand if it is really essential. Early stopping methods, some of which are listed in [1] are able to predict the accuracies at a given epoch by just extrapolating the learning curves. For your accuracy prediction at half of the training epochs, it would be compare against those for Table 3.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors present a novel encoding scheme for neural network architectures based on the GATES method. They propose a significant change to the GATES approach that trains a different embedding for each of the operations even if they are of the same type. To achieve this they present a series of methods for symmetry breaking that allow the embeddings for each operation to vary independently. They perform an exhaustive series of experiments verifying and ablating their claims to show that their approach TA-GATES is indeed superior to the alternative encoding approaches they consider.
strengths_and_weaknesses: # Strengths
- This is a well-written paper. The prose is clear and the arguments are comprehensible. Enough detail is provided to understand the approach suggested and to understand the experimental setup.
- The extensive experiments seem to indicate that indeed TA-GATES is a superior embedding to the suggested alternatives.
- The area explored by the paper is extremely relevant as having better encodings for architectures can lead to better downstream models and better analysis of architectures based on the encodings. 
- The authors show the added benefits of TA-GATES for anytime performance evaluation that can be extremely useful when partial learning curve information is required.

# Weaknesses
- They do not reference other NN encoding schemes, like those suggested in [1].
- Their argument that the architecture should include variation based on the specific parameters of the model seems suspect. While clearly using different embeddings for the same operations does seem to work well, the argument that the architecture should incorporate this seems incorrect since most of the time the specific parameters of the operation do not seem to be relevant when designing architectures.
- The authors don't include the runtime of TA-GATES. While I assume that the runtime is comparable to GATES it would help to know the specific run times of their approach and compare it to the other approaches

# References
[1] White, C., Neiswanger, W., Nolen, S., & Savani, Y. (2020). A study on encodings for neural architecture search. Advances in Neural Information Processing Systems, 33, 20309-20319.
questions: See the Weaknesses part of the Strengths and Weaknesses section.
limitations: The authors have adequately addressed the limitations of their work.
review_title: null

Meta-review: metareview: The paper proposes a new encoding scheme for neural architecture search.   All reviewers agreed that operations with different trainable parameters need different encodings.   The paper is clearly written and well-motivated.   It will shed great light on future work to consider the trainable parameter for NAS. 
As suggested by dXbo and zsxx, the paper still needs polish.  It should improve the competitor methods, especially providing other strong NAS baselines and  NN encoding schemes.  
confidence: Certain
Decision: Accept