Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors use constrastive learning to embed NAS candidates into an informative search space, which is highly predictive of the architectures' actual accuracy. This embedding space allows traditional black box optimizers to perform well on NAS benchmarks.
main_review: The premise is interesting and the implications are exciting. The contributions in this paper are novel - to the best of my knowledge, this is the first paper that demonstrates that it may be possible to do transfer learning for NAS on related tasks.

Section 4.3 - The predicted accuracy looks quite strong. I am curious why you didn't optimize the hyper parameters of your Random Forest. Further, how was this evaluation done? I couldn't find a description of a train-test split. Random Forests can overfit on data really easily, especially in this case since you'll have only 3 features as input.

In Section 4.4, I found the premise of the experiment interesting, but I'm unable to conclude anything interesting based on Figure 4. Can you present the data in a more readable format?

5.1, 5.2 - I appreciate the simplicity of the experiment by not confounding the results by including the hyper parametrs of TPE and DE as part of the search space. However, the results in this section are mixed. It looks like CENA performs better than in the baseline in some cases, sometimes by just a small margin and sometimes even worse than REINFORCE.  Also, as was acknowledge by the authors in this section, computing Jacobians can be slow.

Apart from the final accuracy, it would be interesting to know how long each method took in wall clock time. If CENA is significantly faster than the other methods, you could make the case that it runs faster while preserving performance. On the other hand if CENA is extremely slow, then perhaps it is better to use random search to evaluate more architectures in the same time period as CENA.

5.3 - In my opinion, this is the biggest strength of the method. However, I wish more details were provided. How exactly are you'll doing transfer learning? Are you using the contrastive learning network from one dataset and using it zero-shot on the other, and then regressing on the accuracies using Random Forests? Once again, was a train-test split used for the random forests and are the reported numbers the out-of-sample performance?
summary_of_the_review: The core idea and the benefits of this technique are exciting, however experimental details are lacking and the performance is a little lack-luster. 

- More details are needed about how the Random Forests were used.

- The paper should focus more on the transfer learning aspect of this technique. I suggest you'll evaluate this technique on more datasets to build a strong case.

- The paper should list the cost of evaluating each method.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new technique to generate embeddings agnostic of the search space. This can be achieved by  first computing the data jacobian of the network with respect to datapoints sampled from different neighbourhoods. This jacobian matrix is then input to a 
contrastive network which produces architecture embeddings. The contrastive views in this case are different initializations of the same network, which in turn must yield the same embedding. As these embeddings are high dimensional, they are reduced to lower dimension while ensuring that the distance between the embeddings is preserved in the lower dimensional space and the volume associated with each architecture is also preserved. 
main_review: Strengths:

1. This is a novel way to generate the embeddings that does not require the architecture  to be encoded into a vector representation before it is fed in.
2. It does not require computation of the accuracy.
3. They demonstrated that the downstream embeddings have high predictive power by inputing them to a random forest, which was able to predict the accuracy, size etc. 

Weakness:
1. Neural Architecture Search Without Training also used jacobian matrix and show that the kendall Tau is less than 0.55. So I am curious how you are able to achieve higher performance by using the embeddings whose input is the same. I am assuming that the contrastive learning is as good as predictive power of the input
2. How would you decide which images from the dataset are used to compute the jacobian matrix? This would be crucial to the quality of the embeddings
3. The original dimension of the embeddings, 512, is not that high.Random Forest and the NAS algorithms can handle it. Why should we reduce the dimension to 2? 

More Experiments requested:
1. The experiment conducted on NATS-BENCH is not enough to substantiate the claim that the embeddings can generalize across various search space. In order to bolster the claim, could you predict the accuracies for datasets in table 1 while training the contrastive network on NAS-BENCH 101 and DARTS search space and show its predictive power on NAS-BENCH 201

2. For Figure 5, Could you also include arch2vec + REINFORCE, arch2vec + Bayesian Optimization,CENA + REINFORCE and CENA + Bayesian Optimization in the plot?

3. For Table1, could you also include the performance prediction of random forest when using embeddings from NAO [1], GCN [2] and arch2vec

[1] Neural Architecture Optimization, Luo et al.
[2] Neural Predictor for Neural Architecture Search, Wen et al.


summary_of_the_review: This paper is novel and have demonstrated its potential. In order to make it more compelling, they must add more empirical evaluations requested above. If they do that, I am leaning towards an accept.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a self-supervised embedding learning method to learn embeddings of various-sized neural network architectures. Each network is first represented as a low rank projection of a Jacobian matrix, where the rows are Jacobians (output-averaged if multivariate) evaluated at various inputs at random initialization time, called EDJM. Since EDJM is random, multiple such representations are treated as positive pairs for contrastive learning. From the contrastively learned embedding, a further dimensionality-reduction stage is optimized to (1) preserve distances and (2) achieve uniform volume for each architecture. The main application of the final embedding is NAS, where the method outperform baselines.
main_review: Strengths:
+ Novel neural architecture embedding that is agnostic to topology. 
+ Extensive visualizations on the learned embedding, and its effects in NAS settings. Transfer learning is a very nice experiment.

Weaknesses:
Overall I think the paper suffers from some clarity and motivation issues. I hope the authors can help clarify in the rebuttal period.

+ EDJM computation is unclear

   The exact empirical computation of EDJM is unclear:
     1. What are the sampled inputs $x$? Are they fixed? If so, are they fixed for the same architecture, or across all architectures?
     2. Why is each weight matrix only randomly scaled (instead of re-sampling each entry as usual)?

+ Second dimensionality-reduction stage motivation

   The authors motivated from (1) a dimensionality-reduction perspective and (2) wanting each architecture to have similar volume. I am not convinced by both motivations. 

    For (1), the contrastively-learned embedding is 256-dimensional, which is not too high for random search or differential evolution in usual cases. Does the NAS setting require a smaller dimensionality to work? If so, can there be an ablation studies (on dimensionality, and on 2nd stage)?

   For (2), I am not following why Eqn. (5) can be used to achieve roughly uniform volume assignment. The volume here seems to refer to the volume of the set of embedding vectors. But isn't that given by Eqn. (4), which optimizes the corresponding $y$ vectors for each $x$? Also for Eqn. (4), are the $y$ vectors optimized directly or is there an encoder of some sort? If not, is the method limited to training and testing on the same fixed set of architectures? The notations here are a bit confusing. Is $d_\pi$ a differential or a distance? I was assuming the former (as an OT plan). But why is it operating on $(x, y)$ in Eqn. (3) but $(y,y')$ in Eqn. (5)?

+ Directly using EPDJM

   From Fig. 2a, the top performing architectures are already mostly lined up in EPDJM space. What if one run NAS directly on that?

+ Decoder?

   From my understanding, NAS methods like DE requires a decoding mechanism. What is the decoder here?

Minor presentation issues:
+ It'd be great to see the training time direction in Fig. 4 .
summary_of_the_review: The problem of embedding various architectures into the same fixed-length vector is an interesting one. The present paper proposes an interesting approached based on contrastive-learning. However, the method is not described clearly, and some of the algorithmic choices are not well motivated. Given these considerations, I don't recommend acceptance in its current form.
review_title: null

Meta-review: All reviewers unanimously recommending rejecting this submission and I concur with that recommendation. However, many reviewers were quite pleased with the premise and basic concept of the submission and would have liked to see a clearer version with a bit more in terms of experiments. 

I agree with the submission that the most interesting architecture search research is about the search space, not the search algorithm.
The submission uses measurements of the data Jacobian matrix at different points to construct an extended data Jacobian matrix that then is projected and serves as input to a contrastive embedding learning algorithm. The resulting architecture embeddings can be used for many different things, including architecture search.

Ultimately, I am recommending rejecting this submission not because of one single overriding weakness, but because the totality of issues the reviewers raised make it clear the submission is not strong enough to publish in its current form. I encourage the authors to continue this line of work and produce a stronger submission in the future to ICLR or another venue.
Decision: Reject