Contrastive Embeddings for Neural Architectures
Authors: Daniel Hesslow
Iacopo Poli
Venue: ICLR 2022 Submitted (2022)

Abstract: The performance of algorithms for neural architecture search strongly depends on the parametrization of the search space. We use contrastive learning to identify networks across different initializations based on their data Jacobians and their number of parameters, and automatically produce the first architecture embeddings independent from the parametrization of the search space. Using our contrastive embeddings, we show that traditional black-box optimization algorithms, without modification, can reach state-of-the-art performance in Neural Architecture Search. As our method provides a unified embedding space, we successfully perform transfer learning between search spaces. Finally, we show the evolution of embeddings during training, motivating future studies into using embeddings at different training stages to gain a deeper understanding of the networks in a search space.

Under review as a conference paper at ICLR 2022
CONTRASTIVE EMBEDDINGS FOR NEURAL ARCHI- TECTURES
Anonymous authors Paper under double-blind review
ABSTRACT
The performance of algorithms for neural architecture search strongly depends on the parametrization the search space. We use contrastive learning to identify networks across different initializations based on their data Jacobians and their number parameters, and automatically produce the first architecture embed- dings independent from the parametrization of the search space. Using our con- trastive embeddings, we show that traditional black-hox optimization algorithms, without modification, can reach state-of offthh-art performance in Neural Architec- ture Search. As our method provides a unified embedding space, we successfully perform transfer learning between search spaces. Finally, we show the evolution of embeddings during training, motivating future studies into using embeddings at different training stages to gain a deeper understanding of the networks search space.
INTRODUCTION
Traditionally, the design state-of-the-art neural network architectures informed by domain knowledge and requires large amount of manual work to find the best hyperparameters. How- ever, automated architecture search methods have recently achieved state-of-the-aad results on tasks such image classification, object detection, semantic segmentation and speech recognition (Ren et al. 2020).
While significant amount of work has gone into improving the algorithms used for architecture search, there has been only limited work on constructing the space in which these algorithms operate. There exists number of different ways in which the architectures in search space can be encoded, and the effects of these choices will affect the performance of the search algorithms.
Previous embedding methods such as (Yan et al., 2020) have focused on preserving the edit distance between the computational graph of the architectures in the embedding space. Mellor et al. (2020) have showed that statistics computed on architectures initialization, before training, can be used to infer which will perform better after training. Motivated by this, our intent is to automatically learn such statistics at initialization and generate an embedding space based on them, that the resulting embedding space preserves fundamental properties of the architectures.
To achieve our goal, we leverage contrastive learning, that has gathered interest in the computer vision community, and produced various state-of- the-trrtt results (He et al. 2020) Chen et al. 2020 Caron et al. 2020), Griil et al. 2020 Zhontar al., (02T). In contrastive learning, the model learns an informative embedding space through self-supervised pre- training phase: from the images in a batch, pairs are generated through random transformations, and the model is trained to generate similar (dissimilar) embeddings for similar (dissimilar) images.
In this work, we combine contrastive learning with the theory presented by Wang et al. (2016) on the Jacobians of networks at initialization, in order to find embedding space suitable for Neural Architecture Search. The embedding space that we generate invariant to the search space of origin, allowing accomplish transfer learning between different search spaces.

---

Under review as a conference paper at ICLR 2022
1.1 MOTIVATIONS AND CONTRIBUTIONS
We design a method to produce architecture embeddings using Contrastive Learning and informa- tion available from their initial state. Our technique is capable of generating embeddings that are independent from the parametrization of the search space, and evolve during training. We leverage these contrastive embeddings in Neural Architecture Search using traditional black-box optimiza- tion algorithms. Moreover, since they provide unified embedding space across search spaces, exploit them to perform transfer learning between search spaces.
embeddings NAS methods promise to outperform random search, however the encoding of the architectures must show some structure for the search algo- rithm to exploit. These encodings are typically produced by condensing all the parameters used to generate an architecture into a single vector. The method used to generate architectures from search space thereby implicitly parametrizes it. The parametrization of the search space affects the performance of NAS algorithm, as noted by e.g. White et al. (2020); |iig et al. (2019); Wang et al. (201)). However, when performing architecture search, is not feasible to test multiple different parametrizations of the search space and evaluate which one performs better: once we have started to evaluate networks in search space, there is reason discard previous evaluations. While the current generation of NAS alleviates the need for experts in the design of architectures, now expert knowledge is needed to build and parametrize search space compatible with the chosen search algorithm, implying that it is exceedingly difficult to outperform a simple random search.
Parametrization-inneeeedd
We present in Sec. A the first method to create networks embeddings without relying their parametrization in any search space, through the combination of modern contrastive learning and the theory of data Jacobian matrices for neural architecture search.
Evolution of the embeddings during training In Section 4.4 we show how the embeddings vary during training, noting that the training procedure tends to connect areas with similar final test ac- curacy together. We hypothesize that this information could enable even more efficient architecture search methods in the future.
Leveraging traditional black-box algorithms Existing methods to generate architecture embed- dings rely on metrics from their computational graphs to identify similar architectures, either by explicitly trying to preserve the edit distance the embedding space, or by leveraging more sophisti- cated methods from graph representation learning. Our method leverages the information contained in the Data Jacobian Matrix of networks at initialization to train contrastive model. As such, it can produce embeddings that more meaningfully capture the structure of the search space. As result, traditional black-bor algorithms perform well for architecture search, as shown for
Dong & Yang 2020) in Section 5.1
Transfer learning between search spaces Our method provides a unified embedding space, since it does not depend on the parametrization of networks in any search space. We exploit this feature to perform for the first time transfer learning between the two search spaces. In practice, we perform it between the size and the topology spaces in NATS-Bench (Dong et al., 2020) in Section 5.3
RELATED WORK
Neural Architecture Search Previous works have attempted to improve network embeddings: Klyuchnikov al. (2020) use graph2vec (Narayanan al., 2017) to find embeddings such that networks with the same computational graph share the same embeddings, and similarly Yan et al. (2000) produce embeddings that are invariant to graph isomorphisms. However, the method differs in that this work trains variational graph isomorphic autoencoder to produce the embeddings. They show that their architecture embeddings arch2vec perform better on downstream architecture search than supervised alternatives, additionally the embeddings that they produce approximately preserves the edit distance the DAGs in a continuous space. Wei et al. (2020) uses a contrastive loss to find low dimensional metric space where the graph edit distances of the original parametrization is approximately preserved. In the absence of dense sampling, all these works rely on the prior that the edit distance is good predictor for relative performance. In contrast, method, learns
2

---

Under review as a conference paper at ICLR 2022
find an embedding space based on intrinsic properties of the architectures. It can therefore discover properties about the architectures which are not present in their graph representation.
Data Jacobian Methods based on the Jacobians with respect to the input of trained networks have been shown to provide valuable information for knowledge transfer and distillation (Czarnecki et al. 2017; Srinivas & Fleuret, 2018), as well as analysis and regularization of networks (Wang al.
2016
Neural Tangent Kernel
The Jacobian of the network with respect to the parameters is computed for inference with neural tangent kernels (NTK) (Jacot al. 2018). Using NTK as a proxy for NAS (Park et al. 2020) underperforms the neural network Guussian process (NNGP) kernel. The NNGP provides an inexpensive signal for predicting an architecture exceeds median performance, but worse than training for few epochs in predicting the order of the top performing architectures.
Contrastive learning Different techniques have been developed in contrastive learning. He et al. (2020) train network with contrastive loss against a memory bank of negative samples produced by slowly moving average version of itself. Chen et al. (2020) remove the memory bank and just consider negative samples from within the same minibatch. Grill et al. (2020) remove the negative samples completely but stabilize the training by encoding the positive samples using momen- tum encoder. Zbootar et al. (202I) use the correlation matrix between the features of the different augmentations and maximize the similarity of the same feature while minimizing the redundancy between features.
3 BACKGROUND
3.1 TRADITIONAL ARCHITECTURE EMBEDDINGS
decision tree is created either implicitly or explicitly to sample networks from a search space. To encode an architecture, one records all choices as the decision tree is traversed into vector, which is then used as the embedding of the architecture. Without any additional knowledge, NAS algorithm will assume that all choices in the decision tree have equal importance the characteristics of an architecture. One commonly used encoding scheme consists of choosing the operations on nodes along with binary adjacency matrix connecting them.
3.2
DATA JACOBIAN
Extended Data Jacobian matrices are used by Wang al. (2016) to analyze trained neural networks. We ground our work in their theoretical setting, and introduce the relevant concepts below.
Multi Layer Perceptrons with ReLU activations are locally equivalent to linear model: the ReLU after linear layer can be combined into single linear layer, where each row in the matrix is replaced by zeros if the output pre-activatte activation is negative.
{ [Ws if (Wx), 2 (0
ReLU(Wz) = Wx Wy
otherwise
Since matrix multiplication is closed, within a neighborhood where the signs of all neurons pre- activation is constant, the full network can be replaced by a single matrix. This property can be extended to any model whose layers can be rewritten as linear layers, including convolutional layers and average pooling layers. Max pooling layers also retain this property, and can be treated similarly to ReLU.
Therefore, in a local neighbourhood close to x. the full information of a network, f. is contained within its Data Jacobian Matrix (DJM). DJM(x)
0f(x)
Ox
3

---

Under review as a conference paper at ICLR 2022
and within that neighbourhood
f(x)= (JM(2))
We can evaluate the Data Jacobian Matrix many different points gather information about multiple different neighhourhoods. If we assume the network to have a single output, its DJM is vector, and we can then stack the DJMs different points form the Extended Data Jacobian Matrix (EDJM). If network has multiple outputs we can sum them to get a single output, which we use calculate the EDJM.
Wang et al. ( use the singular values of the EDJM to compute score, and empirically show that the score correlated with the depth of the network, and its model capacity.
3.3 CONTRASTIVE LEARNING
Contrastive learning is a self-supervised method that finds an informative embedding space of the input data useful for downstream tasks. Central to contrastive learning is the concept of view of an object: two different views of the same object are only superficially different, and we should be able to train network to past these differences and identify the same underlying object. To this end, a network is trained to map different views of the same object close to each other in the embedding space and, conversely, views of different objects should be far apart from each other.
CONTRASTIVE EMBEDDINGS FOR NEURAL ARCHITECTURES
We leverage intrinsic properties of the networks to encode them without depending on their parametrization. We must rely on properties of the architectures at initialization, since is not com- putationally feasible to train the architectures to obtain their embeddings. At variance with previous work, we develop a method to find such properties automatically, using contrastive learning.
To this effect, we train a network that takes an architecture at initialization as input and produces an embedding at the output. desirable that the network has the following two properties:
Different initializations of the same architecture should yield similar embeddings.
Different architectures should yield different emheddings.
We can therefore frame our embedding problem contrastive learning task: different initializa- tions of the same network will correspond to different views of the same sample in the contrastive learning framework.
4.1 OUR METHOD
We compute the Extended Data Jacobian Matrix (EDJM) of architectures at initialization, we rescale each weight matrix by a random and log-uniform scalar in the range (3 }) and low-rank projection of it as input to our contrastive network, to limit memory requirements. We will refer to the projected version of the EDJM as the Extended Projected Data Jacobian Matrix (EPDJM).
(11s(X))l ax,
EPDIM = =xx
where ox denotes a projection onto the top-k principal components.
Σ
0
x=
U2]
VT
(x = UUE
(2)
Σg
The contrastive network is then applied to the EPDJM, and trained using Barlow Twins (Zoontar et al. 2021). Once the Contrastive Network is trained, we can obtain an embeddings of any architec- ture the search space, as shown in Figure The embeddings produced by the contrastive network high dimensional and non-uniform which may be detrimental for some downstream methods. In order solve both of these problems, seek to find low dimensional space where the distances
4

---

Under review as a conference paper at ICLR 2022
Search Space
Handom Initinliatiinss
Contrustive Learnnng
Contrastive
Network
Compute
Contrastive
Sample
Embeddings
EPDJMs
Loes
Contrastive
Network
Figure 1: Illustration of our method for obtaining the network embeddings. We sample architectures from the search space, and form a batch of views with different random initializations. We compute the data Jacobians, project them, and feed them to the contrastive network. The contrastive model learns to generate similar embeddings for networks with similar performance.
between the embeddings is approximately conserved and the volume associated with each architec- ture is approximately uniform. Formally, would like to find low-dimensional embedding that distance (Mémoli 2011):
minimizes the
/ x2xx22
arg min #ETT
(d, x²) dd v)d
Where (-) is the distance in the high dimensional embedding space, (. is the distance in the low dimensional space, and II is the set of mass conservnng transport plans between the measures. Since the target measure is continuous, we have semi-discrete Gromov- Wasseertenn optimal transport problem to which we are not aware of any numerical solvers.
Instead we approximate it with the following two step processs.
arg min
Σ 1
|o (2,, du(2).
(4)
( YxY
arg min
dy Pdd..///rr.....
(5)
well
We solve the first problem with gradient descent. The second is semi-discrete Wasserstein opti- mal transport problem, which we solve with multi-scale averaged gradient descent, as described in Leclaire & Rabin (2019). Informally, the first step finds low-dimensional space where the distances between the embeddings are approximately preserved, whereas the second step makes sure that volume associated with each architecture is uniform. We use the cosine distance in the source space and the Euclidean distance in the low dimensional target space.
For simplicity, for the downstream methods the final embeddings. To facilitate interpretability we use two-dimensional embedding space.
the centroids of the resulting Laguerre cells for
Finally, notice that the resulting embedding space does not accurately capture the number of parameters of the architectures which an important predictor of the final accuracy (Hestness et al. (207); Kaplan et al. (2020)), therefore compliment our embeddings with third dimension containing the logarithm of the number of parameters the architecture.
4.2 IMPLEMENTATION
Since the input to the EDPJMs are two dimensional and contains structure both the Jacobian dimension and the data-point dimension, chose MLPMixer architecture (Tolstikhin
5

---

Under review as a conference paper at ICLR 2022
al. 2021) for the contrastive network, which alternates between applying transformations over the
different dimensions.
For the contrastive learning, use Barlow Twins Zbontar al., 2021) with batch size of 512, and À = 10 as suggested in the original paper. We project the Jacobians down to 256- dimensional space. To accelerate the contrastive learning, we precompute the projected Jacobians using four different initializations for each architecture. The computation of the projected Jacobians takes less than hours for NasBench201 on single GPU (NVIDIA RTX 2080Tí). The embedding size is set to 512, and we use a single layer feedforward network for the projection head.
4.3 ANALYSIS OF THE EMBEDDINGS
We plot the t-SNE projections (Van der Maaten & Hinton 2008) at different stages of our method in Figure analyze the influence of the contrastive learning on the embeddings. We note that the EPDJM alone carries some meaning in the t-SNE space. The contrastive embeddings initialization of the network already exhibit more evident structure. The embeddings after the contrastive learning phase contain little noise and clearly separate architectures based on performance. Finally after using the projection based on optimal transport we have a two-dimensional embedding space where all architectures are near-uniformly laid out the unit square.
Further, we predict the performance of the unseen networks the search space using Random Forests (Breiman (00T) with the default hyperparameters, analyse the predictive power of the embeddings. The results for NAS-B BAS-Bench-201 (Dong & Yang 2020) are shown in Figure 3 and Table I and indicate that the contrastive embeddings are highly predictive of the performance of the architectures in this search space.
Table 1: Metrics computed on predicted accuracies for the three benchmarks in NAS- AAS-Bench-201. This provides condensed view of Figure
CORRELATION
KENDALL-T
CIFAR-10 CIFAR- 100
0.94
0.73
0.93
0.73
IMAGENETI6-11 128
0.90
0.68
100
Top Arciteecues
25
10
20
10
30
20
75
-70
50
201
20
0181
14.0
0.50
10.75
(a) EPDIM
(b) Untrained embeddings (c) Trained embeddings
(d) Final embeddings
Figure 2: t-SNE projections of different statistics of the architectures in
4.4 EMBEDDINGS DURING OPTIMIZATION
Once the contrastive network is trained, it can produce embedding for architectures various points during their training. We show the evolution of the embeddings of 20 architectures during the 50 first epochs of training in Figure the embeddings vary during the training procedure, potentially en- abling future methods to learn more information about the search space for each evaluated network. In particular, the training procedure tends to connect areas with similar final test accuracy.

---

Under review as a conference paper at ICLR 2022
Producted Aocuniy
Prodicted Ascurary
00
Preileted
30
Accuracy
Accutary
Accurocy
(a) CIFAR-10
(b) CIFAR-100
(c) ImageNet16-120
Figure 3: Predicted accuracy against actual accuracy. The predictions produced by Random
Forest Regression applied on our embeddings of 1000 randomly selected architectures NAS- Bench-201 (Dong & Yang 2020
70
75
041
25
30
100
Figure 4: t-SNE projections of movement in embedding space during training of 20 architectures in the NAS-Bae Bench-201 benchmark. The lines show the trajectories the embedding space during training of architecture.
5 ARCHITECTURE SEARCH
We evaluate our contrastive embeddings on the task of architecture search. We use two popular gradient-free optimization algorithms for the search phase. Differential Evolution, DE, (Storn & Price 1997) and Tree Parzen Estimators, TPE (Bergstra et al., 2011). While there are countless other search algorithms, embedding space. For both algorithms, we use the default hyper- papamm-rrsseters and perform no hyper- parameter tuning. We compare our embeddings, CENA, with Arch2Vec produced by Yan et al. 22020). Additionally, we compare with REINFORCE Williams (1992) using a traditional adjacency matrix encoding.
argue that both of these algorithms should work well given a good
5.1 NAS-BENCH-205
We show the results on the NAS-Bench-201 benchmark (Dong & Yang 2020) in Figure S For both downstream methods, TPE and DE, our contrastive embeddings significantly outperform the corre- sponding method using Arch2Vec. Additionally. TPE with our embeddings consistently outperforms REINFORCE using categorical distribution over the possible operations each node.
5.2 NAS-BENCH
The search space in Tas-Bench--Be 101 large and computing the data Jacobian matrices for all archi- tectures would take considerable amount of time. We compute the contrastive embeddings for subset of 1 all architectures which results 16945 architectures. We consider the same subset z when using arch2vec noting that using the full search space does not improve the performance.

---

Under review as a conference paper at ICLR 2022
1.5
Togeet
1.0
0.5
100
1.01
2000
53
100
150
2011
Number Evaluated Andhiteedirusss
Sumber Eralunted Architeeizzeess
(a) 201: CIFAR-10
(b) 201: CIFAR-I00 100
Razdomn Soarch
REINFORCT
CENA TPE
Bogen
Rerget
2.0
CENA DE
1.2
A2V DE
1.0
50
180
1:0
200
50
200
Sumber Evalnated Arehitecirures
Number Exalunted Architeefirrve
(c) 201: InageNet16-120)
101: CIFAR-IO
Figure 5: Search results on NAS- 20S-Bench-201 (Dong & Yang 2020 and NAS-Bench-101 2019)
YYng et al.
We show results the search on the benchmark (Ying et al. 2019) in Figure 5d both downstream methods our embeddings reach a lower regret at the end of the training. However, our REINFORCE implementation using a categorical distribution over the operations on the nodes well the elements of the adjacency matrix, consistently outperforms all other methods on this benchmark.
Again, for
5.3 TRANSFER LEARNING
A unique feature of our contrastive embeddings is that they do not depend on any information about the search space used generated architectures, allowing us to merge the embedding spaces of multiple search spaces into single one. With such unified embedding space, we perform transfer learning from one search space another.
(Dong et al. 2020) consists of two different search spaces: a first one (topology) where the topology architectures evaluated, and second (size) where the number of filters different convolutional layers is evaluated. We use random forests (Breiman) 2001) to predict the accuracy in one search space based on the other, and we display the results in Figure 6
To evaluate the performance on the transfer learning task we compute the Pearson correlation co- efficient as well as the Kendall rank correlation, shown in Table 2 For both size-Hoppoloy and topology-ssize, we see significant correlation between predicted accuracies and the actual accura- cies without ever having evaluated a single network from the target search space.
CONCLUSION AND OUTLOOKS
We have developed end-to-end method to produce embeddings for architectures, using infor- mation available from their initial state and contrastive learning, climinating the need for manual tuning of the search space parametrization. Our analysis of the embeddings clearly shows the ad- vantage introduced by every stage of our pipeline, and our results on Neural Architecture Search using contrastive embeddings promising.
More precisely, by visualizing the t-SNE at different stages of our pipeline, we have shown that the embeddings produced by technique discover the structure of the search space. We also
8

---

Under review as a conference paper at ICLR 2022
16
Poohiced Arcurees
401
Proliced Arcursyy
36
100
20
25
30
.55
35
10
20
10
Acemacyy
(a) size + size
(b) size + topology
Predicted Accisarrg
Prodicted Ancrnrry
20
35
20
25
30
35
10
Anmuneyy
Actunacy
(c) topology size
(d) topology + topology
Figure 6: The transferability of features is evaluated using the two different search spaces that are provided by NATSSBench: size and topology (Dong et al., 2020p. A random forests model trained on 5000 samples from one benchmark and evaluated for example means that the model is trained on the size benchmark and evaluated on the topology one. For both size-ttopolgy and topology-size we see significant correlation between the the predicted accuracies and the actual accuracies without ever having evaluated a single network from the target search space.
the other. The notation size size->topology
Table 2: Metrics computed on predicted accuracies obtained from the transfer learning. random forest model is trained on a source search space and evaluated on the target search space. The metrics are reported as mean standard deviation aggregated over ten runs. The results from the first run can be seen in Figure
SOURGE-TTARGET
CORRELATION
KENDALL-T
SIZE
0.84 0.007
0.66 0.007
TYZEOOTPP
0.60 0.010
0.51 0.014
TOPOLOGY- -IZE
0,51 1.1000010
TOPOLOGY - TOPOLOGY
0.003
0.74 0.003
demonstrated how the embedding space evolves throughout training epochs, connecting regions of the search space with similar final performance, opening the possibility of future work to learn addi- tional information by analyzing these trajectories. Moreover, we evaluated the performance of tra- ditional black-box optimization algorithms using embeddings the task of neural architecture search and show that our embeddings perform better on the evaluated benchmarks than previously suggested embedding methods.
Finally, since the embeddings are independent of the search space, our technique provides a unified embedding space and enables the possibility to learn universal properties of the networks. We veri- fied this by performing for the first time transfer learning across search spaces in neural architecture search.

---

Under review as a conference paper at ICLR 2022
REFERENCES
James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyper-paraman parameter optimization. In 25th annual conference on neural information processing systems (NIPS 201 volume 24, Neural Information Processing Systems Foundation, 2011.
Leo Breiman. Random forests. Machine learning, 45(1):5-32, 2001.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 33, 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709 2020.
Wojciech M Czarnecki, Simon Osindero, Max Jaderberg, Grzegorz Swirszez, and Razvan Pascanu. Sobolev training for neural networks. Advances in Neural Information Processing Systems, pp- 42887 2017.
Xuanyi Dong and Yi Yang. Nas-bench- 102: Extending the scope of reproducible neural architecture search. arXiv preprint arXiv:2001.003:: 2020.
Xuanyi Dong, Lu Liu, Katarzyna Musial, and Bogdan Gabrys. Nats-bench: Benchmarking nas algorithms for architecture topology and size. arXiv preprint arXiv:2009.00437, 2020.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a approach to sule-vipd learning. Advances in Neural Information Processing Systems, 33, 2020.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729-97335 9738, 2020.
Joel Hestness, Sharan Narang, Newsha Ardalani. Gregory Diamos, Heewoo Jun. Hassan Kianinejad, Md Patwary, Mostofa Ali, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically. arXiv preprint arXiv: 2017.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and gen- cralization in neural networks. Advances in neural information processing systems, pp. 8571- 8580, 2018.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0836) 2020.
Nikita Klyuchnikov, Πya Trofimov, Ekaterina Artemova, Mikhail Salnikov, Maxim Fedorov, and Evgeny Burnaev. Nas-bench-nll: neural architecture search benchmark for natural language processing. arXiv preprint arXiv:2006.07/ 16, 2020.
Arthur Leclaire and Julien Rabin. A fast multi-layer approximation to semi-discrete optimal trans- port. International Conference on Scale Space and Variational Methods Computer Vision, pp. 341-353. Springer, 2019.
Joseph Mellor, Jack Turner, Amos Storkey, and Elliot J. Crowley, Neural architecture search without training. 2020.
Facundo Mémoli.
Gromov-wasserstein distances and the metric approach to object matching. Foun- dations of computational mathematics, 11(4):4774477 2011.
Annamalai Narayanan, Mahinthan Chandramohan, Rajasekar Venkatesan, Lihui Chen, Yang Liu, and Shantanu Jaiswal. graph2vec: Learning distributed representations of graphs. arXiv preprint arXiv: (00.:/777,00000 050005 2017.
10

---

Under review as a conference paper at ICLR 2022
Daniel S Park, Jaehoon Lee, Daiyi Peng, Yuan Cao, and Jascha Sohl-Dickstein. Towards nngp- guided neural architecture search. arXiv preprint axXiv:2011.06000 2020.
Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po- Po-Yao Huang, Zhihui Li, Xiaojiang Chen, and Xin Wang. A comprehensive survey of neural architecture search: Challenges and solutions. arXiv preprint arXiv:2006.02900 2020.
Suraj Srinivas and François Fleuret. Knowledge transfer with jacobian matching. arXiv preprint WxXiv:/803.00444 2018.
Rainer Storn and Kenneth Price. Differential evolution-a simple and efficient heuristic for global optimization over continuous spaces. Journal of global optimization, 11(4):341-355 1997.
Iyya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Un- terthiner, Jessica Yung, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, et al. Mlp-mixer: An all-mlp architecture for vision. arXiv preprint arXiv:2105.01601, 2021.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 911)), 2008.
Linnan Wang, Saining Xie, Teng Li, Rodrigo Fonseca, and Yuandong Tian. Sample-effciint neural architecture search by learning action space. arXiv preprint 1406::99060088333 2019.
Shengjie Wang, Abdel-rahman Mohamed, Rich Caruana, Jeff Bilmes, Matthai Plilipose, Matthew Richardson, Krzysztof Geras, Gregor Urban, and Ozlem Aslan. Analysis of deep neural networks with extended data jacobian matrix. In International Conference on Machine Learning, pp. 718- 726. 2016.
Chen Wei, Yiping Tang, Chuang Niu, Haihong Hu, Yue Wang, and Jimin Liang. Self-supervised rep- resentation learning for evolutionary neural architecture search. arXiv preprint arXiv:2011.00186, 2020.
Colin White, Willie Neiswanger, Sam Nolen, and Yash Savani. A study on encodings for neural architecture search. arXiv preprint arXiv:2007.04663. 2020.
Ronald J Williams. Simple statistical gradient-followiin algorithms for connectionist reinforcement learning, Machine learning, (()):229-266, 1992.
Shen Yan, Yu Zheng, Wei Ao, Xiao Zeng, and Mi Zhang. Does unsupervised architecture represen- tation learning help neural architecture search? NeurlPS, 2020.
Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter. Nas- bench-101 101: Towards reproducible neural architecture search. In International Conference Machine Learning, pp. 7105-7114 PMLR, 2019.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. arXiv preprint arXiv:2103.03230, 2021.


Reviews:
[{"review_id": "g4o4VuqGBlB", "reviewer": "Reviewer_pbA7", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. This is a novel way to generate the embeddings that does not require the architecture  to be encoded into a vector representation before it is fed in.\n2. It does not require computation of the accuracy.\n3. They demonstrated that the downstream embeddings have high predictive power by inputing them to a random forest, which was able to predict the accuracy, size etc. \n\nWeakness:\n1. Neural Architecture Search Without Training also used jacobian matrix and show that the kendall Tau is less than 0.55. So I am curious how you are able to achieve higher performance by using the embeddings whose input is the same. I am assuming that the contrastive learning is as good as predictive power of the input\n2. How would you decide which images from the dataset are used to compute the jacobian matrix? This would be crucial to the quality of the embeddings\n3. The original dimension of the embeddings, 512, is not that high.Random Forest and the NAS algorithms can handle it. Why should we reduce the dimension to 2? \n\nMore Experiments requested:\n1. The experiment conducted on NATS-BENCH is not enough to substantiate the claim that the embeddings can generalize across various search space. In order to bolster the claim, could you predict the accuracies for datasets in table 1 while training the contrastive network on NAS-BENCH 101 and DARTS search space and show its predictive power on NAS-BENCH 201\n\n2. For Figure 5, Could you also include arch2vec + REINFORCE, arch2vec + Bayesian Optimization,CENA + REINFORCE and CENA + Bayesian Optimization in the plot?\n\n3. For Table1, could you also include the performance prediction of random forest when using embeddings from NAO [1], GCN [2] and arch2vec\n\n[1] Neural Architecture Optimization, Luo et al.\n[2] Neural Predictor for Neural Architecture Search, Wen et al.\n\n", "summary_of_the_paper": "The paper proposes a new technique to generate embeddings agnostic of the search space. This can be achieved by  first computing the data jacobian of the network with respect to datapoints sampled from different neighbourhoods. This jacobian matrix is then input to a \ncontrastive network which produces architecture embeddings. The contrastive views in this case are different initializations of the same network, which in turn must yield the same embedding. As these embeddings are high dimensional, they are reduced to lower dimension while ensuring that the distance between the embeddings is preserved in the lower dimensional space and the volume associated with each architecture is also preserved. ", "summary_of_the_review": "This paper is novel and have demonstrated its potential. In order to make it more compelling, they must add more empirical evaluations requested above. If they do that, I am leaning towards an accept.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "LoQ3AGdxO7t", "reviewer": "Reviewer_oXnB", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+ Novel neural architecture embedding that is agnostic to topology. \n+ Extensive visualizations on the learned embedding, and its effects in NAS settings. Transfer learning is a very nice experiment.\n\nWeaknesses:\nOverall I think the paper suffers from some clarity and motivation issues. I hope the authors can help clarify in the rebuttal period.\n\n+ EDJM computation is unclear\n\n   The exact empirical computation of EDJM is unclear:\n     1. What are the sampled inputs $x$? Are they fixed? If so, are they fixed for the same architecture, or across all architectures?\n     2. Why is each weight matrix only randomly scaled (instead of re-sampling each entry as usual)?\n\n+ Second dimensionality-reduction stage motivation\n\n   The authors motivated from (1) a dimensionality-reduction perspective and (2) wanting each architecture to have similar volume. I am not convinced by both motivations. \n\n    For (1), the contrastively-learned embedding is 256-dimensional, which is not too high for random search or differential evolution in usual cases. Does the NAS setting require a smaller dimensionality to work? If so, can there be an ablation studies (on dimensionality, and on 2nd stage)?\n\n   For (2), I am not following why Eqn. (5) can be used to achieve roughly uniform volume assignment. The volume here seems to refer to the volume of the set of embedding vectors. But isn't that given by Eqn. (4), which optimizes the corresponding $y$ vectors for each $x$? Also for Eqn. (4), are the $y$ vectors optimized directly or is there an encoder of some sort? If not, is the method limited to training and testing on the same fixed set of architectures? The notations here are a bit confusing. Is $d_\\pi$ a differential or a distance? I was assuming the former (as an OT plan). But why is it operating on $(x, y)$ in Eqn. (3) but $(y,y')$ in Eqn. (5)?\n\n+ Directly using EPDJM\n\n   From Fig. 2a, the top performing architectures are already mostly lined up in EPDJM space. What if one run NAS directly on that?\n\n+ Decoder?\n\n   From my understanding, NAS methods like DE requires a decoding mechanism. What is the decoder here?\n\nMinor presentation issues:\n+ It'd be great to see the training time direction in Fig. 4 .", "summary_of_the_paper": "The paper proposes a self-supervised embedding learning method to learn embeddings of various-sized neural network architectures. Each network is first represented as a low rank projection of a Jacobian matrix, where the rows are Jacobians (output-averaged if multivariate) evaluated at various inputs at random initialization time, called EDJM. Since EDJM is random, multiple such representations are treated as positive pairs for contrastive learning. From the contrastively learned embedding, a further dimensionality-reduction stage is optimized to (1) preserve distances and (2) achieve uniform volume for each architecture. The main application of the final embedding is NAS, where the method outperform baselines.", "summary_of_the_review": "The problem of embedding various architectures into the same fixed-length vector is an interesting one. The present paper proposes an interesting approached based on contrastive-learning. However, the method is not described clearly, and some of the algorithmic choices are not well motivated. Given these considerations, I don't recommend acceptance in its current form.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2owNyiPQX0t", "reviewer": "Reviewer_KncE", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The premise is interesting and the implications are exciting. The contributions in this paper are novel - to the best of my knowledge, this is the first paper that demonstrates that it may be possible to do transfer learning for NAS on related tasks.\n\nSection 4.3 - The predicted accuracy looks quite strong. I am curious why you didn't optimize the hyper parameters of your Random Forest. Further, how was this evaluation done? I couldn't find a description of a train-test split. Random Forests can overfit on data really easily, especially in this case since you'll have only 3 features as input.\n\nIn Section 4.4, I found the premise of the experiment interesting, but I'm unable to conclude anything interesting based on Figure 4. Can you present the data in a more readable format?\n\n5.1, 5.2 - I appreciate the simplicity of the experiment by not confounding the results by including the hyper parametrs of TPE and DE as part of the search space. However, the results in this section are mixed. It looks like CENA performs better than in the baseline in some cases, sometimes by just a small margin and sometimes even worse than REINFORCE.  Also, as was acknowledge by the authors in this section, computing Jacobians can be slow.\n\nApart from the final accuracy, it would be interesting to know how long each method took in wall clock time. If CENA is significantly faster than the other methods, you could make the case that it runs faster while preserving performance. On the other hand if CENA is extremely slow, then perhaps it is better to use random search to evaluate more architectures in the same time period as CENA.\n\n5.3 - In my opinion, this is the biggest strength of the method. However, I wish more details were provided. How exactly are you'll doing transfer learning? Are you using the contrastive learning network from one dataset and using it zero-shot on the other, and then regressing on the accuracies using Random Forests? Once again, was a train-test split used for the random forests and are the reported numbers the out-of-sample performance?", "summary_of_the_paper": "The authors use constrastive learning to embed NAS candidates into an informative search space, which is highly predictive of the architectures' actual accuracy. This embedding space allows traditional black box optimizers to perform well on NAS benchmarks.", "summary_of_the_review": "The core idea and the benefits of this technique are exciting, however experimental details are lacking and the performance is a little lack-luster. \n\n- More details are needed about how the Random Forests were used.\n\n- The paper should focus more on the transfer learning aspect of this technique. I suggest you'll evaluate this technique on more datasets to build a strong case.\n\n- The paper should list the cost of evaluating each method. \n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers unanimously recommending rejecting this submission and I concur with that recommendation. However, many reviewers were quite pleased with the premise and basic concept of the submission and would have liked to see a clearer version with a bit more in terms of experiments. 

I agree with the submission that the most interesting architecture search research is about the search space, not the search algorithm.
The submission uses measurements of the data Jacobian matrix at different points to construct an extended data Jacobian matrix that then is projected and serves as input to a contrastive embedding learning algorithm. The resulting architecture embeddings can be used for many different things, including architecture search.

Ultimately, I am recommending rejecting this submission not because of one single overriding weakness, but because the totality of issues the reviewers raised make it clear the submission is not strong enough to publish in its current form. I encourage the authors to continue this line of work and produce a stronger submission in the future to ICLR or another venue.
Decision: Reject