A Gradient-based Kernel Approach for Efficient Network Architecture Search
Authors: Jingjing Xu
Liang Zhao
Junyang Lin
Xu Sun
Hongxia Yang
Venue:  (2021)
Keywords: NAS

Abstract: It is widely accepted  that vanishing and exploding gradient values are the main reason behind the difficulty of deep network training.
In this work, we take a further step to understand the optimization of deep networks and find that both gradient correlations and gradient values have strong impacts on model training. 
Inspired by our new finding, we explore a simple yet effective network architecture search (NAS) approach that leverages gradient correlation and gradient values to find well-performing architectures. To be specific, we first  formulate these two terms into a unified gradient-based kernel and then select architectures with the largest kernels at initialization as the final networks. 
The new approach replaces the expensive  ``train-then-test'' evaluation paradigm  with a new lightweight function according to the gradient-based kernel at initialization.
Experiments show that our approach achieves competitive results with orders of magnitude faster than ``train-then-test'' paradigms on image classification tasks. Furthermore, the extremely low search cost enables its wide applications. It also obtains performance improvements on two text classification tasks.

Under review as a conference paper at ICLR 2021
A GRADIENT-BASED KERNEL APPROACH FOR EFFI- CIENT NETWORK ARCHITECTURE SEARCH
Anonymous authors Paper under double-blind review
ABSTRACT
It is widely accepted that vanishing and exploding gradient values are the main reason behind the difficulty of deep network training. In this work, we take a further step to understand the optimization of deep networks and find that both gradient correlations and gradient values have strong impacts on model training. Inspired by our new finding, we explore a simple yet effective network architecture search (NAS) approach that leverages gradient correlation and gradient values to find well- performing architectures. To be specific, we first formulate these two terms into a unified gradient-based kernel and then select architectures with the largest kernels at initialization as the final networks. The new approach replaces the expensive Ttrain-then-tee evaluation paradigm with a new lightweight function according to the gradient- kernel at initialization. Experiments show that our approach achieves competitive results with orders of magnitude faster than "train-thee thentthet"" paradigms on image classification tasks. Furthermore, the extremely low search cost enables its wide applications. It also obtains performance improvements on two text classification tasks
INTRODUCTION
Understanding and improving the optimization of deep networks has been an active field of artificial intelligence. One of the mysteries in deep learning why extremely deep neural networks are hard to train. Currently, the vanishing and exploding gradient problem is widely believed to be the main answer (Bengio al. 1994; Hochreiter & Schmidhuber, 1997: Pascanu et al. 2013). Gradients exponentially decrease (or increase) from the top layer to the bottom layer in multi- layer network. By extension, the current vanishing and exploding gradient problem also refers to extremely small (or large) gradient values. Following this explanation, several widely-used training techniques have been proposed to assist optimization. These studies can be roughly classified into four categories: initialization-brat based approaches (Glorot & Bengio 2010: He et al. 2015 Zhang cct al. 2019), activation
approaches (Hendrycks & Gimpel| 2016 Klambauer et al. 2017). normalization based approaches (loffe & Szegedy Saliaaa & Kingmal 2016 Ulyanov et al.
2016
Lei Ba et al. 2016 Nguyen & Salazar 2019). and skip-cokneee cili-coonection based approaches (He et al.
2016 Huang al. (0) These techniques provide good alternatives to stabilize gradients and
bring promising performance improvements.
Despite much evidence showing the importance of steady gradient values, we still curious about whether there are unexplored but important factors. To answer this question, we conduct extensive experiments and find some surprising phenomena. First, the accuracy curves of some models converge well with the remaining gradient vanishing and exploding problem. Second, we find several cases where models with similar gradient values show different convergence performance. These results indicate that gradient values may not be vital as much as we expect, and some hidden factors play a significant role optimization. To answer this question, we explore other gradient features, such as covariance matrix, correlations, variance, and so on. These features are used to evaluate gradients from different perspectives. The covariance matrix evaluates the similarity between any two parameters, which is widely used in optimization studies (Zhao & Zhang 2015 Faghri et al., al. (022)). Gradient correlation is defined to evaluate the similarity between any two examples. Among them, experiments show that gradient correlation is important factor. Compared to widely-explored
We will release the code on publication.

---

Under review as a conference paper at ICLR 2021
gradient values, gradient correlations are rarely studied. As gradient values affect the step size in optimization, vanishing gradients prevent models from changing weight values. In the worst case, this could completely stop training. Gradient correlations evaluate the randomness of gradient directions in Euclidean space. Lower gradient correlations indicate more "random" gradient directions and more "conflicting" parameter updates. For better understanding, we visualize absolute gradient values and gradient correlations (See Section 4 for more details). It illustrates that models with either small values small correlations show worse convergence and generalization performance, indicating the importance of these two factors.
Following the new finding, we take further step to explore gradient correlations and gradient values in network architecture search. We first formulate two factors into a unified gradient- gradient-based kerne,, which is defined as the average of a gradient dot-product matrix. Motivated by the observation that architectures with larger kernels at initialization tend to have better average results, we develop lightweight network architecture search approach, called GT-NAS, which evaluates architectures according to the gradient-base based kernel at initialization. Unlike other NAS approaches that evaluate architectures via full training, which takes massive computation resources, GT-NAS only relies on few examples for kernel calculation. We first select top-k architectures with largest kernels as candidates, which are then trained to select the best one based on validation performance. In practice, k is usually set to be very small value. Experiments show that GT-NAS achieves competitive results with extremely fast search on The low search cost allows us to apply NAS on diverse tasks. Specifically, the structure searched by the proposed policy outperforms the naive baseline without NAS on 4 datasets covering image classification and text classification.
(Ying et al. 2019), a NAS benchmark dataset.
The main contributions are summarized as follows:
We propose that both gradient correlations and gradient values matter in optimization, which gives new insight to understand and develop optimization techniques.
Following the new finding, we propose a gradient-based kernel approach for NAS, which is able to search for well well-performing architectures efficiently.
Experiments show that GT-NAS achieves competitive results with orders of magnitude faster than the naive baseline.
2 RELATED WORK
Understanding and improving optimization Understanding and improving the optimization of deep networks has long been a hot research topic. [Bengio et al. (1994) find the vanishing and exploding gradient problem in neural network training. To address this problem, a lot of promising approaches have been proposed in recent years, which can be classified into four research lines.
The first research line focuses on initialization (Sutskever et al. 2013: Mishkin & Matas, 2016
Hanin & Rolnick 2018). Glooot & Bengio (2010) propose to control the variance parameters via appropriate initialization. Following this work, several widely-used initialization approaches have been proposed, including Kaiming initialization (He et al. 20T5), Xaiver initinalization (Glorot & Bengio 2010), and Fixup initialization (Zhang et al. 2019). The second research line focuses
normalization (loffe & Szegedy| 2015 Lei Ba et al.] 2016 Ulyanov et al., 2016 Wu & He 2018 Nguyen & Salazar| 2019). These approaches aim to avoid the vanishing gradient by controlling the distribution of intermediate layers. The third is mainly based on activation functions to make the derivatives of activation less saturated to avoid the vanishing problem, including GELU acti- vation (Hendrycks & Gimpel| (0166), SELU activation (Klambauer et al. (017).. and so on. The motivation behind these approaches is to avoid unsteady derivatives the activation with respect to the inputs. The fourth focuses on gradient clipping (Pascanu et al. 2013).
Gradient-based Kernel Our work is also related to gradient-basee kernels (Advani & Saxe 2017 Jacot et al., 2018). NTK (Jacot et al.) 2018) is a popular gradient-based kernel, defined as the Gram matrix of gradients. is proposed to analyze model's convergence and generalization. Following these studies, many researchers are devoted understand current networks from the perspective of NTK (Lee et al. 2019 (Hasti et al.] 2019 Allen -hu et al., 2019 Arora et al., 2019). Duu et al. (2019)) use the Gram matrix of gradients to prove that for m hidden node shallow neural network
2

---

Under review as a conference paper at ICLR 2021
with ReLU activation, long is large enough, randomly initialized gradient descent converges to globally optimal solution at a linear convergence rate for the quadratic loss function. Following this work, Du et al. (2019a) further expand this finding and prove that gradient descent achieves zero training loss in polynomial time for a deep over-parameterized neural network with residual connections.
Neural architecture search aims replace expert-designed networks 2019 Li et al. 2019 /hou et al. 2020 He et al., 2020aa] Fang et al. 2020 He al., 2020b; You al.
Network architecture search
with learned architectures (Chang et al.
2020 Alves & de Oliveira 2020) he first NAS research line mainly focuses on random search. The key idea is to randomly evaluate various architectures and select the best one based on their validation performance. However, despite promising results, many of them require thousands of GPU days to achieve desired results. To address this problem. Ying al.] (2019) propose a reinforcement learning based search policy which introduces an architecture generator with validation accuracy as reward. Another research line is based on evolution approaches. Real et al. (2019) propose two-stage search policy. The first stage selects several well pell-perfonning parent architectures. The second stage applies mutation these parent architectures to select the best one. Following this work, [o et al. (2019) apply the evolution search on Transformer networks and achieve new state-of-the of---h--ttt-tttt- art results on machine translation and language modeling tasks. Although these approaches can reduce ex exploration costs, the dependence on validation accuracy still leads to huge computation costs. To get rid of the dependence on validation accuracy, several studies (Jiang et al. 2019 Liu et al. 2019 Dong & Yang 2019 Zela et al.) 2020 Chu et al.) (2020) re- re-formulate the task differentiable manner and allow efficient search using gradient descent. Unlike these studies, we propose a lightweight NAS approach, which largely reduces evaluation cost. Furthermore, the
new approach does not rely on additional search models, thus also saves memory usage.
3 NOTATIONS: GRADIENT VALUE AND GRADIENT CORRELATION
Assume g(); = (o((),,,,... (((r)r } is the gradient vector of example with respect to all parame- ters in the 1-th layer. is the total number of parameters. We combine all gradient vectors together to construct a gradient matrix Gil) E dd where d refers to the number of training examples. We define a matrix P(I) to store the dot-product of gradients. The dot-product between two gradient vectors is:
P() = (G()),, G())).
(L)
We divide P(() by the norm of gradient vectors to get gradient correlation matrix C(I):
P()
C( |[G(|| |G)
(2)
Then, we use the average value of C()) as the final correlation score:
R = B(C())
(3)
Here also evaluate the absolute gradient value:
V V= B(G())
(4)
Since it requires too much time to calculate It and V for plenty of training examples, we randomly sample μ examples to estimate and V in implementation for fast calculation. The dot-product between vectors with too many elements will cause the overflow problem. We also randomly sample 0 parameters when calculating R. To avoid the sampling bias that might affect our findings, we test different fa E [50, 100, 200, 300, 400, 500) and 0 E [50, 100, 200, 300, 400, 500], and observe the same phenomena. For simplicity, we show our findings in the next section with μ 50 and = 50.
4 RETHINKING VANISHING AND EXPLODING GRADIENTS
In this section, we first analyze how the problem of vanishing and exploding gradient arises. Then, we show some experimental cases to prove that vanishing and exploding gradient values are not the only factors behind the difficulty of deep network training.
3

---

Under review as a conference paper at ICLR 2021
4.1 VANISHING AND EXPLODING GRADIENTS IN TRAINING DEEP NETWORKS
We consider a simple multi-layer fully-connected neural network with L layers. Assume that the output of h-th layer is:
(h) (w (w(h) (( (1-1))
(5)
where a is the activation function and wh(h) the weight matrix. The gradient for the h-th layer with respect to the weight matrix (h) is:
ac
OC "y(l))
(L J j(h)
ac "yy
II kmn+11
J (N)). J(m))
(hn(h)
(2)
0y
-ayin
where L is the loss function and Ty output of layer. (kk is the diagonal matrix where the main diagonal is the list of the derivatives of the activation with respect to the inputs in the k-th layer:
BC is the derivative of the output of the last layer ( is the
J (k) diag(o' (ww) ( ( ).... o'' (un (1) (())))))))))))))) E Rdddd
where d the dimension of the activation and w (k) the d-th row in matrix w(k) in the layer.
From this equation, we can see that the norm of weight matrix w. the output of previous layer y (hh-1) and the derivative of activation play an important role in controlling the gradient stability. Most previous studies mainly focus on these key terms address the vanishing and exploding problem.
0.207
7000
0.25
0.208
0.000
0.505
3.0
0.025
5000
0.15
0.004
0.3
0.2
0.129
5.020
5.015
3000
0.10
5.000
1000
526,105
0.05
20
20
20
20
20
bayer
layer
layer
(a) Baseline
(b) LayerNorm
SELU
(d) Residual
ResNorm
Figure 1: Gradient values on different models. The X-axis the index of layers and Y-axis denotes the average absolute gradient value in each layer. The dotted line shows the average of all absolute gradients in a model. "Baseline" has extremely low gradient values on all layers and its gradients are multiplied by 1000 for clear presentation. Even with advanced training techniques, the vanishing and exploding problem not addressed completely. The gradients between the top layer and the bottom layer are still in orders of magnitude in "LayerNorm", "Residual", and "ResNorm".
4.2 GRADIENT VALUES AND GRADIENT CORRELATIONS BOTH MATTER
To empirically evaluate the effects of unsteady gradient values, we implement widely-used optimiza- tion techniques, including activation sraiaatioo-bbsed techniques (e-g., SELU), normalization based techniques (e.g., LayerNorm), and skip-connection-bsse
techniques (e.g.. Residual). We adopt 32-layer multi-layer perceptron (MLP) network as the backbone for MNIST (LeCun al. 2010), an image classification dataset. The bottom is a 2-layer convolutional component and the top is 30-layer foed-forward component. The baseline model uses ReLU activation and Xavier initialization (Glorot & Bengio 2010). The evaluated techniques settings for fair comparison. We use the AdaGrad optimizer DDcch et al. 2011) with learning rate 1.0 and adopt learning rate decay mechanism with 7 0.7. The batch size is set to 64. All experiments run on single NVIDIA V100 GPU. The total number of running epochs is 15.
then added into the baseline. All the models use the
same hyper-
Figure T shows gradient value curves different layers at initialization, and Figure demonstrates the distribution over gradient values initialization and at training stages. "Baseline" has serious gradient vanishing problems with extremely small values in all layers at initialization. With the
Here adopt numerator layout notation.
4

---

Under review as a conference paper at ICLR 2021
SEU
SEW
10.0
10.0
1271
10P
1971 1277
12
Figure 2: From left to right, we demonstrate the charts of gradient values at initialization and at training stages, gradient correlations at initialization, and at training stages, "Baseline" and "Layer- Norm" with either lower gradient correlations or lower gradient values achieve worse convergence performance. "ResNorm" and "Residual" with higher gradient correlations or higher gradient values, obtain better convergence performance.
increase of training steps, the vanishing problem is slightly addressed but still serious. is as expected that "Baseline" achieves the worst convergence performance with 89.52 test accuracy, shown in Figure With advanced training optimization techniques, some layers have larger gradient values. The accuracy curves of "SELU", "Residual" and "ResNorm" converge well. These results verify the importance steady gradient values in optimization. However, there are still some cases that are hard to understand from the perspective of gradient values. First, it is out of our expectation that the accuracy curves of "LayerNorm", "Residual", and "ResNorm" converge well although the gap of gradients between the top layer and the bottom layer in these models is still in orders of magnitude. These results demonstrate that a deep network is robust to vanishing or exploding gradients to some extent. Second, "SELU" and "ResNorm" with similar ranges of gradient values show distinet convergence speed. It indicates that there are still unknown factors behind the optimization of deep networks.
In addition to gradient values, we observe that gradient correlations contribute the convergence performance.
We compute gradient correlations based on Eq. Figure 2 shows gradient correlation distributions. Compared to "SELU" and "ResNorm". "LaayerNorm" has much lower gradient correlations and shows worse convergence per- formance. Despite extremely low gradient values, "Base- line"with higher average gradient correlations at initial- ization achieves faster convergence than "LayerNorm". These results demonstrate that gradient correlations also matter in optimization.
Accuracy (Ng)
70
Residaal
30
Restorm
Layerkorm
Selu
Baseline
Epoch
Figure 3: Test accuracy of different mod- els on CIFAR10.
LEVERAGING GRADIENT CORRELATIONS AND GRADIENT VALUES IN NETWORK ARCHITECTURE SEARCH
We also explore how to leverage this new finding to improve the performance of deep networks in real-world tasks. First, we unify these two items into gradient-based kernel, defined as the average of the Gram matrix of gradient, i.e., P in Eq. Starting with several simple architectures, one observation is that networks with larger kernels initialization tend achieve higher average convergence performance. Motivated by this observation, we propose to adopt the gradient- based kernel on NAS search for the optimal architecture.
First, we generate s possible architectures as the search space S. In this work, we do not fix the generating policy to keep the flexibility on diverse tasks and scenarios. For each architecture, we calculate the gradient-based kernel based on Eq.1 and select i architectures with the highest scores as candidates, which are then trained from scratch to get their results on a validation set. For fast calculation, set μ = 50 and 0 50 when estimating kemel. The architecture with the highest validation accuracy is selected as the final architecture. We normalize gradients by subtracting the average of sampled gradients before calculation. The new evaluation function does not need any training steps. Compared to other NAS approaches that train hundreds of architectures, the new approach largely reduces search costs.
5

---

Under review as a conference paper at ICLR 2021
6 EXPERIMENTS
In this section, we evaluate the proposed approach on MAS-Bench-201, NAS benchmark dataset. provides a fair setting for evaluating different search techniques.
6.1 DATASETS
MAS-Bench-22 is benchmark dataset for NAS algorithms, constructed on image classification tasks, including CIFARIO. IO, CIFAR and ImageNet- 16-120 (ImageNet). CIFARIO and CIFARIO0 are two widely used datasets CIFAR-10 consists of 60,000 32x32 color images in 10 classes, with 6,000 images per class. There are 50,000 training images and 10,000 test images. CIFAR 100 has 100 classes containing 600 images each. There are 500 training images and 100 testing images per class. ImageNet is provided by [Chrabaszzz et al. (017). NAS- Bench-221 chooses 4 nodes and representative operation candidates for the operation set, which generates 15,625 cells/architectures as search space. In summary, NAS-Bench-200 enables researches to easily re-implement previous approaches via providing all architecture evaluation results. However, since these results are unavail- able in real-world tasks, we take the evaluation time into account when computing the search time for a fair comparison.
6.2 BASELINES
We compare the proposed approach with the following baselines:
Random search algorithms It includes two baselines, random search (RS) (Bergstra & Bengio 2012) and random search with parameter sharing (RSPS) (Li & Talwalkar 2019).
Reinforcement learning based algorithms It includes one baseline, RL Williams 1992 RL is a method that introduces an architecture generator to generate well-performing architectures.
Evolution based search algorithms search (REA) (Real et al. 2019).
It includes one baseline: regularized evolution for architecture
Differentiable algorithms It includes two baselines, DARTS (Liu al. 2019) and GDAS (Dong & Yang, 2019). DARTS reformulates the search problem into a continues search problem. Following this work, GDAS updates a sub-grap sub-graph rather than whole graph for higher speeds.
Hlyper-parameter optimization algorithms It includes one baseline, BOHB (Falkner et al. 2018). This approach combines the benefits both Bayesian optimization and bandit-based methods.
We use the released code provided by Dong & Yang (2020) for baseline implementation, See Appendix A for more implementation and baseline details.
6.3 RESULTS
Table [ shows the comparison between GT-NAS and the state-of-the-arr approaches. The details of search time computing are described in Appendix A. RS a naive baseline, which randomly selects architectures for full-training and evaluation. achieves good accuracy improvements with 0.46 and 1.25 gains on CIFAR and ImageNet. RL adds learning into the search process and can explore more well-performing architectures. However, due to its unsteady training characteristic, RL does not beat RS three datasets. REA contains two search stages: parent search and mutation. The second stage explores new architectures based on best- best-performing parent architectures. This approach achieves the highest performance with 93.72, 72.12, and 45.01 accuracy scores on CIFARIO, CIFAR 100, and ImageNet. In conclusion, despite good results, these basclines require considerable time for architecture evaluation.
one of the most time-consuming approaches and
In comparison, there are also several approaches that achieve much better speed-up performance, such as RSPS, GDARTS and DARTS. However, the accuracy of architectures searched by these approaches
hhtpss/wwww.es.toromto. edu/ kriz/eifar.html

---

Under review as a conference paper at ICLR 2021
Table I: Performance of different approaches CIFARIO, CIFAR1OM 100, and ImageNet. "Time" means the search time. GT-NAS achieves competitive results with an extremely higher speed. In GT-NAS, all search steps can be finished within few hours on single GPU.
CIFARIO
CIFAR100
ImageNet Time(s)
Type
Model
Acc
Time(s)
Speed-up
Acc
Time(s) Speed-up
Ace
Speed-up
w/o Search
ResNet
93.97
N/A
N/A
70.86
N/A
N/A
43.63
N/A
N/A
RS
93.63
216K
1.0x
71.28
460K
1.0x
44.88
IM
1.0x
RL
92.83
216K
1.0x
70.71
460K
1.0x
44.10
IM
1.0x
Search
REA
93.72
216K
1.0x
72.12
460K
1.0x
45.01
IM
1.0x
BOHB
93.49
216K
1.0x
70.84
460K
1.0x
44.33
IM
1.0x
RSPS
91.67
10K
21.6x
57.99
46K
21.6x
36.87
104K
9.6X
GDAS
93.36
22K
12.0x
67.60
11.7x
37.97
130K
7.7x
Gradient
DARTS
88.32
23K
9.4x
67.34
80K
5.8x
33.04
110K
9.1x
Kernel
GT-NAS
93.42
7K
30.8x
71.42
25.5x
45.35
31K
32.2x
not high enough. Unlike these studies, GT-NAS achieves competitive results with an extremely higher speed. Compared to the time-consuming approaches, including RS, RL, REA, and BOHB, the proposed GT-NAS reduces search cost and brings 30.8x, 25.5x and 32.3x speed-up on CIFARIO. CIFAR100, and ImageNet. Compared to approaches with higher search speed, including RSPS, GDAS and DARTS, the proposed GT-NAS obtains large accuracy improvements. The searched architecture also outperforms ResNet with 0.56 and 1.72 accuracy improvements on CIFARIO 100 and ImageNet. Furthermore, all search steps only take a few hours on a single GPU in the proposed approach. For better understanding, and test accuracy, as shown in Figure The kernel is the average of a dot-product matrix in Eq. 1. Each dot represents single architecture with its kernel (X-axis) and accuracy (Y-axis). We draw two plots with a different granularity of kernel ranges. On three datasets, higher kernels bring better average model performance. Since the kernel is decided by gradient values and correlations, this result also verifies our findings that both gradient values and gradient correlations matter.
visualize the relation between the kernel at initialization
Universal to different architectures MAS-Bench-201 contains over 100K architectures as search space, covering common network blocks. On such diverse dataset, promising results of proposed method show that GT-NAS is universal to different architectures.
CIFAR10
(b) CIFAR100
ImageNet
Figure 4: Higher kernels brings more steady performance. Each dot represents single architecture with its kernel (X-axis) and accuracy (Y-axis). For each dataset, we draw two plots with a different granularity of kernel ranges.
7 GENERALIZATION ON DIVERSE TASKS
We are also interested in whether GT-NAS can find better structures at acceptable cost more datasets. To verify this, we conduct experiments on text classification tasks. Text classification includes two datasets, MRPC evaluating whether two sentences semantically equivalent and RTE recognizing textual entailment. Two classification datasets are provided by Wang et al. (2019). See Appendix B for more dataset details.
7

---

Under review as a conference paper at ICLR 2021
Table 2: Results text classification tasks. "Time" refers to search time.
MRPC
RTE
Models
Acc
Time(s)
Acc
Time(s)
Baseline
92.08 93.32 (+1.24)
N/A
83.51
N/A 2K
GT-NAS
0.4K
83.75(+0.24)
7.1 SEARCH SPACE
For ease of implementation, we first generate some architectures search space. We define several skip-connection structures, each with possible candidate architectures. We set the same layer number for all architectures: 12 encoder layers and 12 decoder layers. These architectures the same hype-parameters as the baseline. Here are the details different structures:
Highway structure It connects all cells in chain way. each cell, there exists highway connection from the first layer to the last layer. We adopt an average fusion way to merge all inputs the last layer. The cell length ranges from 2 to and there are 10 possible architectures.
Look-ahead structure connects all cells in chain way. In each cell, the layer is connected in a chain way except for the last layer. The last layer takes all outputs of previous layers in cell as inputs. We adopt average fusion way merge all inputs the last layer. The cell length ranges from 2 to 11 and there are 10 possible architectures.
DenseNet structure splits the whole network into different cells (Huang et al. 2017b). The cell is connected in fully connected way where the input of current cell comes from all outputs of previous cells. In each cell, the layer is connected in a chain way. The original network adopts a concatenation way fuse the input vector together and elaborately designs dimension shrinking and expanding policy. For better generalization, we replace the concatenation fusion with an average fusion way that the dimension design efforts required. The cell length ranges from 11 and there are 10 possible architectures.
7.2 RESULTS
The results shown in Table 2 The architectures searched by GT-NAS achieve better results all datasets. Though pre-trained models are widely believed to the state-of-the--e approaches, GT-NAS still obtains performance improvements accuracy improvements on MRPC and RTE datasets. For two classification datasets, we choose top-2 and top-5 architectures for full training and evaluation. It proves that GT-NAS generalizes well on various datasets. In addition, these results show the importance of skip-connections in deep networks. Without adding any parameters, simply adjusting the way of connections can bring large performance improvements.
these strong baselines with 1.24 and 0.24
Universal to different initialization approaches We conduct experiments on two different tasks, each with different initialization. The former uses Kaiming initialization, and the latter adopts Xaiver initialization. These two both are popular initialization methods. Table and Table demonstrate that our method is universal different initialization approaches.
CONCLUSION
In this work, propose explanation that gradient values and gradient correlations both matter in optimization. We believe that this can give new insight into further optimization research. Following this new finding, develop fast yet effective approach which obtains competitive results with much higher speeds on a NAS benchmark dataset. Furthermore, GT-NAS generalizes well and brings improvements diverse tasks including image classification and text classification.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Madhu S. Advani and Andrew M. Saxe. High-dimensional dynamics of generalization error in neural networks. CoRR, abs/1710.03667,, 2017.
Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu I Liang. Learning and generalization in overparameterized neural networks, going beyond two layers. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc. Emily B. Fox, and Roman Garnett (eds.). Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeuriPS 2019, 8-14 December 2019, Vancouver, BC, Canada, pp. 6155-6166, 2019.
Jeovane Honório Alves and Lucas Ferrari de Oliveira. Optimizing neural architecture search using limited GPU time in a dynamic search space: A gene expression programming approach. CoRR, abs/2005.07669, 2020.
Sanjeev Arora, Simon S. Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 322-322-2232 -22. PMLR, 2019.
Yoshua Bengio, Patrice Y. Simard, and Paolo Frasconi. Learning long-term dependencies with gradient descent difficult. IEEE Trans. Neural Nerworks, 5(2):157-166) 166, 1994.
James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. J. Mach. Learn. Res., 13:281-305, 2012.
Jianlong Chang, Xinbang Zhang, Yiwen Guo, Gaofeng Meng, Shiming Xiang, and Chunhong Pan. DATA: differentiable architecture approximation. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, 8-14 December 2019, Vancouver, BC, Canada, pp. 874 884, 2019.
Patryk Chrabaszez, Ilya Loshchilov, and Frank Hutter. downsampled variant of imagenet as an alternative to the CIFAR datasets. CoRR, abs/1707.088 2017.
Xiangxiang Chu, Bo Zhang, and Xudong Li. Noisy differentiable architecture search. CORR, abs/2005.03366,, 2020.
Xuanyi Dong and Yi Yang. Searching for robust neural architecture in four GPU hours. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 1761-1770, 2019.
Xuanyi Dong and Yang. Nas-hench-201: Extending the scope of reproducible neural architecture search. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, 2020.
Simon S. Du, Jason D. Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, pp. 1675-1685, 2019a.
Simon S. Du, Xiyu Zhai, Barnabás Póczos, and Aarti Singh. Gradient descent provably optimizes pner-parameterized neural networks. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019b.
John C. Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. J. Mach. Learn. Res., 12:2121-2159, 2011.
Fartash Faghri, David Duvenaud, David Fleet, and Jimmy Ba. A study of gradient variance in deep learning. CoRR, abs/2007.00 00522007.0443,, 2020.
Stefan Falkner, Aaron Klein, and Frank Hutter. BOHB: robust and efficient hyperparameter optimiza- tion at scale. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, pp. 1436-1445, 2018

---

Under review as a conference paper at ICLR 2021
Jiemin Fang, Yuzhu Sun, Kangjian Peng, Qian Zhang. Yuan Li, Wenyu Liu, and Xinggang Wang. Fast neural network adaptation via parameter remapping and architecture search. In 8th International Conference Learning Representations. ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, 2020.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statisties, AISTATS 2010, Chia Laguna Resort, Sardinia, Italy, May 13-15, 2010, pp. 249-256, 2010.
Boris Hanin and David Rolnick. How to start training: The effect of initialization and architecture. In Advances in Neural Information Processing Systems 31: Annual Conference Neural Information Processing Systems 2018, NeurIPS 2018, 3-8 December 2018, Montréal, Canada, pp. 569-579, 2018.
Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J. Tibshirani. Surprises in high- dimensional ridgeless least squares interpolation. CoRR, 0tbs/1903.0855. 2019.
Chaoyang He, Murali Annavaram, and Salman Avestimehr. Fednas: Federated deep learning via neural architecture search. CoRR, has/2004.08546 2020a.
Chaoyang He, Haishan Ye, Li Shen, and Tong Zhang. Milenas: Efficient neural architecture search via mixed-level reformulation. CoRR, abs/2003. 12238, 2020b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pp. 1026-1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 770-778, 2016.
Dan Hendrycks and Kevin Gimpel. arXiv: 08415, 2016.
Gaussian error linear units (gelus). arXiv preprint
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8):
1735-1780, 1997.
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pp. 2261-2269, 2017a
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017b.
Sergey Toffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, pp. 448-456, 2015.
Arthur Jacot, Clément Hongler, and Franck Gabriel. Neural tangent kernel: Convergence and generalization in neural networks. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, 3-8 December 2018, Montréal, Canada, pp. 8580 8589, 2018
Yufan Jiang, Chi Hu, Tong Xiao, Chunliang Zhang, and Jingbo Zhu. Improved differentiable architecture search for language modeling and named entity recognition. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong. China, November 3-7, 2019, pp. 3583-3588, 2019.
10

---

Under review as a conference paper at ICLR 2021
Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-normalizing neural networks. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pp. 972-981, 2017.
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http://yynn.leccrk. http:///www....oo/////m/tt/ee 2, 2010.
Jachoon Lee, Lechao Xiao, Samuel S. Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' A'Achhé-Bi Buche-Buc, Emily B. Fox, and Roman Garnett (eds.). Advances Neural Information Processing Systems 32: Annual Conference Neural Information Processing Systems 2019, NeurlPS 2019, 8-14 December 2019, Vancouver, BC, Canada, pp. 8570-8581, 2019.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv: 160.:0660.0064500 2016.
Guohao Li, Guocheng Qian, Itzel C. Delgadillo, Matthias Müller, Kl Thabet, and Bernard Ghanem. SGAS: sequential greedy architecture search. CoRR, abs/ abs/1992.00115, 2019.
Liam Li and Ameet Talwalkar. Random search and reproducibility for neural architecture search. In Proceedings of the Thirty-Fifth Conference Uncertainty in Artificial Intelligence, UAI 2019, Tel Aviv, Israel, July 22-25, 2019, pp. 129, 2019.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: differentiable architecture search. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019.
Dmytro Mishkin and Jiri Matas. All you need good init. 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
Toan Q. Nguyen and Julian Salazar. Transformers without tears: Improving the normalization of self-attention. CoRR, abs/ 1b1//190,,5589,, 2019.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. Proceedings of the 30th International Conference on Machine Learning, ICML 2013, Atlanta, GA, USA, 16-21 June 2013, pp. 1310-1318, 2013.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V. Le. Regularized evolution for image classifier architecture search. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 February 1, 2019, pp. 4780 4789, 2019.
Tim Salimans and Durk Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. In Advances in Neural Information Processing Systems, pp. 901-909, 2016.
David R. So, Quoc V. Le, and Chen Liang. The evolved transformer. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, pp. 5877-5886, 2019.
Ilya Sutskever, James Martens, George E. Dahl, and Geoffrey E. Hinton. On the importance of initialization and momentum in deep learning. In Proceedings of the 30th International Conference on Machine Learning, ICML 2013, Atlanta, GA, USA, 16-21 June 2013, pp. 1139-1147, 2013.
Dmitry Ulyanov, Andrea Vedaldi. and Victor S. Lempitsky. Instance normalization: The missing ingredient for fast stylization. CoRR, abs/1607 00002202 2016.

---

Under review as a conference paper at ICLR 2021
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference Learning Representations. ICLR 2019, New Orleans, LA, USA, May 6-9, 2019.
Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach. Learn., 8:229-256, 1992.
Yuxin Wu and Kaiming He. Group normalization. In Computer Vision ECCV 2018 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part XIII, pp. 3-19, 2018.
Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter. Nas- bench-101: Towards reproducible neural architecture search. In Proceedings of the 36th Interna- fional Conference Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, pp. 7105-7114, 2019.
Shan You, Tao Huang, Mingmin Yang, Fei Wang, Chen Qian, and Changshui Zhang, Greedynas: Towards fast one-shot NAS with greedy supernet. CoRR, abs/2003. 1236, 2020.
Arber Zela, Thomas Elsken, Tonmoy Saikia, Yassine Marrakchi, Thomas Brox, and Frank Hutter. Understanding and robustifying differentiable architecture search. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, 2020.
Hongyi Zhang, Yann N. Dauphin, and Tengyu Ma. Fixup initialization: Residual learning without normalization. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019.
Peilin Zhao and Tong Zhang. Stochastic optimization with importance sampling for regularized loss minimization. In Francis Bach and David M. Blei (eds.), Proceedings of the 32nd International Conference Machine Learning, ICMI. 2015, Lille, France, 6- 11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pp. 1-9. JMLR.org. 2015.
Dongzhan Zhou, Xinchi Zhou, Wenwei Zhang, Chen Change Loy, Shuai Yi, Xuesen Zhang, and Wanli Ouyang. Econas: Finding proxies for economical neural architecture search. CoRR, abs/2001 Abs/2001.01233, 2020.
12


Reviews:
[{"review_id": "c9eDmQdA7WA", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper consists of two parts, where it first studies the difficulty of neural network training, then proposes an efficient criterion for neural architecture search (NAS). In the first part, the authors propose two criteria, the average gradient absolute values and the average gradient correlations (across pairs of examples). It is claimed that both quantities are import for network training. In the second part, they propose to use gradient correlation as the reward for NAS, where it is assumed that the larger the gradient correlation the better the architecture is. \n\npros:\n\n+The idea of using a kernel based metric as an efficient NAS criterion is interesting and have great potentials to simply expensive NAS procedures.\n\ncons:\n\n-The first part of the paper is intended to motivate the value of the gradient correlation, but it does not do a great job IMO. For one, the analysis of the gradient value part is not very interesting and seems obvious (thus redundant). It is also not super convincing to me that gradient correlation is either necessary nor sufficient for the model to achieve good test accuracy. More empirical results on this side will help, eg varying the dataset, base architecture and the diversity of model variants. \n\n-Related to the previous point, is there a \"theoretical\" motivation for looking at gradient correlation? \n\n-When studying the difficulty of training, it is extremely important to look at the training loss, rather than only showing the test accuracy. Fig 3 needs to include the training curves, otherwise it is hard to tell difficulty of training apart from generalization ability.\n\n-The experimental results on NAS are not very convincing. All the numbers reported seem to be pretty weak in Table 2, including the baselines and the NAS variants. More specifically, on cifar10, none of the NAS variants outperform the resnet baseline. On imagenet, the models considered are too weak to be interesting (we know that a simple resnet18 model achieves close to 70% top1 acc). The results (both absolute and relative to baseline ones) need to be much improved.\n\nIn summary, I like the main idea of using the kernel statistics as a proxy for efficient NAS, but the paper needs to improve both the presentation and results to make it a significant contribution. Based on the current version, I vote for rejecting.\n\n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "FfdLsCqgG70", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper defines a new scoring function for efficiently selecting between different network architectures, without requiring the expensive training of the models.  This scoring function is based on the expected dot-products between the stochastic gradient vectors corresponding to different pairs of training examples.  The paper shows that this method can much more quickly select between different architectures than previous network architecture search (NAS) algorithms, while attaining relatively similar performance.\n\nStrengths\n- Intuitively, it makes sense that higher correlations between the stochastic gradient updates will lead to less noisy updates, and thus hopefully result in a smoother optimization process and a better final model.\n\nWeaknesses\n- The main weaknesses of the paper in my opinion are lack of clarify and unconvincing empirical (and theoretical) results.\n- The scoring function used for network selection is never clearly defined.  The text in section 5 simply says “average of the Gram matrix of gradient, i.e., P in Eq 1”, which is vague because P is a 3-dimensional tensor.   Furthermore, in the second paragraph of Section 5 it states that the gradients are normalized by “subtracting the average of sample gradients before calculation”, which is a very meaningful modification to the above-mentioned definition, which requires justification and explanation.  Lastly, it is claimed in the intro and throughout the paper that the metric unifies “gradient correlations” and “gradient values”, but this is never really explained.\n- The proposed metric is not normalized in any way.  This suggests that larger models should automatically expect to have higher metric values. \n- It seems the central argument of the paper is that there is a strong positive correlation between the proposed metric and the final test accuracy of the model.  However, the empirical validation of this result is unconvincing; the dots in Figure 4 seem very scattered/random, and the choice the zoom in at two different granularities for the x-axis in this plot is very mysterious—how were the x-axis ranges chosen?  It seems arbitrary.  And even with these hand-picked choices of x-axis, the positive correlation looks quite bad.  To make this convincing, it would be important to compute the actual correlations, and corresponding p-values, to see if these trends are significant.\n- The take-aways from Figure 2 are quite unclear in my opinion.  Also, the figures are not clearly labeled.\n- The claims that “GT-NAS is universal to different architectures” and that it is “universal to different initialization approaches” seem way too strong, given the limited empirical evidence.\n- It was unclear to me how the math in Section 4.1 explains why the proposed metric is a good idea.  In order to better justify this metric choice, I would have hoped for a theorem formally analyzing in a simplified setting why this metric was a good idea.\n\nOverall, due to the lack of clarity and unconvincing results, I recommend rejection for this paper.", "extra_scores": {}, "extra_text": {}}, {"review_id": "5raflgxMmIm", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper shared some interesting observations about gradient values and correlations and utilized these findings in the context of neural architecture search to improve the search efficiency. To be specific, they use the gradients at initialization to filter out many \"bad\" architectures and then fully train the rest to find the best one.\n\nPros:\n- The analysis of gradient correlations looks interesting to me, while I'm not sure whether it has been studied in previous works.\n- The proposed approach is a kind training-free NAS method, which is an interesting direction in NAS.\n- A comprehensive appendix.\n\nCons:\n- The presentation should be improved. After reading the introduction, I'm still unclear about the proposed approach; if there could have a figure to explain some high-level concept of the proposed gradient-based kernel approach, it would be helpful to understand the paper.\n- To make the paper self-contained, I would suggest the authors include the technical details in the main paper, such as how to sample?\n- The proposed method is highly related to nas-without-training, where both of them can somehow estimate the quality of an architecture without the need of training. Therefore, I would encourage the authors compare the proposed method with https://github.com/BayesWatch/nas-without-training in a fair setting.\n- The proposed method needs to sample-s-models and train-k-models. For a relatively small search space, it can be efficient. However, on a large search space such as 10^20 candidates, I feel it is much worse than differentiable methods. Coould the authors comment on that?\n- A following question is the generalization ability of the proposed method on other search space. Would the authors try the proposed method on other search spaces, such as the size search space in NATS-Bench (an extension version of NAS-Bench-201) or NAS-Bench-101?\n\nSome minor issues:\n- In the second paragraph of Page 2, the reference for NAS-Bench-201 is incorrect, where the cited is NAS-Bench-101.\n- In the \"NAS\" paragraph in the related work section, \"Ying et al. (2019) propose a reinforcement learning based search policy\" the reference seems incorrect, Ying et al. (2019) is NAS-Bench-101 instead of the RL-based NAS algorithm.\n- In the paragraph, the authors said \"the dependence on validation accuracy still leads to huge computation costs.\", which is incorrect. The computation of validation accuracy is not expensive (depends on how to compute it). The huge computation costs are because that those approaches will need to fully train and evaluate each candidate architecture to get their validation accuracy. Also, the efficient search is due to the weight sharing approach, where the differentiable approach is a kind of popular direction.\n- Therefore, I would suggest the authors revise the related works to fix these misleading parts and inaccurate references.\n\n**Post Rebuttal**: The authors did not provide any responses, so I decreased the scores.", "extra_scores": {}, "extra_text": {}}, {"review_id": "2ZxT2WwtMX1", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper pointed out that both gradient correlations and gradient values have strong impacts on model training, which is an insightful finding. Based on this finding, they explore a simple yet effective network architecture search (NAS) approach. The new approach replaces the expensive “train-then-test” evaluation paradigm with a new lightweight function according to the gradient-based kernel at initialization. Experiments show the high speed of the approach.\n\nPros:\nThe finding that gradient correlations have strong impacts on model training is interesting to some extent\n\nThe idea that using gradient correlations to evaluate architectures is incremental, which breaks the limitation of the ‘train-then-test’ framework. As a result, it can help us take a step towards high-speed NAS.\n\nConcerns:\nThe key concern about this paper is the lack of rigorous experimentation to study the usefulness of the proposed method. For a set of architectures, it is very important for a NAS method that the correlation between their ranking by the validation accuracy and their ranking by a NAS method. However, I did not see such an experiment in this paper. Further, comparison methods are not new enough. There is even no 2020’s method, even though there is a lot of excellent work in 2020 such as ISTA-NAS, PD-DARTS, which makes the experimental results not convincing enough. \nThe second concern is that the finding that gradient correlations have strong impacts on model training is similar to works such as SynFlow. I think it seems that the idea is not innovative enough. It would be better if the authors would compare these works thoroughly.\n\nThe third concern is about the figures and tables in the part of the experiment. For example, Table 1 reports that DARTS can achieve an accuracy of 88.32% on CIFAR10. However, our group also reconducted DARTS and achieve an accuracy of ~97%, which is similar to the results reported in many papers (e.g. DARTS, PC-DARTS). So I have a little doubt about the results in the table. For another example, figure 4 seems can only show the very limited correlation between kernel score and accuracy. Moreover, the two granularity here is hard to understand. I hope that you can explain it more specifically.\n\nThe organization of section 7 is a bit messy. For example, you list several search spaces in section 7.1, but you do not report how to use these search spaces. Are you use them separately or you aggregate them? \n\nMinor comments:\nTable2: It seems that you forget to report which is the “baseline”.\nEqution1: It would be more nice to see the definition of j.\nIntroduction:with largest kernels ->with the largest kernels\nSection 4.2:a learning rate 1.0 -> a learning rate of 1.0\nSection 6.1: enables researches to -> enables researchers to\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new criterion for neural architecture search that does not require the expensive step of training the model. The reviewers found the proposed approach of relying on gradient statistics promising. However, the reviewers found that the clarity of the paper needs to be improved and that the empirical evidence is too limited to support some of the claims.
Decision: Reject