Sweet Gradient Matters: Designing Consistent and Efficient Estimator for Zero-Shot Neural Architecture Search
Authors: Longxing Yang
Yanxin Fu
Shun Lu
Zihao Sun
Jilin Mei
Wenxiao Zhao
Yu Hu
Venue: Submitted to ICLR 2023 (2023)
Keywords: Neural Architecture Search
Zero-Shot
Estimator
Sweet Gradient

Abstract: Neural architecture search (NAS) is one of the core technologies of AutoML for designing high-performance networks. Recently, Zero-Shot NAS has gained growing interest due to its training-free property and super-fast search speed. However, existing Zero-Shot estimators commonly suffer from low consistency, which limits the reliability and applicability. In this paper, we observe that Sweet Gradient of parameters, i.e., the absolute gradient values within a certain interval, brings higher consistency in network performance compared to the overall number of parameters. We further demonstrate a positive correlation between the network depth and the parameter ratio of sweet gradients in each layer. Based on the analysis, we propose a training-free method to find the Sweet Gradient interval and obtain an estimator, named Sweetimator. Experiments show that Sweetimator has superior consistency to existing Zero-Shot estimators on four benchmarks with eight search spaces. Moreover, Sweetimator achieves state-of-the-art performance on NAS-Bench-201 and DARTS search spaces.

Under review as a conference paper at ICLR 2023
SWEET GRADIENT MATTERS: DESIGNING CONSISTENT AND EFFICIENT ESTIMATOR FOR ZERO-SHOT NEURAL ARCHITECTURE SEARCH
Anonymous authors Paper under double-bling review
ABSTRACT
Neural architecture search (NAS) is one of the core technologies of AutoML for designing high-performance networks. Recently, Bero-Shot NAS has gained growing interest due to its training- free property and super-fast search speed. However, existing Zero-Shot estimators commonly suffer from low consistency, which limits their reliability and applicability. In this paper, we observe that Sweet Gradient parameters, i.c., the absolute gradient values within a certain interval, brings higher consistency in network performance compared to the overall number of parameters. We further demonstrate a positive correlation between the network depth and the proportion of parameters with sweet gradients in each layer. Based on the analysis, we propose a training-free method to find the Sweet Gradient interval and obtain an estimator, named Sweetimator. Experiments show that Sweetimator has superior consistency compared to existing Zero-Shot estimators in four benchmarks with eight search spaces. Moreover, Sweetimator outperforms the-art art Zero-Shor estimators in NAS- JAS-Bench-201 and achieves competitive
state-of -f- performance with 2.5x speedup the DARTS search space.
INTRODUCTION
The computer vision field has witnessed the great success of deep learning. Iconic works such as ResNet (He et al. 2016), MobileNet (Howard et al. 2017: Sandler et al., 2018), and EfficentNet (Tan Le| 2019) are widely applied for variety of real-world tasks such as object detection and semantic segmentation. To tackle the trail-and-eree shortcomings of handerafted architectures, Neural Architecture Search (NAS) (Elsken et al. 2019) has been proposed to automatically search powerful networks that even outperform manual designs (ooh al. 2018).
major theme in NAS development is efficiency. From this perspective, NAS can be broadly classified into three categories: All-Shot, One-Shot, and Zero-Shot NAS. All-Shot NAS utilizes approaches such as reinforcement learning (Zoph & Le| (017) or evolutionary algorithms (Real et al. 2019) to train the sampled architectures one by one during the search process, which costs hundreds or even thousands of GPU days. Based on weight sharing (Pham et al. 2018). One-Shot NAS trains one supernet and utilizes sampling-based (Guo al. 2020; Chu al. 2021b; Yu et 2020 or gradient-based (Liu et al.) 2019: Chen et 20199 Xu et al. 2020) approaches, thus reducing the search cost to few GPU days. Zero-Shot NAS leverages training-free estimators (Mellor al., 2021 Abdelfattah et al.] 2021) to evaluate network performance. As no networks are trained, the search time is reduced to a few GPU hours or even seconds.
However, Zhro-Shot NAS commonly suffers from low consistency. Figure illustrates the Spearman's 's rank between the test accuracy obtained by training the network from scratch and the estimated performance score of mainstream Zero-Shot methods in BeAS-Bench--01 (Ying al. 2019). NAS- Bench-201 (Dong & Yang 2020; Dong et al.) 2022), and PAAS-Bench-301 (Zela et al. 2022). The results demonstrate that these methods do not consistently outperform the simple metric of the number of parameters, which limits their reliability and applicability. Moreover, a question is also naturally raised: could we find a Zero-Shot estimator with superior consistency to parameters?
For the networks in the MAS-Bench-100 101, the NAS-Benchh Bench-201, and the observe that some specific parameters, whose absolute gradient values are in certain interval, have
space, we

---

Under review as a conference paper at ICLR 2023
NAS Bench-101
NAS Bench-201
NAS Ba---Bench- 301 -- Spearman's rank of Farameters
Spcarman's rank
04
0.2
PLANy Parametery
Rioeer
Parametor
Sovetimatt
Iisden
Mear
den
cractian
Zero-Shot Estimators
Figure 1: The Spearman's 's rank correlation coefficient of Shcro-Shot estimators NAS- AAA--Benc--lle 101, NAS-Bench-201, and NAS-Bench-301. The dotted line indicates the spearman's rank of Parameters.
stronger consistency with the network performance than the overall number of parameters (Parameters for short). For the sake of brevity, named the gradient in the such interval Sweet Gradient. We found an interesting property of Sweet Gradient that the proportion of parameters with Sweet Gradient in each layer is positively correlated with the depth of the network. Based this property, we propose Sweetimator, an estimator that computes Sweet Gradient interval without training. Figure shows that Sweetimator outperforms the Parameters estimator and achieves the best consistency in all three benchmarks.
The contributions of this work are:
We observe the Sweet Gradient phenomenon, i.e., the number of parameters with absolute gradient values certain interval has better performance consistency than Parameters.
We demonstrate that there is positive correlation between the network depth and the proportion of parameters with Sweet Gradient in each layer.
We propose a simple and effective Zero-Shot estimator, Sweetimator, that can find Sweet Gradient intervals without training.
In the consistency experiments, Sweetimator outperforms the existing Zero-Shot estimators in four benchmarks with eight search spaces. the search experiments, Swectimator has superior performance to state-of-the-an Zero-Shot estimators in NAS-Bench-201 and achieves competitive results with 2.5x speedup the DARTS search space.
2 RELATED WORK
Neural Architecture Search. Neural architecture search aims at automatically designing the best- performing network for specific task. In the early days, Zoph & Le (2017) proposed reinforcement learning framework to search hyper-parameters of an entire network. Inspired by the modular design paradigm of handcrafted neural networks, NASNet (Zoph al. 2018 searched cell structures and stacked the searched best normal cell and reduction cell to form a network. Subsequently, Pham al. 2018) proposed greight-sharing strategy reduce the search overhead a few GPU Days. Afterward, sampling-basee
approaches (Guo et al. 2020 Chu et al. 2021b; Yu et al. (020) trained the supernet by path sampling and utilized sub-networks accuracy for evaluation. DARTS (Liu et
2019) and its variants (Chen et al.) 2019 Xu et al. 2020: Zela et al. 2020 Chu al. 202 Wang ctt al. 2021 Sun et al. 2022) leveraged differentiable strategies to optimize the supernet and select the final architecture.
Non-Zero-Shot Estimator. To facilitate the performance evaluation process, various estimators have been proposed. It is natural to use the validation loss accuracy (Zoph & Le 2017: Real et al.) 2019 Liu et al.] 2018) as a performance estimator. Subsequently, SPOS (Guo et al. (020) and similar
works (Pham al. 2018 Yu al. 2020 Chu et al. 2021b) utilize the accuracy of sub-networks as
2

---

Under review as a conference paper at ICLR 2023
proxy for efficient evaluation. Another path for estimators is to utilize machine learning models to predict network performance. NAO (Luo et al. 2018) utilizes an encoder and estimator to find high-performance network. WWen et al. (2020); ; Chen et al. (202lc) utilized graph convolutions networks to regress network performance. GP-NAS (Li et al. 2020) proposed a gaussian process based NAS method obtain the correlations between architectures and performances. TNASP (Lu et al. 2021) uses a transformer and self-evolutionary frameworks to predict network performance. Although the above works can effectively evaluate the network performance, large search overhead still exists with training networks.
Zero-Shot Estimator. Zero-Sho estimators are applied search architectures without training.
NASWOT (Mellor et al., (2021) utilizes activations of rectified linear units to evaluate networks with random initialization. Abdelfattah et al. (2021) proposes series of zero-cost proxies based
on pruning literature (Lee et al. 2019b Wang et al., 2020 Tanaka et al. 2020). TE-NAS (Chen
al.
2021a) combines neural tangent kernel (NTK) (Lee et al., 2019 and linear region (Raghu
ct al. proposes Zen-Score to measure network performance based on network expressivity. KNAS
2017) evaluate the trainability and expressivity of networks. Zen-NAS (Lin et al.
1021
(Xu
et al., 2021) finds that the gradient kernel of the initialized network correlated well with training loss and validation performance. Zhhng & Jial (2022) analyzes the sample-wisg wise optimization landscape and proposes GradSign for performance evaluation. A remarkably efficient search facilitated by the zero-shot estimators. However, compared to network parameters, these methods are not competitive in terms of ranking consistency, raising concerns about their applicability.
3 METHOD
3.1 PRELIMINARIES
Zero-Shot NAS utilizes network information at initialization for scoring and expects strong rank consistency with the performance. In particular, gradient information is widely adopted in Zero-Shot estimators. For example, SNIP (Abdelfattah al. (021) applies gradient values to approximate the change of loss, and GradSign (Zhang & Jia 2022) calculates the gradient conflict between data samples. Considering that the network parameter is a strong performance estimator as shown in Figure we therefore combine the network parameter with the gradient. Suppose the loss function J, the network parameters are E I and corresponding average gradients in a mini batch are V E R". Then the Zero-Shot estimator to be explored is as follows.
173
OJn (09) 00g
Score(thr thrz)
Σn thr S k=1
thrs
(I)
where thr1, thr2 2 O are two thresholds, 8 the network initialization parameter, is the indicator function with value of 1 when the condition is true and O otherwise. Equation (1) describes the number of parameters whose absolute gradient values are within a certain interval. Score(O, +o0) represents the overall network parameters. For the sake of brevity, we utilize [thr], thrr) to describe a interval. We provide theoretical analysis for the proposed Zero-Shot estimator in Appendix A
3.2 SWEET GRADIENT
We then analyze the rank consistency of Equation [ with different [thr,, thrz) intervals. Figure illustrates the Spearman's 's rank between the test accuracy of one hundred networks and their scores of Equation T on MAS-Benchh-101 and MAS-Bench-225 We can see the following patterns:
2
The Spearman's 's rank in the upper right corner of the heatmap can be regarded as the consistency of Parameters which are covered overwhelmingly when thr; 0 and thr, 5.
The score with interval [thr1, thr2) in the dark blue area yields better rank consistency than Parameters. We name the gradient in such intervals Sweet Gradient. The Sweet Gradient Interval is defined as: the absolute gradient interval where the number of parameters have better performance consistency than Parameters.
3

---

Under review as a conference paper at ICLR 2023
5e:10
le:s
56-50
05
004
Se-dk
04
10:01
5c
3u-07-
044
Se
-2
0.0011
0.0101
5.0001
0.0225
0001
0.02
0.013
0.201
0.0
605
0.1
011
0.5
05-
10-
Se30
1
1-
5-
00001-
00005
on0
5.01
5.0%
03
S
10:3
0,0023
02003
00011
thr2
(a) NAS-Bench-101
(b) NAS-Bench-201
Figure 2: The Spearman's 's rank under different intervals on MAS--ench-Ile 10c and NAS-Bench- 201 (CIFAR10), Each rank is calculated with 100 architectures and a batch size of 128.
Although Figure 2 only shows the Spearman's 's rank of one hundred architectures two search spaces, Sweet Gradient exists across different search spaces, datasets, the number of architectures, batch size, and initializations (please refer to Appendix [B] for more details). This observation indicates that a proper threshold setting can obtain an estimator with better rank consistency than the overall network parameters. However, the thresholds derived based on architecture accuracy still need network training. Does there exist method obtain two thresholds without training? This motivates us to investigate the underlying reason for the Sweet Gradient phenomenon.
Reviewing the development of neural networks, depth one of the most critical factors affecting performance (Simonyan & Zisserman 2013). Inspired by He et al. (2016); Balduzzi et al. (2017). we dissect the parameters of the different intervals from the perspective of depth. To eliminate the effect of the parameter magnitude, the parameter proportion is used for analysis:
Proportion: (core thr2)
Scorer +00)
where Proportion, is the proportion of parameters with the gradient interval [thr,, thr2) in the (-th layer, whose range is 10, 1]. Figure exxibiiss the parameter proportion under different layers in NAS- Bench-101 and NAS-Bench-201. More figures can be seen in Appendix Surprisingly, the parameter proportion in the Sweet Gradient interval increases as the depth increases, e.g., [le-7, 5e-7) in NAS-Bench-101 and [0.0005, 0.001) in NAS-Bench-201. In contrast, the parameter proportion in the Non-Sweet Gradient interval decreases with increasing depth or has a large variance, e.g., [0.01, 0.05) in NAS-Bench- 101 and [le-8, 8, 5) 3) in TAS-Bench-200 Another interesting observation that the parameter proportion tends to increase for small gradients and decrease for large gradients with increasing depth, e.g.. [1e-5, 5) and 0.5)
MAS-Bench-18
Therefore, there is positive correlation between network depth and parameter proportion of Sweet Gradients in each layer. We can seek Sweet Gradient based on the property: Sweet Gradient is more likely to be located in intervals where the parameter proportion increases network depth increases.
Besides the network depth, we also investigated the gradient and activation distribution but found no correlation between the two factors and Sweet Gradient. Please refer to Appendix D for more details.
3.3 SWEETIMATOR
Sweetimator represents the score in Equation [ when [thr1, thr2) is a Sweet Gradient interval. The discovered property answers the motivational question of how obtain the Sweet Gradient interval without training, and thus the Sweetimator. Heuristically, we propose sweetness to estimate how sweet the gradient of an interval is:
4

---

Under review as a conference paper at ICLR 2023
Sweet: (1e-7. Se-7) p-0.60 > 0.39
61 p-0.59 0.39
Non-Sweet: 10.01. 0.05)
Non-Sweett
pw p- 0.39
pa-0.43
20:00
popo 15n
00:0
0:15
5415
0.2
BKE
0MK
Septi
depth
depth
depth
(a) NAS-Bench-100
Sweet: [0.0001, 0.0006)
Sweet: (0 0005, 0.001)
Non-Swed: [1e-8, 5e-8)
Non-Sweel: 10.01. 0.05)
pn0.91 > 0.75
p=0.51 > 0.71
p=0.2 i 0.71
p=0.47 < 0.71
311
03175
025
6.1%
4150
512
0.354
0.0
413
03575
699
342
083
080
(b) NAS-Bench-201
Figure 3: The parameter proportion along cell depth of Sweet and Non-Sweet Gradient intervals NAS-Bench-Ioe 101 and JAS-Bench-201 (CIFAR-10), Each layer represents a cell the network and there 9 and 15 layers in NAS- ASS-Bench-101 and NAS- NAS--ench-2001, respectively.
Sweetness(thr thr2)
sign(Proportion..1 Proportion.)
where n is the number of architectures, 4s denotes the depth of the k-th architecture, and the sign function is used to determine whether the parameter proportion is ascending the depth increases. Equation 3 has range of [-1, indicating interval incrementality with the depth. Sweetness = indicates monotonically ascending and Sweetness Although both Sweetness and GradSign utilize the sign function, the purposes are completely different. GradSign evaluates the gradient conflict between samples, while evaluate the interval incrementality for Sweetimator.
indicates monotonically descending.
In practice, one problem in obtaining the Sweet Gradient is the interval setting, which can be described by the triplet (mar, min, split). max means the largest order of magnitude, min means the smallest order of magnitude except zero, and split means each order of magnitude divided equally into several parts. For example. (1, 21) means three interval [0, 0.1), [0.1, 0.5), [0.5, 1.0).
We empirically found that mar 10 and min = 1e-10 are sufficient. And the ablation study of split is provided in Section Algorithm describes how to obtain Sweetimator.
Algorithm Sweetimator
Input: m initialized architectures; p intervals based on (mar, min, split); Output: Best Interval for Sweetimator
1: marSweetness = -1
2: bestInterval None
3: for i do
4: Sureetness; Equation
5: if Sweetnesss > max-Sweelness then
6:
mazSweetness Sweetness; 8: end
7:
best/ntereal = Intercal;
9: end for
return best. rnteraal
5

---

Under review as a conference paper at ICLR 2023
4 EXPERIMENT
In this section, we first evaluate the ranking consistency of Sweetimator in NAS-Bench-101 Ying al. 2019), NAS-Bench-20) (Dong Yang 2020 Dong al. 2022). NAS-Bench-300 (Zela et al. 2022). and NDS (Radosavovic et al. 2019) benchmarks. Then, we use Sweetimator to conduct search experiments NAS-Bench-200 and DARTS (Liu et al., 2019) search spaces. Finally, we provide ablation experiments for further analysis. Experimental details can be referred to Appendix
4.1 CONSISTENCY RESULTS
NAS-BNecc-BS3 and NDS search spaces. NAS-Bench- 101 the first large-scale NAS Benchmark with 423k architectures and corresponding accuracy CIFAR-10. NAS-Bench-200 201 is extension to
Benchmarks. We conduct experiments in NAS-Bench- 101,
containing 15625 architectures and CIFAR-10, CIFAR- and ImageNet16- 120 datasets. NAS- Bench-301 is the first surrogate NAS benchmark, with 10 architectures statistically analyzes multiple network design spaces, including NAS-Net (Zoph et al. 2018), Amoe- baNet (Real al.) 2019), PNAS (Liu 2019) search space. And each space has thousands of architectures in the NDS benchmark.
CIFAR-10. NDS
2018), ENAS (Pham et al. 2018). and DARTS (Liu et al.)
Baselines. We compare common Zero-Shot estimators, including SNIP (Lee al. 2019b), GraSP (Wang et al., 2020). SynFlow (Tanaka et al. 2020), Fisher (Trrner et al. 2020). GradNorm (Abdelfat
tah 2021). NASWOT (Mellor al. [021). TE-NAS (Chen al. 2021a). Zen-Score (Lin et al.
2021). GradSign (Zhang & Jia 2022). and FLOPs and Parameters of networks. A comparison with
Non-Zero-Shae estimators is in Appendix
Settings. The consistency experiments are divided into two groups. The first group contains NAS- and NAS-Bench-300 with 4500, 15625, and 5000 architectures to test consistency, respectively. For fairness, the batch size of all Zero-Shot estimators is 64. The evaluation metric Spearman's rank. The second group includes NAS-Net, AmoebaNet, PNAS, ENAS, and DARTS spaces of NDS benchmark with 4846, 4983, 4999, 4999, and 5000 architectures for evaluation, respectively. The architectures are trained on CIFAR 10. All Zero-Shot estimators have batch size of 128. The evaluation metric is Kendall's Tau.
Bench-101. 101,
Results. Tables T and 2 show that Sweetimator significantly outperforms existing Zero-Shot estima- tors. For example, Sweetimator is better than the estimator Parameters by 46% (0.423 vs. 0.618) in NAS-Bench-ID 101 in Ta The results verify the effectiveness the proposed method. Moreover, observe that existing Zero-Shot estimators do not consistently outperform Parameters in terms of rank consistency, which is also found by Ning et al. (202T). In contrast, Sweetimator significantly outperforms Parameters, suggesting that Sweet Gradient is direction worth exploring for NAS.
Table 1: Rank Consistency of Zero-Shot estimators in NAS-Bench-100 101, NAS-Bench-301 by Spearman's 's rank.
and
NAS-Bench-101
NAS-Bench-200
NAS-Bench-301
Estimators
CIFAR-10
CIFAR-10
CIFAR-100
ImageNet16-120
CIFAR-I0
Fisher
0,302
0.546
0.548
0.491
0,392
SNIP
0.191
0.638
0.637
0.578
0.153
GraSP
0.329
0.551
0.549
0.553
0.379
GradNorm
0.265
0.637
0.637
0.578
0.155
SynFlow
0.360
0.777
0.763
0.751
-0.167
TE-NAS
0.065
0.376
0.350
0.335
-0.055
Zen-Score
0.261
0.251
0.260
0.319
-0.119
NASWOT
0.327
0.691
0.704
0.700
0.053
GradSign
0.422
0.808
0.792
0.783
0.424
FLOPs
0.422
0.733
0.708
0.673
0.423
Parameters
0.423
0.751
0.727
0.690
0.458
Sweetimator
0.618
0.888
0.859
0.835
0.596

---

Under review as a conference paper at ICLR 2023
Table 2: Rank Consistency of Zero-Shot estimators in five search spaces of NDS by Kendall's Tau.
Estimators
DARTS
ENAS
PNAS
NASNet
Amoeba
GradNorm
0.227
0.055
0.109
-0.080
-0.116
SynFlow
-0.001
-0.092
-0.090
-0.191
-0.001
NASWOT
0,480
0.387
0.363
0.299
0.208
GradSign
0.537
0.424
0.396
0.290
0.250
FLOPs 0.500
0.413
0.395
0.288
0.238
Parameters
0.493
0.411
0.387
0.289
0.241
S Sweetimator 0.568
0.506
0.458
0,449
0.360
Table 3: Mean std accuracies in NAS-Bench-201. The upper and lower parts indicate results of Architecture Selection and Assisted NAS, respectively. All searches for 500 times. For fair comparison, architectures are searched on CIFAR-10 and evaluated on CIFAR-10, CIFAR-100 and ImageNet16-120. Note that GradSign-assisted NAS experiments using their released code.
CIFAR-10 10
CIFAR-100
ImageNec16-120
Methods
validation
validation
validation
test
Architecture Selection
Random
13.20
13.41
10.70+12..55
60.83++1.58
33.34:19.39
33.13:+9.66
NASWOT (N=10)
89.14=1.44
92.44=1.13
68.50-12.03
68.62-+2.04
41.09
41.31:4.111
Sweetimator (N=10)
1.17
92.75=0.93
68.89
19.02-t1.96
41.54:13.25
41.71++3.34
Optimal (N=10)
89.92+0.75
93.06=00.59
69.61+1.21
1..7
13.11+1.85
43.30+1187
SynFlow (N=100)
89.83=0.75
93. 03.12:0.522
69.89-11.87
1.894
41.9414.13
42.26:14.26
NASWOT (N=100)
89.55=0.89
92.91-10999
1.73
1.74
42.81+3.05
43.10:13.16
GrudSign (N=100)
89.84=0.61
93.31=0047
1..22+1.32
70.33+11.28
42.0742.77
42.42+-2.1
Swectimator (N=100)
9..47=0.74
93.45
70.40+11.54
70,54+1.522
43.4942.10
43.71+2219
Optimal (N=100)
91.05+0.28
93.84+0.23
71.45+0.79
71.56+00.8
45.37+0.61
45.67+0066
NASWOT (N=1000)
89.69-0.73
92.96-0.811
1..98
69.86-+1.21
44.4412.10
43.95:12.05
Sweetimator (N=1000)
9.400=0.48
9346
1.33
71.62-+1.34
44.68
45.05=:1445
Optimal (N=1000)
91.3440.18
94.05=0.222
12.15
72.17=00.83
45.57+0073
45.79++0.78
Assisted NAS
REA
91.08=10.45
93.85:0.44
1..3
71.64:1..25
44.90
1..25::1.11
A-REA
91.20=10.27
71.95=0..9
45.70
G-REA
91.25=#0.57
94.10=0.48
1.556+11.53
12.62++1.46
45.62+1.31
45.77+:1.28
S-REA
91.43++0.24
94.23
1.09
72.96+1.01
45.98+0.77
46.10+-0.69
REINFORCE
90,32-0.89
93.21-10.82
1..0
10.1441..73
43.57-2.09
43.64.12.24
G-REINFORCE
90.78-0068
93.61=0.57
1..8
71.02++1.36
44.53:41.46
44.65-11.58
S-REINFORCE
91.01=0.46
93.76=00.66
1.26
71.44+1.20
44.76++1.22
45.02-+1.37
BOHB
91.8410.49
93.6410.49
70.82+1..29
70.92-11.26
44.36.11.7
44.50
G-BOHB
91.18-0.29
93.96-0.28
71.91
71.96
45.32-10.89
45.51
S-BOHB
91.2110.22
0.29
72.01
72.06:10.76
45.38
45.62-10.85
4.2 SEARCH RESULTS IN NAS-BENCHI-20
Following Mellor al. (02T): Zhang & Jia (022). we conduct architecture selection and assisted NAS experiments evaluate the ability of Sweetimator search gegh-performance architectures.
Architecture Selection Settings. Architecture selection refers to randomly sampling candidate architectures from the search space and then evaluating the architecture with the largest Sweetimator score. The comparison methods include NASWOT, GradSign, Random and Optimal, where Random uniformly selects architectures from the search space, and Optimal means picking the network with the highest test accuracy among N architectures. Following Mellor et al. (2021), we conduct Sweetimator for sample size of N 10, N 100 and N 1000.
7

---

Under review as a conference paper at ICLR 2023
Sweetimator-Assiee NAS Settings. In the experiment, the NAS algorithms of REA (Real et al. 2019), REINFORCE (Williams) 1992) and BOHB (Falkner et al. 2018 are assisted with Sweetimator to verify the applicability. For fair comparison,
utilized the assisted algorithm proposed in GradSign (Zhang & Jia 2022). In particular, the random selection of each NAS algorithm is replaced by Sweetimator-asse assisted selection. The NASWOT-, GradSign-, Sweetimator-assing NAS algorithms are abbreviated as A-, G-, S-algorithm, e.g., S-REA, respectively. Following Zhang & Jia (022). all results are searched with time budget of 12000s.
Results. Table summarizes the search results. In architecture selection experiments, Sweetimator outperforms compared baselines with closer distance Optimal, indicating the ability to search high-performance architectures. In the Sweetimator assisted NAS experiments, the average accuracy of Sweetimator-assinn was higher than the original algorithm and other estimator-assisted algorithms with lower standard deviation, further validating the efficiency and applicability of Sweetimator.
4.3 SEARCH RESULTS IN DARTS SEARCH SPACE
Settings. DARTS (Liu et al. 2019 a popular search space to evaluate NAS algorithms. We conduct experiments CIFAR- 10 Krizhevsky & Hinton 2009) and ImageNet (Olga al. 2015) dataset. In the search phase, we utilize T00 architectures and a batch size of 128 to obtain the best interval of Sweetimator. Further, integrate Sweetimator with REA algorithm (Real et al. 2019 for searching. The hyper-parameters of REA are followed by Dong & Yang (2020) with 200 cycles. Despite small cycles, the score converges rapidly and the search results can competitive with other Zero-Shot methods, which also bring fast search speed. In the retraining phase, we follow DARTS settings to build and train searched network for fair comparison. More details can be referred Appendix
Results. In Table 4 the middle part shows the results on CIFAR-10. Compared to Zero-Shot methods TE-NAS and NASI-ADA, Sweetimator achieves 2.5x speedup (0.01 vs. 0.004) and better performance (2.63 vs. 2.62). When compared to One-Shot methods such as PC-DARTS, Sweetimator has a significantly faster search speed (0.1 vs 0.004). The right part shows the results on ImageNet which also demonstrate that Sweetimator achieves competitive performance and speed compared to other Zero-Sho methods. The searched cells are visualized in Appendix
Table 4: Comparison with state-of-the-ard NAS methods on CIFARIO 10 and ImageNet. The Sweetimator results on CIFAR-10 come from four independent searches with different random seeds. Top-1 and Top-5 represent the test error. GDays is GPU days. t Architectures are directly searched on ImageNet.
$ Time is recorded on GTX 1080Ti.
CIFAR-10
ImageNet
Methods
Test Error (%)
Params (M)
GDays
Top-1 (%)
Top-5 (96
Params (M) GDays
All-Shot NAS
NASNet Zoph et 2018
2.65
2000
26.0
5.3
2000
AmoebaNet Real 2019
3.34.10.06
3.2
3150
24.3
7.6
6.4
3150
PNAS
[Lu
3.41+00.0
3.2
225
25.8
8.1
5.1
225
One-Shot NAS
DARTS (2nd) (Liu et
2019
2.76++0.09
1.0
26.7
8.7
4.7
1.0
(Xie al. 2019
2.85+0.022
27.3
4.3
GDAS [Dong Yane
2.82
0.17
26.0
5.3
BayesNAS
2.81+:0004
0.2
26.5
0.2
P-DARTS Chen 2019
2.5
3.4
0.3
24.4
4.9
0.3
SDARTS Chen Hisieh
2120
2.61:10.02
3.3
1.3
25.2
7.8
5.4
1.3
PC-DART'S Xu
2020
2.57=00.07
0.1
24.2
5.3
3.8
DARTS-
al.
2.5910.08
23.8
4.9
DrNAS Chen et al.
20216
2.54+00.3
4.0
0.4
24.2
7.3
5.2
3.9
3-DARTS (Ye at.
2022
2.53-10.88
0.4
24.2
Zero-Shot NAS
TE-NAS Chen al.
2021a
2.63++0.06
3.8
0.05
24.5
7.5
5.4
0.17
NASI-ADA Shu ata..
2022
2.90:10.13
3.2
0.01
24.8
7.5
5.2
0.01
Sweetimator
2.62+00.05
0.004
24.8
5.2
0.01
8

---

Under review as a conference paper at ICLR 2023
4.4
ABLATION STUDY
Batch size, architecture numbers, and split of the interval are three major factors that influence the gradient calculation and threshold acquisition of Sweetimator. For further analysis, we conduct
ablation studies in NAS-Bench-201, which are shown in Table S
Batch Size. The batch size is an important hyper-parameter that determines the quality of the gradient. The larger the batch size, the more accurate the gradient. Table 5 demonstrates that Spearman's s rank commonly becomes higher with the increasing batch size. Therefore, Sweetimator can be more effective by using a larger batch size.
Architecture Number. The architecture number affects threshold acquisition. Table 5 shows that a larger architecture number does not change Spearman's 's rank. This is because the obtained interval is the same across different architecture numbers. The results suggest that a small number of architectures (e.g.. 100) is enough to obtain the best interval.
Split of Interval. The split influences the granularity of the interval. In Table 5 the Spearman's ranks are improved CIFAR-I0 (0.888 - 0.913) and CIFAR-100 (0.859 0.887) but almost kept on ImageNet16-122 (0.835 + 0.836) when split is larger. Consequently, the interval granularity of different datasets is different, indicating that Sweetimator can be more efficient by designing flexible interval selection algorithm in the future.
Table 5: Rank Consistency of Sero-Shot estimators in FAS-Bench-201 by Spearman's rank. The baseline has batch size of 64, architecture number of 100, and split of 2.
Batch Size
Architecture Number
Split
Dataset
Baseline
128
256
512
150
200
300
3
4
CIFAR-10
0.888
0.908
0.917
0.921
0.888
0.888
0.888
0.913
0.910
0.913
(1FAR-100
0.859
0.881
0.888
0.892
0.859
0.859
0.859
0.886
0.883
0.887
ImageNet16611
0.835
0.815
0.836
0.838
0.835
0.835
0.835
0.835
0.824
0.836
LIMITATIONS AND FUTURE WORK
We only consider computer vision classification tasks and cell-based search spaces in this work. And finding the optimal gradient interval is still a process of discrete selection rather than continuous differentiable optimization process. future works, we intend to apply Sweetimator to diverse search spaces (e.g.. Transformer space) and explore an optimization--fiendly method to obtain the best interval. Besides, it is still mystery that the Sweet Gradient interval exhibits the property of increasing parameter proportion as network depth increases. We believe more profound reason exists behind this observation which helps us better understand deep neural networks. Therefore, theoretical analysis is necessary in the future work.
6 CONCLUSION
This work observes the Sweet Gradient phenomenon that some specific parameters, whose absolute gradient values are certain interval, have a stronger consistency with the network performance than Parameters. To obtain the Sweet Gradient interval without training, investigated the relationship between network depth and Sweet Gradient. We found that Sweet Gradient tends to exist intervals with increasing parameter proportion network depth increases. Based the property, Sweetness to obtain the best interval. Experiments demonstrate that Sweetimator can achieve superior rank consistency and excellent search results, verifying the effectiveness of the proposed estimator.
utilize
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
Mohamed S Abdelfattah, Abhinav Mehrotra, Lukasz Dudziak, and Nicholas D Lane. Zero-cost proxies for lightweight nas. International Conference on Learning Representations, 2021.
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over- parameterization. International Conference on Machine Learning, 2019.
David Balduzzi, Marcus Frean, Lennox Leary, JP Lewis, Kurt Wan-Duo Ma, and Brian McWilliams. The shattered gradients problem: If resnets are the answer, then what is the question? In International Conference on Machine Learning, 2017.
Amir Beck. First-order methods in optimization. SIAM, 2017.
Wuyang Chen, Xinyu Gong, and Zhangyang Wang. Neural architecture search on imagenet in four gpu hours: A theoretically inspired perspective. In International Conference Learning Representations, 202
Xiangning Chen and Cho-Jui Hsieh. Stabilizing differentiable architecture search via perturbation- based regularization. In International Conference on Machine Learning, 2020.
Xiangning Chen, Ruochen Wang, Minhao Cheng. Xiaocheng Tang, and Cho-Jui Hsieh. Drnas: Dirichlet neural architecture search. International Conference on Learning Representations, 2021b.
Xin Chen, Lingxi Xie, Jun Wu, and Qi Tian. Progressive differentiable architecture search: Bridging the depth gap between search and evaluation. In International Conference on Computer Vision, 2019.
Xin Chen, Lingxi Xie, Jun Wu, Longhui Wei, Yuhui Xu, and Qi Tian. Fitting the search space of weight-sharing nas with graph convolutional networks. In Association for the Advancement of Artificial Intelligence, 2021c.
Xiangxiang Chu, Xiaoxing Wang, Bo Zhang, Shun Lu, Xiaolin Wei, and Junchi Yan. Darts-: robustly stepping out performance collapse without indicators. In International Conference on Learning Representations, 2021a.
Xiangxiang Chu, Bo Zhang, and Ruijun Xu. Fairnas: Rethinking evaluation faimesss of weight sharing neural architecture search. In International Conference Computer Vision, 2021b.
Xuanyi Dong and Yi Yang. Searching for robust neural architecture in four gpu hours. In Conference Computer Vision and Pattern Recognition, 2019.
Xuanyi Dong and Yang. Nas-hench-201 Extending the scope of reproducible neural architecture search. In International Conference Learning Representations, 2020.
Xuanyi Dong, Lu Liu, Katarzyna Musial, and Bogdan Gabrys. Nats-bench Benchmarking nas algorithms for architecture topology and size. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. The Journal of Machine Learning Research. 2019.
Stefan Falkner, Aaron Klein, and Frank Hutter. Bohb: Robust and efficient hyperparameter optimiza- tion scale. In International Conference on Machine Learning. 2018.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, 2010.
Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng. Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uniform sampling. In European Conference Computer Vision 2020.
10

---

Under review as a conference paper at ICLR 2023
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, 2015.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Conference on Computer Vision and Puttern Recognition, 2016.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint 17X1v::704.04861, 2017.
A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Master's thesis, Department of Computer Science, University of Toronto, 2009.
Jachoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. In Advances Neural Information Processing Systems, 2019a.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr. Snip: Single-shot network pruning based on connection sensitivity. International Conference on Learning Representations, 201 9b.
Zhihang Li, Teng Xi, Jiankang Deng, Gang Zhang, Shengzhao Wen, and Ran He. Gp-nas: Gaus- sian process based neural architecture search. In Conference on Computer Vision and Pattern Recognition, 2020.
Ming Lin, Pichao Wang, Zhenhong Sun, Hesen Chen, Xiuyu Sun, Qi Qian, Hao Li, and Rong Jin. Zen-nas: A zero-shot nas for nigh-performance image recognition. In International Conference Computer Vision, 2021.
Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In European Conference Computer Vision, 2018.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. In International Conference Learning Representations, 2019.
Shun Lu, Jixiang Li, Jianchao Tan, Sen Yang, and Ji Liu. Tnasp: A transformer-based nas predictor with self-evolution framework. In Advances in Neural Information Processing Systems, 2021.
Renqian Luo, Fei Tian, Tao Qin, Enhong Chen, and Tie-Yar Yan Liu. Neural architecture optimization. 2018.
Yash Mehta, Colin White, Arber Zela, Arjun Krishnakumar, Guri Zabergja, Shakiba Moradian, Mahmoud Safari, Kaicheng Yu, and Frank Hutter. Nas-bench-suite Nas evaluation (now) surprisingly easy. 2022.
Joe Mellor, Jack Turner, Amos Storkey, and Elliot J Crowley. Neural architecture search without training. In International Conference on Machine Learning, 2021.
Xuefei Ning, Changcheng Tang, Wenshuo Li, Zixuan Zhou, Shuang Liang, Huazhong Yang, and Yu Wang. Evaluating efficient performance estimators of neural architectures. In Advances in Neural Information Processing Systems, 2021.
Russakovsky Olga, Deng Jia, Su Hao, Krause Jonathan, Satheesh Sanjeev, Ma Sean, Huang Zhiheng, Karpathy Andrej. Aditya Khosla, Bernstein Michael, Berg Alexander C., and Fei-Fei Li. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 2015.
Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. Efficient neural architecture search via parameters sharing. In International Conference on Machine Learning, 2018
Ilija Radosavovic, Justin Johnson, Saining Xie, Wan- Yen Lo. and Piotr Dollár. On network design spaces for visual recognition. In International Conference on Computer Vision, 2019.
11

---

Under review as a conference paper at ICLR 2023
Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, and Jascha Sohl- Sohl-Dickstein. On the expressive power of deep neural networks. In International Conference on Machine Learning, 2017.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe Le. Regularized evolution for image classifier architecture search. In AAAI Con-ference Artificial Intelligence, 2019.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Conference on Computer Vision and Pattern Recognition, 2018.
Yao Shu, Shaofeng Cai, Zhongxiang Dai, Beng Chin Ooi, and Bryan Kian Hsiang Low. Nasi: Label-and data-agnostic neural architecture search initialization. In International Conference on Learning Representations, 2022.
K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015.
Zihao Sun, Yu Hu, Shun Lu, Longxing Yang. Jilin Mei, Yinhe Han, and Xiaowei Li. Agnas: Attention- guided miero and -acht-architecture search. In International Conference Machine Learning. 2022.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, 2019.
Hidenori Tanaka, Daniel Kunin, Daniel L Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. In Advances in Neural Information Processing Systems, 2020.
Jack Turner, Elliot Crowley Michael O' O'Boyle, Amos Storkey, and Gavin Gray. Blockswap: Fisher- guided block substitution for network compression budget. In International Conference Learning Representations, 2020.
Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. In International Conference on Learning Representations, 2020.
Ruochen Wang, Minhao Cheng, Xiangning Chen, Xiaocheng Tang, and Cho-Jui Hsieh. Rethinking ar- chitecture selection in differentiable nas. In International Conference on Learning Representations, 2021.
Wei Wen, Hanxiao Liu, Yiran Chen, Hai Li, Gabriel Bender, and Pieter-Jan Kindermans. Neural predictor for neural architecture search. In European Conference on Computer Vision, 2020.
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 1992.
Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. Snas: stochastic neural architecture search. In International Conference Learning Representations, 2019.
Jingjing Xu, Liang Zhao, Junyang Lin, Rundong Gao, Xu Sun, and Hongxia Yang. Knas: green neural architecture search. In International Conference on Machine Learning, 2021.
Yuhui Xu, Lingxi Xie, Xiaopeng Zhang. Xin Chen, Guo-Jun Qi. Qi Tian, and Hongkai Xiong. Pe-darts: Partial channel connections for memory- architecture search. In International Conference on Learning Representations, 2020.
Peng Ye, Baopu Li, Yikang Li, Tao Chen, Jiayuan Fan, and Wanli Ouyang. b-darts: Beta-decay regularization for differentiable architecture search. In Conference Computer Vision and Pattern Recognition, 2022.
Chris Ying, Aaron Klein, Esteban Real, Eric Christiansen, Kevin P. Murphy, and Frank Hutter. Nas- toss-bnch-1100 101: Towards reproducible neural architecture search. In International Conference on Machine Learning, 2019.
12

---

Under review as a conference paper at ICLR 2023
Jiahui Yu, Pengchong Jin, Hanxiao Liu, Gabriel Bender, Pieter-Jan Kindermans, Mingxing Tan, Thomas Huang, Xiaodan Song, Ruoming Pang, and Quoc Le. Bignas: Scaling up neural ar- chitecture search with big single-stage models. In European Conference on Computer Vision, 2020.
Arber Zela, Thomas Elsken, Tonmoy Saikia, Yassine Marrakchi, Thomas Brox, and Frank Hutter. Understanding and robustifying differentiable architecture search. In International Conference on Learning Representations, 2020.
Arber Zela, Julien N. Siems, Lucas Zimmer, Jovita Lukasik, Margret Keuper, and Frank Hutter. Surrogate nas benchmarks: Going beyond the limited search spaces of tabular nas benchmarks. In International Conference Learning Representations, 2022.
Zhihao Zhang and Zhihao Jia. Gradsign: Model performance inference with theoretical insights, In International Conference on Learning Representations, 2022.
Hongpeng Zhou, Minghao Yang, Jun Wang, and Wei Pan. Bayesnas: A bayesian approach for neural architecture search. In International Conference on Machine Learning. 2019.
Barret Zoph and Quoe Le. Neural architecture search with reinforcement learning. International Conference Learning Representations, 2017.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoe Le. Learning transferable architectures for scalable image recognition. In Conference on Computer Vision and Pattern Recognition, 2018 8.
13

---

Considering network performance ranking, ignore the effect of Under review as a conference paper at ICLR 2023
A THEORETICAL ANALYSIS
The dataset is denoted as { Mo) where the input E Rd and the output ys E R.
The loss function
N ( Σ e (ys, f(0;
(4)
where (d;x)) is the network architecture, E Rm and ((.,)) is the loss function.
Then the optimal network parameter 0* is
0"
argmin JN (0)- mER
(5)
Score index
OJN (09)
Ein
Σ
(6)
00k
where 72 T1 > 0 and 6 the initialization parameter. Here Tι and T2 correspond thri and thre in
Regarding the optimal parameter 6* and the initial parameter 80, we provide theoretical analysis from two perspectives.
Higher performance. network performance. We analyze the upper bound of JN (#*) and expect it small as possible;
The small loss at the optimal parameter Ju (6*) implies the high
Easier to optimize: The small distance between the initial parameter and the optimal parameter means the easy optimization process. We analyze the upper bound of |100 - 00112 and expect as small as possible.
A.1 THEORETICAL RESULTS
The theorems in All al. 2019 reveal that the neighborhood around random initialization has excellent properties that are almost convex and semi-smooth. Based on the work, we introduce a slightly stronger assumption here. Denote the vector & norm as ||-112-
Assumption (f) is h-strong convex and H-smooth in the neighborhood T (00,, R) [0]]|e 10112 s R of the initialization 80, where R > 0 is the neighborhood radius.
&
Lemma (O) is h-strong convex and H-smooth which equivalent to V= 1 H vO T (B²,)..
Theorem (The upper bound of the loss in the optimal point.) ) Assume that Jx(() is h-strong conver in the neighborhood (80, R) which in Assumption 7 hold. Then
OJN (09)
JN (0") < JN(
2 27)
(7)
2H
2H
80
(0)) (o)( 064
where 0 S min
< <T.
00
Remark The upper bound of ( is influenced by two items of Jx((") and B.xx (00)
Eie I
20
TI
the first item since JN(00) of different networks is similar. In practice, the network initialization
14

---

Under review as a conference paper at ICLR 2023
Glooot & Bengio 2010 He et al.) 2013 tends maintain the mean and variance of the output in each layer, so the final output logits of different networks have similar distribution, resulting in similar losses. In S----Bnnch-2-- the loss of x (8") of all candidate networks CIFARIO, CIFARIO0 and ImageNet16- 120 are 2.343 + 0.043, 4.651 + 0.042, and 4.842 + 0.047, respectively. Small deviation indicates that the JN (0))
) of different networks is indeed similar. Therefore, we only need to focus on
8JN 80
the second item, i.e, the larger 2k=11
Tij
the smaller the upper bound of Jx(00).
Theorem 2 (The upper bound of distance between the initial point and the optimal point.) ) Assume
Jy(0) is H-smooth in the neighborhood (0", R) in Assumption hold. Then
|100
11112
1 -
M M.m. m
(M - T2)
m
I1
OJN (09)
<T2) 72 1)
h
00k
(om) 80p
(09) 56
where M A max
2 T2
12 0.
[xx((")) 80s
Remark 2 To make the upper bound of |1000 - 00 small, the index Ek= I
<
need to be as large as possible.
considering the two goals of high performance and easy optimization, we expect the upper bound of JN (8") and ||e0 - 8" | to be as small as possible. Combining Theorem and the index
For given 72 > ri >
m Σn
OJN (09)
I
OJN (09)
OJN (89) <
T
00k
2
06k
< T2
=
1
11 S
06,
T2
k=
need be as large as possible.
According to Theorems and
we can obtain the proposed Zero-Shot estimator.
A.2 PROOF OF LEMMA
I
Let's just simplify JN (0) to J(N) for the sake of the statement. Denote J(0), V0 E T (00 R) is h-strong convex and H -moothh "Left" and h I = (x) = H I, v0 E I (0", RR) "Right" respectively.
Left Right.
Since (0) is h-strong convex, F(0) = (0) - +,1 | is convex. Then for the convex function P(O), we have
VV (8)) - FF (82)) (0, - 02) 2 0, vA, 0g E T (8", RR)
(8)
Notice that VF (0) VJ (0) - het and substitute it into Equation (8) Then
(0,) - JJ (02) ||0 - 021l2 2 (V,( - VJ (02)) (0, - 02) 2 h [10 - 0211
(9)
where the first equality comes from Cauchy-schwartz inequality.
Since J(0) is I-smooth, ((0) is Lipsehitz continuous and the Lipschitz constant is H, i.e..
[[v)(01)- ((22) 12 I H |01 02l12, V01, E I (0", R). By Cauchy-schwartz inequality, we
can obtain that
(VJ (0:) VJ (02)) T (0) 02) < VJ (0) JJ (02)ll2 10, -2112 S H 110, 02 (10)
Combing Equation 9 and 10 we have
h |0 02lla 2 < V (02 + tv) V. (02) S H1 1|0 02112
(11)
Inspired by Theorem 5.12 in Beck 2017 assume 0, = 82 + to where O. then
V (02 tv) VJ / (0, + 20)odz
(12)
15

---

Under review as a conference paper at ICLR 2023
Thus
hellell2 S 1/8 V2 V2J(02 + 20)d2 1
[[J (02 + (t) (J2(2)ll2 S Hillulla (13)
12
Divide both sides by and let + 0+. Then
A||u|12 - [(22(0) 1112 S M|/la/2
(14)
By the arbitrariness of 0, and 82, we have
h h=1= H
(15)
Right + Left.
For v01, 0g E (09, R), take J(f) out of the Taylor expansion at 0g and substitute 01 into [(}).
J(0i) ((2)) + (VJ(02)) (01 02) (0: 0₂) V2J (B8, (1 3)02) (01 02) (16)
where 0 3 < 1. By h. 1 V-) = H I, we have
J(O:) 2 ((0)) + (V)(e2)) T (01 (02) 2 110 -0212
(17)
and
((0)) v ((02) + (V)(())) (01 - 02) + H 10 0212.
(18)
By the arbitrariness of 0,, 02, the following conclusion can be obtained by exchanging the position of 0 and 82
J(02) 2 + (02 (1)-0:)+ 1/2m z 110 0211
(19)
and
((02) <
(02 - -1) + z H1 110 - -2112
(20)
Adding Equation 17
Equation
19 we can get
(v.(02) ((())) (02 0,) 2 h 100 - 0211
(21)
Similarly, by adding Equation 18 and 20 have
(V)(0)) - v(())) (02 01) < 10 02112
(22)
Thus (0) is h-strong convex and Hssmooth.
A.3 PROOF OF THEOREM I THE UPPER BOUND OF THE LOSS THE OPTIMAL PARAMETER
Take the loss function (O) out of the Taylor expansion at the initialization parameter 09, For vO E I (80, R), we can obtaín that
(0) JN (08) JN (09) T (0 80)
+
1
(0
80)
VJJN (al + (1- (b) (0 80)
2
S JN (08)
JN [T")] T (0 80)
+ H
1
80 001
OJN (80)
= JN JN (00) +
(0p. 00)
+
H
(0x - 00) & LN(())
(23)
a0
2
1=1
where 0 < <
Denote
argmin Ly (0). We assert that (ET(0).1)
JN (0)) $ LN
88)
(24)
16

---

Under review as a conference paper at ICLR 2023
Otherwise JN (0*) > LN (8). we have
JN (0)) 2 JN (0") LN (0')
(25)
which is in contradiction with Equation 23
Notice that LN (0) is quadratic function 09, To make the upper bound of J (0*) to be as small possible, we expect L (0) be the global minimum in I (4", RR. The minimum point of L (0) is
OJN (00)
0x - 0%
k E {1, 11,.,.,.. m
(26)
002 H
Next prove that the optimal point of n (0) is in the feasible domain I' (0", R). Since 8* is the optimal point of (f) and JN (0) is H-smooth, we have
V JN (0") VJN (09) VJN (00) 0| 1 S HR.
(27)
Then
|0- 09/
1 H JJN (09)1z s R.
(28)
further
we have 0" E (0", RR.
Thus the global minimum of L (0) is
m
OJN (89)
2
IN ②0
))= = JN (09)
JN
Σ
(29)
2H
00
k=1
For the upper bound of JN (0*), we obtain that
OJN (80)
JN (0") S LN (0') = JN (00)
2H
00
OJN (00)
OJN (09)
S JN (09)
T
2
< TT
<nl
2H
08%
2H
OHk:
k=
k= 1
1
1
UJ (B0)
(09)
m
x2)
2
(30)
2H
2H
00k:
k=1
where S 2 min {
(6))
(ox(
66
29
<
Ti
<T
A.4 PROOF OF THEOREM PARAMETER AND THE OPTIMAL PARAMETER
2 THE UPPER BOUND OF DISTANCE BETWEEN THE INITIAL
Denote the vector & norm as ||-I11- Take the gradient of loss function (#) out of the Taylor expansion at the optimal parameter 0** and substitute the initialization parameter 0 into the gradient of loss function.
JJN (09) = VJN (0") V2JN (700 + (1 - ---)))) )- (00 - 0*) JV (780 (1 7)0)) (0" -8")
(31)
where 0 K and the last equality comes from VJy (0*)
0. By Assumption
I we have
h. I > JJJN (700 + -7)0r) x H Thus h- 1 0* (11 <
|JJ (0") H
0 0"
17

---

Under review as a conference paper at ICLR 2023
Further obtain that
00
0-d
1
1
VJN (6") (1)) - h
1
||N (6")
1 m
OJN (00)
h
00k
1
m
OJN (09)
m
aJN (09)
72
< 72
+ h
M
72
h
00k
DAk
OJN (00)
1
m
OJN (89)
T2
T2
+
m
< T2
h
00k
h
00
k=1
k=
1
OJN (09)
M - m
(M - T2)
<
T2
(32)
h
00k
[d)x(0))
(0")
where M max
80
80
T2
12
T1
0.
18

---

Under review as a conference paper at ICLR 2023
B SWEET GRADIENT PHENOMENON
Sweet Gradient widely exists across different search spaces, datasets, the number of architectures, batch size, and initializations. Here list the heatmaps of Sweet Gradient under various configura- tions.
B.1 DIFFERENT SEARCH SPACES
Sm-10
1c:0
16:00
56-10
03
0.4
05
1o:07
Se-0
50-07
le-d
04
6.0001
0.0
02
2.0005
0.0355
0401
0.001
0.055
0.
01-
-02
0.
05
1.
10.
000000
0.0000
201
53
2007
Se40
3000
2002
000
003
32
thr2
(a) NAS-Bench-101
(b) NAS-Bench-201
1e-09
5e-10
56:00
16:00
0.
5e-0
36:09
Sm-07
10:05
5c-5-
0.0
5.0005-
0.0255
0001
0.001
0003
0.005
0.
2.01
1.
05
10.
-02
sel0
tedi
1e47
te47
2000
14:3
selt
5200
25
20-07
Teal
10:03
1000
000
1000
500
65
73
thr2
(c) NAS-Bench-301
(d) NDS-NA
Figure 4: Sweet Gradient across different search spaces.
19

---

Under review as a conference paper at ICLR 2023
B.2 DIFFERENT DATASETS
5e-10
56.30
03
10:05
16:00
5e:9
36:09
044
06
Sm-0
1ed0
1n:06
bc05
3u:06
-02
50:55
0.0002
0001
0.001
0002
0.005
-.2
05-
10.
Tedt
tedi
1e47
14-05
9.0005
0.03
0.005
S05
63
65
20.0-
5.0033
0.05-
5.1
205
65
thr2
(a) CIFAR-10
(b) CIFAR-100
0.8
1e:56
50-20-
10:2
5e-28
1s:50
50-16-
1e:5
6e-35
0.01
0.05
0.21
-04
10
1e00
1
1206.
5000
00009
oms.
605-
61
0.5
1.-
(c) ImageNet16-120
Figure 5: Sweet Gradient across different dataset in WAS-Bench-201..
20

---

Under review as a conference paper at ICLR 2023
B.3 DIFFERENT NUMBER OF ARCHITECTURES
5e-10
10:05
0A
56:30
16:09
5e:9
36:09
Sm-07
5e:77
5c05-
1e-0
01
1n:66
50:55
0.0002
0.0255-
0001
0.001
0002
501
-12
0.005
05-
10.
Tede
Je-8-
tedi
1e07
5e07-
2000-
scot
se4s
5.0001
5.000s
201
s0t
63
leal.
tear
5.0033
5.000-
0.00
nos-
501
thr2'
(a) Number = 25
(b) Number 50
1--56
52-20
10:25-
5e-28
1-:56
02
5e-16-
1e:5
6e-35
0.011
0.055
0.21
-644
So 1010-
1e00
1x07
0000
00009
0001
605-
61
0.5
(c) Number 100
Figure 6: Sweet Gradient across different number of architectures in
21

---

Under review as a conference paper at ICLR 2023
B.4 DIFFERENT BATCH SIZES
05
5e-12
56.30
10:05
16:00
5e:9
36:09
Sm-07
1e-0
10:06
02
5c
50:65
000
0.0002
0.0355
0001
0.001
0002
0.005
05-
Tede
Je-8-
tedi
1e07
5e47-
2000.
Scot
1405
seas
5.0000
9.000s
201
50t
63
65
10.
10-0-
leal.
tear
10.0
5.0033
5.000-
0.05.
501
65
-444
thr2
(a) Batch Size 32
(b) Number 64
1e-56
52-22
10:25
5e-28
1s:56
02
5e-16-
1e:5
6e-35
0.011
0.055
0.21
-644
So 10
1e00
1x07
1206
0000
00009-
0001
685-
61
0.5
(c) Number 128
Figure 7: Sweet Gradient across different batch size in NAS-Bench-101.
22

---

Under review as a conference paper at ICLR 2023
B.5 DIFFERENT INITIALIZATION
06
05
5e-12
56.30
10:05
16:00
5e:9
36:09
Sm-07
1e-0
02
10:06
5c05-
50:45
000
0.0002
0.0355
0001
0.001
0002
0.005
-2.2
05-
Teds
Je-8-
tedi
1e07
5e47-
2000.
14-5
se4t
5.0001
9.000.
0.01
0.00.
201
505
63
65
10.
10-0-
10.0.
3.0003
0.05-
501
-44
thr
(a) Seed = 0
(b) Seed
0.6
1e-56
52-20
10:25
5e-28
1e:56
02
5e-16-
1e:5
6e-35
0.011
0.055
0.21
-62
1e00
1x07
1206.
5406
0000
60009-
0001 -
605-
61
0.5
Seed
Figure 8: Sweet Gradient across different initialization in 115-Bench-101.
23

---

Under review as a conference paper at ICLR 2023
C MORE INTERVALS IN NAS-BENCH-101 AND NAS-BENCH-201
C.1 INTERVALS IN NAS-BENCH-101
Sweet (5e-10, 10.8)
Sweett [1e-8, Sa-8)
Sweet [Be-8, 1e-0)
Sweet: [Te-6, be-d)
35
14
10
104813
50000
80000)
Sweet: [Be-8, 1e-7)
Sweet [10-7, 5o-7)
Sweet [5e-7, 10-
Sweet: [16-6, Sa-6)
0.000
c.01
04020
ccm
0400
5:00
00000
0000
5.54
00000
00000
5250
04000
5000
04008
Sweet [5o-6, 16-5)
Sweet [1e-5, 5e-5)
Sweet: (8e-6, 0.0001)
Non-Swest: (0.0001, 0.0005)
0.03
0.0
0.03
cam
033
0383
040
Non Sweet: j0.0005,
Non-Sweet: j0.001, 0.005)
mann-Saatt:000005
Non-Sweot [0.01. 0.05)
0.25
0.001
5.01
propotonn
0275
6100
8.39
0383
Non-Sweet (0.05, 0.1)
Non-Swen Son-Sweet: [0.1, 0.5)
[0.5, 1.0)
Non-Swest (100,5.0)
0435
03000
0.64
00000
0455
Onll depth
oct dapth
Figure 9: More Intervals in NAS-Bench-IDe
24

---

Under review as a conference paper at ICLR 2023
C.2
INTERVALS IN WAS-BENCH-201
Non-Sweel: [5e-10, 1e-9)
Non-Sweel: [1e-9, 5e-9)
Non-Sweelt 1e-6)
6.01
0216
2000
50011
0.13
5533
0.00
0:001
50100
5004
0.00
0301
40001
270
9400
10
22
Swen-Seet:Boom [5e
Non-Swiir Swaet: [1o-7,
Non Vicn-Swaat:
Non-Sweet: 11a-5
0.03
szt
022
037
5m
[5e-6,
Non-Sweet: [1e-5. (1e-5.5e-5)
5e-5)
0.0001)
10.0001,
6000
cmm
proorton
0333
112
630
0333
542
6675
1000
2
&
21
11
Sweel: (0,0005. 0.001)
Sweet: (0 (0.001, 0.005)
Non Seen-Seat: (0.005, 0.01)
[0.01, 0.05)
1.70
0.3
papp 800
6:
Non- Swoet [0.05, 0.15)
Non-Sweet: (0.1.0.5)
Non-Sweet: [0.5. 1.0g
Nor-Sweet [1.0, 5.0)
0.80
0.07
proport.n
0.01
MC2
0/000
depph
degth
Figure 10: More Intervals in NAS-Bench-201.
25

---

Under review as a conference paper at ICLR 2023
D OTHER INVESTIGATIONS
Investigation Is the Sweet Gradient related to gradient distribution? A natural conjecture is that Sweet Gradient more likely exist in intervals with concentrated gradient values. Figure IT and [bb show the distribution of the absolute gradients values in NAS-Bench-10) and NAS-Bench- 201, which disproves the conjecture. One reason that there are Non-Sweet Gradient intervals where Another reason is that there
the gradients are concentrated, e.g., [0.001, 0.005) in are also Sweet Gradient intervals where the gradients are not concentrated, e.g., [5e-7. 7, le- 6) in NAS-Bench-101.
Investigation 2: Is the Sweet Gradient related to activation distribution? The activation reflects the expressibility (Chen et al. (2211a of the network, so does the activation value distribution affect the sweet gradient? Figure (C) and [[dd) show the distribution of absolute activation values in NAS-Bench- 101 and NAS-Bench-208
The activation values are mainly distributed in [0.01, 5), and
the corresponding interval is almost staggered with Sweet Gradient with a large order of magnitude difference, which reveals that they not related.
1.2
1.0
counts
03
0.
10
3:000 100
E2es en
01-45
conn on
noves
5031
5300-11
(a) Gradient, AAS-Bench-101
(b) Gradient,
248
32
1.7
2.0
counts
0000
605-33
20-51
thr1-thi2
thr1-thr2
(c) Activation, MAS-Bench-100
(d) Activation, NAS-Bench-201
Figure 11: The gradient and activation distribution under different intervals NAS-Bench-201 (CIFAR-10).
101 and
26

---

Under review as a conference paper at ICLR 2023
E EXPERIMENTAL DETAILS
E.1 DETAILS OF CONSISTENCY EXPERIMENTS
Experiments are mainly based on the publicly available codes from Mellor al. (202] : Abdelfattah
et al. 2021h; Zhang & Jia (202)); Chen et al. (2021a) Lin et al. (2021); Mehta et al. (2022) (MIT or Apache-2.00 2.0 licence). Following Zhang Jia (002)), the seed 42 which randomly chosen. The batch size is 64 in NAS-Bench-IO0 101, NAS- Benchh2,, and NAS-Bench-301, and 128 in NDS space, which is also the same as Zhang Jia (022). For Sweetimator, we set the architecture number to 100, and the split 2.
E.2 DETAILS OF
EXPERIMENTS
Sweetimator has a batch size of 128, architecture numbers of 100, and a split of 2 in architecture selection and proposed in Mellor et al. (2021); Zhang & Jia (0022)
NAS. For fair comparison, we strictly followed the algorithm
E.3 DETAILS OF DARTS EXPERIMENTS
In the search phase, Sweetimator is obtained by a batch size of 128, architecture numbers of 100, and split of 2 CIFAR-10 10 and ImageNet. Then integrate Sweetimator into REA algorithm by replacing training networks with computing the score of Sweetimator. Following Doog Yang (2020), we set the number of cycles 200, the population size 10 and the sample size to 3. Although the cycle is only 200, the scores converge rapidly and the search results can be competitive with other Zero-Shot methods, which also bring fast search speed. Therefore, we did not do more hyper-parameter tuning. Please note that the search time includes computing the best interval and searching for the optimal architecture.
In the retraining phase, we follow DARTS settings to train architectures. On CIFAR-10, the network consists of 20 layers with 36 initial channels. We utilized the SGD optimizer to train the network for 600 epochs with a batch size of 96. The learning rate decays from 0.025 to 0 by the cosine scheduler. Other settings like cutout, auxiliary, and path dropout are the same as DARTS. On ImageNet, the network consists of 14 cells with 48 channels, which is restricted to be less than 600M FLOPs. The SGD optimizer is used to train the network with 250 epochs, a learning rate of 0.5, weight decay of 3e-5, and batch size of 1024. We train the network on eight NVIDIA V100 for around three days.
27

---

Under review as a conference paper at ICLR 2023
F COMPARISON WITH NON-ZERO-SHOT ESTIMATORS
Settings. We compare Sweetimator with Non-Zero-Shov estimators VAS-Benchh2001 The baselines include SPOS (Guo et al. (020), Neural Predictor (Wen al. 2020), NAO (Luo al. 2018). TNASP (Lu et al. 2021 For SPOS, we trained supernet with 250 epochs and a batch size of 256, then utilized the validation accuracy of sub-networks as the estimator. For Neural Predictor, NAO and TNASP, we directly used the results in Lu et al.) (2021). For Sweetimator, we set the batch size to 128, the architecture number to 100, and the split to The compared metric is Kendall's Tau between the score of estimators and the test accuracy on CIFAR-10 of the benchmark.
Results. Table 2 shows the comparison results. Sweetimator can achieve superior rank consistency to which further demonstrate the effectiveness of the
Non-Zero-Shan estimators on proposed method. Moreover, Sweetimator takes only a few minutes to obtain the best interval, which more efficient than the process of training networks in Non-Zero-Shot estimators.
Table 6: Rank Consistency of Non-Zero-Shot estimators and Sweetimator in NAS-Bench-201.
Estimators
Kendall's Tau
Type
SPOS
0.621
Non-Zero-Shot
Neural Predictor
0.646
Non-Zero-Shot
NAO
0.526
Non-Zero-Shot
TNASP
0.724
Non-Zero-Shot
Sweetimator
0.736
Zero-Shot
28

---

Under review as a conference paper at ICLR 2023
G VISUALIZATION OF SEARCH RESULTS
G.1 NAS-BENCH-201
DFAR-10
CIAR-100
Imageleet 6-120
Mooe MOC AOCLENTY
Mi00 BOCLICC
L00000
30000
20008
T00000
49806
20000
Figure 12: Visualization of model test accuracy versus Sweetimator metric score on CIFAR-10, 1000 ImageNet16-120.
G.2
DARTS CELLS
skip
cone 5xd
12p cOmY 3cl
paol
coty
up com
sip cornect
[k(
c.
sip_ronnse
asg_pool_hs Axc
tep.
[k-1]
sep_cctn__
C. [e
poo_
uqu
CIFAR-10, Normal Cell
(b) CIFAR-10, Reduction Cell
spp pop_oov 3x5
skip pontret
scp cozy
[Lik]
max_pool_2t3
max pool 3x3
spp
Sed
diL.ooy__50
(1-2)
c.lkl
dil Sx5
(c) ImageNet, Normal Cell
(d) ImageNet, Reduction Cell
Figure 13: Visualization of the best searched cells CIFAR-10 and ImageNet.
29

---

Under review as a conference paper at ICLR 2023
H MORE DIFFERENT TASKS
We conduct experiments
MAS-Bench-NLP and NAS-Bench-ASR benchmarks to further validate our method. The compared baselines include SynFlow, GradSign, and Parameters. The results the following table show that Sweetimator has superior performance consistency than other methods. However, the Spearman's 's rank NAS- Bench-NLLP drops significantly compared to that CV tasks. We conjecture that the text generation tasks and RNN architectures more complicated than the classification tasks and CNN architectures, and furthermore, the hyper-parameters of the architecture candidates in the NAS-Bench-NLPP search space are not well-tuned. Due the limited time available for rebuttal, we will improve the consistency of Sweetimator for more tasks in the future.
Table 7: Rank Consistency of Non-Zero-Shot estimators and Sweetimator and other Zero-Shot estimators by Spearman's rank in WAS-Bench-NLP and AAS-Bench-ASR.
Benchmark
SynFlow
GradSign
Parameters
Sweetimator
NAS-Bench B.SBBench-NLP
-0.062
0.134
-0.046
0.159
HAS-Bench--SSH
0.244
0.288
0.305
0.311
30


Reviews:
[{"review_id": "yukz030i7gB", "reviewer": "Reviewer_FxrW", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "While existing zero-shot NAS algorithms can usually achieve competitive performance in practice, their zero-shot estimators are unfortunately observed to perform inconsistently in different tasks (e.g., search spaces and datasets), which is even not comparable to the most straightforward one, i.e., the number of parameters, abbreviated to #params. This is known to be a critical problem in the zero-shot/training-free NAS field. So, this paper aims to develop a new zero-shot estimator that can perform consistently well on different benchmark datasets and search spaces. Before realizing this, this paper firstly observes that the number of Sweet Gradient (i.e., gradient within a special interval) consistently has better rank correlations than other zero-shot estimators in practice, which indicates that the number of Sweet Gradient may be a better choice for zero-shot NAS. Then, inspired by the close connection between the increasing proportion of Sweet Gradient and the increasing neural network depth, this paper proposes to leverage network depth to compute the Sweetimator and hence can find Sweet Gradient intervals without training. Finally, this paper uses extensive experiments to show that its zero-shot NAS based on Sweetimator can outperform existing zero-shot estimators in both ranking correlations and search results on various NAS benchmarks and datasets.", "summary_of_the_review": "In general, my major concerns lie in the soundness and clarity of this paper as the empirical results are already quite good in this paper. I hope the authors can address my concern in the rebuttal period.", "strength_and_weaknesses": "## Strength\n1. This paper studies an essential problem in the zero-shot/training-free NAS field, i.e., improving the practical consistency of zero-shot estimators, and develops an interesting zero-shot estimator Sweetimator to tackle it, which empirically outperforms other zero-shot estimators in extensive experiments.\n2. Based on the interesting Sweetimator, this paper proposes a zero-shot NAS algorithm that performs better than other zero-shot baselines on various NAS benchmarks and datasets.\n3. This paper provides many ablation studies to help better understand their proposed Sweet Gradient interval.\n\n## Weakness\n1. The theoretical motivation or justification for the study of eq.1 is missing in this paper. Such a motivation or justification is necessary because it helps to understand why studying this eq.1 (rather than other forms of gradients) may help develop a consistent zero-shot estimator for NAS and how it helps improve the rank consistency over different NAS benchmarks and datasets. The whole paper only relies on empirical observations to support its conclusions, which is not so convincing to me.\n2. This paper doesn't provide a formal, clear, and effective definition of Sweet Gradient intervals, which leads to some problems. Specifically, in this paper, it's informally defined by comparing with the rank consistency of #params in Sec. 3.2. This means that to obtain these Sweet Gradient intervals, people need to measure the rank correlations of #params on various NAS benchmarks, which seems to contradict with eq.3 that only leverages the target NAS benchmark to obtain Sweet Gradient intervals (Problem 1). In addition, based on such an informal definition, Sweet Gradient intervals adapt when the rank correlation of #params changes. So, the eq.1 based on Sweet Gradient intervals may not be able to outperform other zero-shot estimators when the rank correlation of #params performs very poorly in certain tasks (since Sweet Gradient intervals only need to improve over #params even slightly) and thus can not guarantee to achieve better consistency than other zero-shot estimators (Problem 2).\n3. The effectiveness of using eq.3 to determine the best Sweet Gradient interval remains questionable. This is because (a) the discussion below eq.2 shows that the Sweet Gradient interval shifts with the increasing network depth and therefore the Sweet Gradient intervals obtained using the increasing network depth may differ from the desired one for the search space with a different fixed network depth (this is known to be common in NAS by searching with a smaller network depth and applying with a larger network depth), and (b) the parameter proportion in the Non-Sweet Gradient intervals may also increase according to Appendix B.2 and therefore eq.3 may not really output the best Sweet Gradient interval (it can also output Non-Sweet Gradient interval). This begs the question: Is the Sweet Gradient interval really the one that leads to the good search results in the experiments of this paper?\n4. The details of Fig. 3 need to be clearer, e.g., the rank correlations of these Sweet Gradient intervals and which layer is used to plot (since eq.2 is defined on a specific layer l). These are important to understand the difference among Sweet Gradient intervals and the possible impacts of different layers on the observations in the paper.\n5. As mentioned in the limitation section of this paper, this paper only focuses on classification tasks with different search spaces to compare the consistency of different zero-shot estimators. I think it may also be important to examine the improved consistency of the proposed method in different tasks to further support its superiority.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "dTzoziT8aMu", "reviewer": "Reviewer_3NxU", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Verifying Sweet Gradient: The paper first discovered through experiments that the Sweet Gradient of parameters, i.e., the absolute gradient values within a certain interval, brings higher consistency in network performance than the overall number of parameters. Then this paper also uncovered a positive correlation between the network depth and the proportion of parameters with sweet gradients in each layer.\n\nNAS: Based on the observation, this paper also proposes a Zero shot NAS method based on Sweet Gradient.", "summary_of_the_review": "The motivation for the research is clear. But the paper lacks the necessary theoretical analysis. I need the author to provide more details in the rebuttal to improve my score.\n", "strength_and_weaknesses": "Strength:\n1. The writing of the paper is clear and understandable.\n2. The comparison between Sweet Gradient and other indicators is completely sufficient.\n\nWeaknesses:\n1. There is a lack of theoretical analysis on why Sweet Gradient is better than other measurements.\n2. Lack of innovation in improving the Zero-shot NAS search method.\n\nQuestion:\n1. Figure 1 compares the results on three datasets. Has the author analyzed why there are gaps between the three datasets? If the differences between the three datasets are all very large. How does the author guarantee the versatility of Sweet Gradient?\n\n2. Does the author theoretically analyze why Sweet Gradient is better than other indicators?\n\n3. Is the improvement to the existing Zero shot NAS just updated with new metrics? I have a question about how the paper implements zero-shot NAS.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Ufc2iLiTQ_", "reviewer": "Reviewer_g7DU", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper finds that the absolute gradient values within a certain interval, called Sweet Gradient of parameters can be good zero-cost proxy evaluating candidate models in zero shot NAS. The authors find a positive correlation between the depth and the proportion of sweet gradient parameters, and further propose a way to automatically determine the Sweet Gradient interval. The effectiveness of the proposed method is validated on four benchmarks with eight search spaces.", "summary_of_the_review": "The paper is clearly written. The main concern is about insightful discussion and theoretical analysis.", "strength_and_weaknesses": "Strength:\n\n1. Utilizing the absolute gradient values within a certain interval as zero cost proxy evaluating candidate models in NAS is novel and interesting. Building a consistent zero-cost proxy is a very good contribution. \n2. The method is simple and easy to implement. \n3. The paper is clearly written. Experiments and ablation studies are well performed.\n\nWeakness:\n\n1. The proposed method, in its current form, is mostly heuristic. It is still not clear how the authors find that Sweet Gradient of parameters can be good proxies and why it works consistently well. More insightful discussion or theoretical analysis may be required to fully justify the proposed method.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "97a8BuF5EF", "reviewer": "Reviewer_tGTD", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a zero-cost evaluation metric to improve zero-shot neural architecture search efficiency in this work. This work uses gradient distribution and network information at initialization for scoring candidate models and achieves a strong rank consistency with the performance of candidate models. Extensive experiment results are provided to validate the efficiency of the zero-cost metric.", "summary_of_the_review": "This paper is well written and technically sound, but the theoretical novelty is limited. ", "strength_and_weaknesses": "Strengths\n1. The paper is very well-written and easy to follow.\n2. The proposed idea to search “sweet gradients” in candidate networks seems novel and interesting.\n3. The paper has provided extensive evaluations over different metrics and datasets. The results are presented in clear manner and are fairly convincing.\t\n\nWeakness\n1. The theoretical novelty is limited, although I do acknowledge the contribution of the proposed metric to zero-shot NAS.\n2. The proposed method in this paper looks like a universal evaluation metric, not limited to the computer vision field. So the diversity of experiments is not enough. The authors only use image datasets to validate the effectiveness of the proposed metric. It is better to validate the proposed metric on NAS-Bench-NLP and NAS-Beach-ASR should\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject