$\Lambda$-DARTS: Mitigating Performance Collapse by Harmonizing Operation Selection among Cells
Authors: Sajad Movahedi
Melika Adabinejad
Ayyoob Imani
Arezou Keshavarz
Mostafa Dehghani
Azadeh Shakery
Babak N Araabi
Venue: ICLR 2023 poster (2023)

Abstract: Differentiable neural architecture search (DARTS) is a popular method for neural architecture search (NAS), which performs cell-search and utilizes continuous relaxation to improve the search efficiency via gradient-based optimization. The main shortcoming of DARTS is performance collapse, where the discovered architecture suffers from a pattern of declining quality during search. Performance collapse has become an important topic of research, with many methods trying to solve the issue through either regularization or fundamental changes to DARTS.
However, the weight-sharing framework used for cell-search in DARTS and the convergence of architecture parameters has not been analyzed yet. In this paper, we provide a thorough and novel theoretical and empirical analysis on DARTS and its point of convergence.
We show that DARTS suffers from a specific structural flaw due to its weight-sharing framework that limits the convergence of DARTS to saturation points of the softmax function. This point of convergence gives an unfair advantage to layers closer to the output in choosing the optimal architecture, causing performance collapse. We then propose two new regularization terms that aim to prevent performance collapse by harmonizing operation selection via aligning gradients of layers. 
Experimental results on six different search spaces and three different datasets show that our method ($\Lambda$-DARTS) does indeed prevent performance collapse, providing justification for our theoretical analysis and the proposed remedy.

Published as a conference paper at ICLR 2023
A-DARTS: MITIGATING PERFORMANCE COLLAPSE BY HARMONIZING OPERATION SELECTION AMONG CELLS
Sajad Movahedi', Melika Adabinejad', Ayyoob Imani", Arezou Keshavarz', Mostafa Dehghani Azadeh Shakery!: and Babak N. Araabi
University of Tehran, LLU Munich, Google Brain,
Institute for Research in Fundamental Sciences (IPM)
{s movahedi, mel 1 iak aikia.aaabi adabi ada ac. ir, ayyoob. iimaii @cis. Rec.s.lm..........m...... lml.m.
are arezoukkeshavara net, @ehghani@gooogee com, {shakery, araabi ac. ir
ABSTRACT
Differentiable neural architecture search (DARTS) is a popular method for neural archi- tecture search (NAS), which performs cell-search and utilizes continuous relaxation to improve the search efficiency via gradient-based -based optimization. The main shortcoming DARTS performance collapse, where the discovered architecture suffers from a pat- tem of declining quality during search. Performance collapse has become an important topic of research, with many methods trying to solve the issue through either regulariza- tion or fundamental changes to DARTS. However, the weight-sharing framework used for cell-search in DARTS and the convergence architecture parameters has not been analyzed yet. In this paper, provide thorough and novel theoretical and empirical analysis on DARTS and its point of convergence. We show that DARTS suffers from a specific structural flaw due its weight- framework that limits the convergence of DARTS to saturation points of the softmax function. This point of convergence gives unfair advantage to layers closer the output choosing the optimal architecture, causing performance collapse. We then propose two new regularization terms that aim to prevent performance collapse by harmonizing operation selection via aligning gra- dients of layers. Experimental results six different search spaces and three different datasets show that our method (A-DARTS) does indeed prevent performance collapse, providing justification for theoretical analysis and the proposed remedy. We have published our code at Iittt::///// //gas://itie github. com/ /t//ddrfaaa faustus Latps/Laaaaaaaaaaa
INTRODUCTION
With the growth the popularity of deep learning models, neural architecture design has become one of the most important challenges of machine learning. Neural architecture search (NAS), a now prominent branch of AutoML, aims to perform neural architecture design in an automatic way (Elsken ct al., 2019) Ren
al. 2021). ). Initially, the problem of NAS addressed through reinfoncement learning evolutionary
algorithms by the seminal works of (Zoph & Le 2017 Baker et al. 2017) Staaley & Miikkulainen 2002
Real et al. 2017). But despite the recent advancements in efficiency (Baker al. [ et al. [018 Cai ct al. 22088), these methods remain impractical and not widely accessible. One-shot methods aim to address this impracticality by performing the architecture search in a one-shot and end-to-end manner (Liu et al.
2019; Guo et al. [2020] Pham et al. [0188] Brock et al., 2018 Bender et al. [0118). Another technique for increasing the efficiency of search cell-search (Zoph et al. 2018). which performs NAS for a set of cells archtttrc Differentiable neural architec- ture search (DARTS) (Liu et al. 2019) a one-shot method that performs cell-search using gradient descent and continuous relaxation. It performs the cell-search using weight-sharing framework. As result of these innovations, DARTS is one the most efficient methods of architecture search (Elsken et al. 2019).
stacked on top of each other according to a pre-defined
One of the most severe issues of DARTS is the performance collapse problem (Zela et al. 2020) which states that the quality of the architecture selected by DARTS gradually degenerates into an architecture solely consisting of skip-connede copp-onnnction. This issue is usually attributed to gradient vanishing (Zhou et al., 2020) or large Hessians of the loss function (Zela et al. 2020). and mostly addressed by heavily modifying
DARTS (Chu et al. 2021 Wang al. 2021
Gu et 2021 Ye et al. 2022).

---

Published as a conference paper at ICLR 2023
DARTS
A-DARTS
Output Distribution
Cell AN
0C
60
KPooor
Kppoo
dC DE
XPico
corv
s4p
pool
3D
BC
0C
60p
Sparo
Phwn
OF 10 currrry (%6
C2
6.7
0.7
Alggment
0.0
B
CIFAR- 1n Test Aucuraity
Aiggmment
3.
#1
0.3
0.3
Inputs
6.1
0.1
8
20
10
40
Epoch
Eppch
Figure 1: An illustration of the differentiable formulation for NAS by DARTS, and the effects of layer alignment over the trajectory of the performance of the discovered architectures by DARTS and A-DARTS. The experiments are performed on the NAS- JAS-Bench-201 search space, averaged over 4 runs with 95% confidence interval.
In this paper, we argue that these theories fail to grasp the main reason behind the performance collapse: the conditions imposed on the convergence of DARTS by the weight-sharing framework. Specifically, we first define a measure to show the corrllation between the gradient of each layer corresponding to the architecture parameters- dubbed layer alignment (A). Then, following careful analysis of the convergence conditions of DARTS and the effects of weight-sharing, we find that:
1.
Due to the value of A, DARTS only achieve convergence by reaching the saturation point of the softmax function, where the normalized architecture parameters become almost one-hot vectors. This convergence point does depend directly the loss function, giving an unfair advantage to the layers that do not suffer from gradient vanishing.
Low value for means that the optimal architecture corresponding to each layer varies wildly, with layers that are closer to the output and therefore not suffering from gradient vanishing mainly preferring non non-paramen parametric operations.
3. The aforementioned issues are direct contributors to the performance collapse problem. Further- more, increasing the depth of the search model or the number of iterations used for the search increases the severity of performance collapse.
In Figure- I we can see an illustration of DARTS and the concept of layer alignment (which we will define Se.tion and observe a clear correlation between the layer alignment and the performance discovered architecture on the NAS-Bench-201 search space (Dong & Yang. 2020). In this paper, we will use a combination of analytical and empirical evidence to support the claim that this relationship is in fact causal relationship, resulting from the weight-sharing framework used in DARTS.
the
Following these conclusions, we introduce a regularization term that aims to alleviate the problem of perfor- mance collapse by increasing the layer alignment. Then, through comprehensive empirical examination, we show that our method dubbed A-DARTS significantly improves the performance of DARTS on various search spaces and datasets without any modification the algorithm of DARTS or the structure of the cells. Specifically, our method achieves an average accuracy of 96.57% and 83.85% on the CIFAR-10 and CIFAR-100 datasets on the DARTS search space, improving upon the current state-of-the-art by 0.06% and 0.39%, respectively. To the best of our knowledge, our work is the first to investigate DARTS from the point-of-view of its wight-sharing framework and convergence conditions.
2 RELATED WORK
DARTS (Liu et al. 2019) proposed a continuous and differentiable search space through weighting a fixed set of operations to make NAS more scalable. It trains super-graph with gradient descent and chooses the sub-graph consisted of weightiest operation edges. Its simplicity made very popular and many variations emerged to address its theoretical and empirical setbacks:
2

---

Published as a conference paper at ICLR 2023
Overlooked effects of design decisions in DARTS. (Wang et al. 2021 and (Gu et al. 2021) claim that DARTS chooses the final architecture with regard to operation weights, whereas these magnitudes do not necessarily correlate with the performance. (Wang et al. 2021) tries to measure operation impact directly and DOTS (Gu et al. 2021) tries to decouple topology from operation during search. IDARTS (Xue et al., (20T)) posits that the innate relationship between the architecture's s parameters is ignored by the gradient descent method used in DARTS and formulates their optimization as a bi-linear problem.
Generalization ability. The differentiable NAS methods are often executed on a smaller dataset of certain task to reduce the required computational resources. AdaptNAS (Li et al. (020). MixSearch (Liu ct. 202T). P-DARTS (Chen et al. 2021b), and DrNAS (Chen et al. 202 show that the discovered architecture does not always perform as well as on the proxy task on another more challenging dataset.
They solve this issue by providing the network with domain information, making the search network progressively deeper, and reformulating the search algorithm.
Performance collapse. Zela et al. (2020); Chu al. (2021) point out the performance collapse problem of DARTS. They show that DARTS consistently discovers networks with mainly skip-connection operations, which causes severe performance declination. Zela et al. (2020) monitors the eigenvalues of the Hessian of the architecture parameters and performs early stopping. Based on their work, SmoothDARTS (Chen & Hsich 2020) performs Hessian regularization to smooth the architecture Hessian. PC-DARTS (Xu et al. 2020) reduces the number of output channels of each operation, for the sake of lower memory consumption and a larger batch size during search. iDART'S (Zhang et al. (021) focuses on improving the optimization of the architecture parameters by estimating the descent direction more accurately. DARTS- (Chu al. 2 argues that skip-connection operations have a potentially sharp loss curvature that leads to their ad advantage during search, and tries to solve by adding auxiliary skip-connections cell structure. 3-DARTS (Ye iet al., (022)) replaces the -rregularizztion on the architecture parameters with new regularization term to alleviate the unfair advantage of skip-conmn coineconnetions. In this paper, we provide a novel analysis of DARTS from the point-of-view of its convergence, diagnosing an overlooked issue of its weight-sharins system that sends convergence down an irrelevant path and may the root cause of performance collapse. We propose a solution for the matter without modifying the core formulation of DARTS.
Chh
3 AN ANALYSIS ON THE CONVERGENCE OF DARTS
We start with an analysis of the weight-sharing framework used for cell-search in DARTS, and the required conditions for its convergence. We then empirically investigate whether DARTS can satisfy any of these conditions, and show that it is only capable of reaching convergence under one of them. Finally, using these observations, we will show that this type of convergence causes performance collapse, providing a motivation for A-DARTS, For the uninitiated reader, we provide a brief explanation of DARTS and the weight-sharing framework used by it for cell-search in Appendix-A.l
3.1 THE CONVERGENCE OF DARTS
Assuming the primary loss function L(...) to be differentiable, we can write the necessary and sufficient condition for optimality of the architecture parameters, (a:) as (Boyd & Vandenberghe 2014):
V (w,
where w represents the parameters corresponding to the operations, and (Td( is the loss function calculated over the validation set. Let o= be the softmax normalized architecture parameters, where (.) corresponds the softmax function. Then we can re-write as:
Jo (((a"))[((
where Jrr (a) = ( (T) corresponds to the Jacobian of the softmax function all edges, and diag(.) being the diagonal matrix. Note that given the weight-sharing framework, we can write V vpL(w,a") = dc
Op (w,a² as the sum of gradients received by each layer:
(V, =0.
Ja(a")
Now note that (.) is a block diagonal matrix, with each block corresponding to one of the edges in the architecture. Therefore, J.(.) is a positive semi-definite matrix with its null-space containing
3

---

Published as a conference paper at ICLR 2023
0.10
3x3 Gradient First Layers
Gradlent Last Layers
Skip-connection Gradient First Layers
0.05
Skip-connection Gradient Layers
ac (w, a)
0.00
apla
-0.05
-0.10
0
5000
10000
15000
20000
# Mini-batch
The exponential average (with decay rate 0.999) of the sum of the gradients of the 3 convolution and skip-connection operations for the first 5 layers and the last 5 layers averaged over all edges of the cell on the NAS-Bench-22 search space, averaged over runs with 95% confidence interval. Note that negative values in the gradient mean the value of the parameter corresponding to the operation will increasing.
Figure 2:
vectors of the following form:
T [
(4)
where the vector v consists of concatenated vectors es E of the form of E (c-111 1 1 |GE
[] (Gao & Pavel 2017). Considering the diverse set of operations used in DARTS, it is not an unfair assumption that ((w,a)) won't be in the form presented in (4). In fact, we have seen empirical evidence to support this claim, which will be discussed in Section 3.2 So assuming that V Vp[(w,)) does not belong the null space (ax), satisfying ( can only be achieved in two ways:
1. VpL(w,a") approaching O.
2. The Jacobian Jo (a*) approaching O.
3.2 THE LAYER ALIGNMENT ISSUE
Assuming that : VipL(w.a") + 0, necessary condition for V VpL(w,a") be able approach zero is that for each f we have 36" fl: V. VV
Because otherwise, we can write IV ((,,a >I2 # O. Empirically, noticed that this is not the case in DARTS. More specifically, we define the function A which we dub layer alignment as:
1 1 Σ
Vipllw,a) V (e
Aw,a)==
(2) EEEE (Vpp(waa))ll Ve ((w,)))
We noticed that regardless of the search space, the layer alignment is always near zero, i.e. the layer gradients almost orthogonal. In Figure [ we can the value of A(.,-) the NAS-Bench-200 search space (Dong& & Yang 2020) for both DARTS and A-DARTS. In Appendix A.2 we try to detect the reason behind the low layer alignment issue.
As a result of this orthogonality, we expect that under the aforemeentioedd assumptions DARTS is incapable of reaching convergence the first way, since is likely incapable of satisfying the necessary condition for it. So we will propose lower-bound on the value [VW which formalizes our claim:
Lei V.L(w.a) be orthogonal to the null space of Ja(a). and assuming we have L layers )., Let >1(a) >0 be the of eigenvalues corresponding to Jo(a). Then the square norm of gradient V.L(wa)) can be bounded by:
Proposition 1
of cells in search nefwork, have +f Vepf(w,,o)) Le L(w,a)).
[VaL(w,))]13 min As(a)) T.-in V, (. (,((a)700
(6)
4

---

Published as a conference paper at ICLR 2023
We provide proof for this proposition in 1opendixxA.100 This proposition shows that for any value of mine|] VI V we have a large enough value for L that prevents the right-hand-side of (6) to become too small. As result, for deep enough search network, the only way for to reach convergence is through reaching the softmax saturation point, wherein we have near zero eigenvectors correpooning to (a), and the normalized architecture parameters become close to one-hot vectors. This phenomenon has been observed in the literature (Bi et al. 2019 Dong & Yangl 2020), where DARTS usually converges to a softmax saturation point with all of the edges selecting the skip-connection or none operations, depending on the search space. In Figure-33 3] we can see this effect, where the absolute changes in the softmax nor- malized architecture weights of DARTS during search is rapidly growing, compared to A-DARTS which reaches plateau during search. Note that converging to a softmax saturation point is not an inherently bad outcome, because it corresponds to near zero optimality gap when using continuous relaxation (Boyd & Vandenherghe| 2014). But as we will see, this characteristic gives an advantage to layers closer to the output in selecting the optimal architecture, contributing to performance collapse (Zela et al. 2020).
3.3 THE PERFORMANCE COLLAPSE
Another issue caused by low layer alignment is that the optimal architectures corresponding to each layer are going to be vastly different. In Figure we can see the exponential average of the sum of the gradients for convolution and skip-connection operations in the first 5 layers (furthest from the output of the network) and the last 5 layers (closest to the output of the network) averaged over all edges in a search cell on the NAS-Bench-201 search space (Dong & Yang 2020). As evident, the optimal cell corresponding to the shallower layers (closer to the input) mostly contains convolution operations, while the optimal cell corresponding to the deeper layers (closer the output) mostly contains skin-connection operations. We attribute this observation to skip- ciipcconestions having proclivity towards allowing the gradients to flow fur- ther into the network, while convolutions provide more complex features for upper layers. Furthermore, due to gradient vanishing, the gradients corresponding to the kkip-connection operations on average have larger magnitudes, with the magnitude of the layer gradients gradually becoming smaller the deeper we go. As a result, (u.(w will have much smaller negative values corresponding to the skip-connection operations.
Using these observations, we can clearly bridge the analytical gap between the performance collapse problem and the layer alignment. Due to the weight-sharing structure, the value of the function a(w,a) is close to zero. As result, DARTS can only achieve convergence in the softmax saturation points. Furthermore, given the low layer alignment, the optimal ar- chitecture from the perspective of the layers are going to be extremely varied, with deeper layers having strong inclina- tion towards ikip-connections. This gives skip-comin skip-connections a clear advantage due to the larger magnitude of their gradients. So as the search progresses, the architecture parameters cor- responding to the skip-connection operations become larger than the convolution operations, eventually resulting in sat- uration point that overwhelmingly selects skip-connection operations (Bi et al. 2019 Chu et al. 2021). As the depth of the search network increases, vanishing gradient becomes more severe, further exacerbating the issue (He et al.] 2016). This is evident when comparing the results of DARTS its original search space and the NAS-Bench-201 search space, which utilizes deeper search networks (Dong & Yang 2020). Similarly, increasing the number of iterations used for the search results further exacerbation the performance collapse problem (Bi al. 2019). which can be attributed to the severity of vanishing gradient increasing as the loss function approaches its optimal point (He et al. 2016). These observations motivate us alleviate both of these issues caused by low layer alignment through increasing the value of A(w,a). In Appendix A.3 we provide a thorough analysis that connects our method to some of the prior works.
1.4
MARTS
1.2
1.0
0.8
0.0
0.4
0.0
0
10
20
30
40
50
Epach
Figure 3: The El norm of changes in Noftmax-normalle
architecture weights, averaged over 4 runs with 95% confi- dence interval. The search is performed on CIFAR-10.
4 THE PROPOSED METHOD TO MITIGATE PERFORMANCE COLLAPSE
In the previous segment, we saw that the main contributor to the performance collapse problem is low layer alignment (i.e. a(w,a))))) Now we will introduce two regularization terms based on this value to solve
5

---

Published as a conference paper at ICLR 2023
performance collapse. Furthermore, will introduce method to estimate the gradient corresponding to the regularization terms that grow linearly with the size of the network.
4.1 THE REGULARIZATION TERM
For the sake of simplicity, let's assume aim search for single cell. To increase the value of A (w,aa), we can directly use the function itself, since it is a differentiable function. So we define our main regularization term the function A(waa,, which we aim to maximize.
In our preliminary experiments, we noticed that in case the dimension of a is large, A(w,a) a) only concentrates on the operations with relatively larger gradients. This means that while the angle between the gradients of the layers is reduced, only the gradient of small portion of the operations will have the same sign, resulting in lower performance. So given "g = Vip((w,a) and g Ver ((waaa))) we will also introduce another regularization term that aims increase the correlation between the signs of the gradients of each layer:
1
E TE gge
(+(wa))=
ECE
where l-] corresponds to the clement-wise absolute value function. Note that while the optimal point of where e is a constant, the optimal point of A (w;o) ()), corresponds to sigu( V)p[(w,aa)) ((w,,)) where sign(.) .) is the element-wise signum function.
(w:ar) corresponds to V=
Given one of the regularization terms above, the li-level optimization problem of A-DARTS will become:
min
Cual (ual(w" (w)(()))))
s.t. w
(train(w,0) - AA Arain (w,a),
where insert the regularization term in the inner objective. In Appendix A.4 show that optimizing the regularization term w.r.t. the outer objective performs poorly. Similar to most variants of DARTS, use linear scheduling for À (Zela al. 2020; Chen & Hsieh, (2020): i.e. A = Xx 4. where is the
number of current epoch and T Is the total number of epochs used for search.
4.2 ESTIMATING THE GRADIENTS OF THE REGULARIZATION TERMS
Assuming the gradients corresponding to the operations are non-zero, both A(w,a) and A (w) are differentiable. So their gradient with respect to w can be written as:
A[w(aa== =
2 C(w,a) Sh
(9)
where we have:
g A
$g
g |1
I
1l'
(10)
Te
g N4
diag(sign( (g)) diaggsign( 9))
1 gl
Po Pol gl
with I corresponding the identity matrix. Note that calculating VaA(w.a) will be of ||/l due to the existence of second derivative matrices for each layer. So a direct calculation of (9) is out of the question. But similar to (Liu al. 2019), we can use finite difference approximation to estimate (u,A))))
Let P(w,P) be the loss function function of w and Pe gLxaa where:
(=) (a)
a(a))"
(11)
corresponds to the softmax normalized architecture weights used for each layer in the search network. Furthermore, let A E [ al be:
A Eeegr Eezz
(12)

---

Published as a conference paper at ICLR 2023
Table 1: Comparison with state-of-the-art on the NAS-Bench-22 search space. Mean and standard deviation of the accuracy were calculated over 4 independent runs. The search is performed on the CIFAR-10 dataset. The best performance is boldface. (1st): first-order, (2nd): second-order
CIFAR-10
CIFAR-100
ImageNet166120
Method
Valid
Test
Vald
Test
Valid
Test
DARTS (1st) Luu ct al.
2019
0..00
51:0=00000
15.03+0000
10.0
16.43=0.00
16.32=0.00
DARTS (nd)
Lau
2019
39.77 0.00
54.30=00.00
15.03++0.00
15.61+00.00
16.43+0.00
16.32=0.00
GDAS Dong
Yang
2019
80.89+0..0
93.61 0.09
+1.34+0..4
70.70+0.30
41.39+1.33
-1.7
SNAS Xie el
2019
00.10=11.04
92.77-00.88
69.69-12.30
69.34-11.99
42.8411.799
43.16
DSNAS Hirrer di. 2020
89.66=0.229
93.08
30.87
31.01+16.00
40.61+00.09
41.07=0.09
PC-DARTS Xii
2020
80.96+0.315
93.41 0..30
67.12++0.39
67.48+00.99
40.83+0.08
41.31-0.22 00.2
IDRRTS Zhung
89,86 0.60
93.58
70.57
70.83:10.48
40.38110.59
40,89-0.68
DARTS- hu ei al
2021
91.03:0.44
93.80=0.40
1..36:11.51
71.53:1.511
44.811146
45.12-0.82
B-DARTS Yo al 2022
91 01.55+0000
94.36 + 0.00
73.49 0.00
73.51 0.000
46.37
46.34 0.00
A-DARTS Miuier
0.055=0.00
94.36=0.00
03.49=0.00
73.51
46.37=0.000
0.034=0.00
Optimal Dong & Yang 2020
91.61
94.37
73.49
73.51
46.77
47.31
Then we can write:
(w,a) 1
-uull, +0(2),
(13)
2e
where we set == F The cost of using this approximation technique is two extra
pass, which is of O[w]| al) [[]]). So we have eliminated the L-|a: factor from the complexity of A-DARTS, which significantly reduces the computational complexity, while completely eliminating the memory overhead. In Appendix A.8 we provide the cost of our method in terms of GPU days along with another approximation technique which can reduce the overhead of A-DARTS significantly.
5 EXPERIMENTAL RESULTS AND DISCUSSIONS
In this section, we will show the effectiveness of A- A-DARTS by performing experiments on the 201 (Dong& & Yang 2020) and DARTS (Liu et al. 2019) search spaces. We will then demonstrate the robustness of A A-DARTS by performing more experiments on the reduced DARTS search spaces proposed different datasets. Finally, by performing the effects of the hyper-parameters of A-DARTS over the performance of DARTS. For all experiments, we report the mean and the standard deviation of A-DARTS for 4 independent searches with different random seeds, as well as the result of the best performing architecture, except for the experiments on the reduced DARTS search spaces, wherein only the best performance is reported. In all expeeeimeeeet unless explicitly mentioned otherwise, the search is performed on CIFAR-10. For details of the experiment settings, we direct the reader to Appendix-AA. In Appendix-AA.I I we provide the architectures discovered by A-DARTS on these search spaces and datasets. For further experiments, we refer the reader to Appendix-AA.. Appendix AA6 and Appendix-A.7
by (Zela et al. 2020)
ablation study, will analyze
experiments,
5.1 NAS-BENCH-200 SEARCH SPACE
In order to demonstrate the effectiveness of A-DARTS in reproducible setting, we used the NAS- Bench-201 search space for our first set of experiments. In Table we can see the results of A-DARTS compared to other DARTS-based methods. We omit from reporting the results of A+ (w.ax) here, since it performed similar to A(w,a). As evident, A-DARTS performs exceptionally well compared to the baselines, surpassing all but one of them in terms of the mean and standard deviation of the performance. Compared to the current maate-of-hhe-and YYe et al. 2022), A-DARTS discovers the same architecture in all 4 experiments. Considering how close to the optimal the performance of A-DARTS is, we attribute the slightly less than optimal performance to the high variance of the CIFAR- 10 dataset. Furthermore, we note that the method proposed in (Ye et al. 2022 converges its optimal architecture very early in the search process (the first 10 epochs), which may indicate premature convergence to sub-optimal architecture.
In comparison, A-DARTS does not reach its optimal performance until late in the search procedure, where the performance of the search model is close to optimal and the correlation between the gradient of the layers is very high. In fact, in the DARTS search space we will show further evidence for this claim, where A-DARTS is capable of surpassing the performance of YYe et al.. 2022) on the DARTS search space by a large margin on two datasets. In Figure [ and the accuracy of the architecture CIFAR-10, which reaches its optimal point around the 40th
we can see a clear correlation between the layer alignment
7

---

Published as a conference paper at ICLR 2023
Table 2: Comparison with lfte-of-the-ar the DARTS search space. The first block contains methods that report the best performing architecture, while the second block contains methods that report the average of multiple searches (except for results ImageNet, where always the best performance is reported). Our reported mean and standard deviation of the accuracy are calculated over 4 independent runs. We perform the search the CIFAR-10 dataset. The best performance is boldface each block. (Ist): first-order, (2nd): second-order, (avg): average performance, (best): best performance. Different search space used for ImageNet.
CIFAR-10 10
CIFAR-I00
ImageNet
Method
Params (M)
Acc
Parms [M)
Test Acc
Parass (M)
Test (58)
NASNeE-A Zoph nl.
2018
9.3
97.35
33
83.18
5.3
74.0
DARTS (IsO Li ot 2019
3.1
97 37.00=0.11 1/
3.1
82/16
DARIS (2n))
97.24220.0
SNAS Xie et al. 2019
2.8
97.15.10.02
2.8
82.45
4.3
72.7
GDAS Dong Yang.
2019
97.07
5.3
P-DARTS [Luu
2018
344
97 50
3.6
82.51
5.1
75.3
PC-DARTS SEDARISTAUULE cl (VAA
3.6
97 03.07
3.6
83.10
5,3
75.8
DINAS Chen
400
97 00.06
5.2
75.8
R-DARTS (Zaln al (1:0
07.0
F0:26
P-DARTS (iu 2018
3.3+0.21
97.19=0..1
SDARTS-ADN CCixa de Hiscch 2020
3.3
97.39
5.1
71.8
(uu
97.51110006
DARTS#PT Wang el al. 2021
3.0
97.39-40.08
4.6
74.5
DAKIS- (Ch al. 2012
3.3.10113
97.41
3.4
02.49:1:.25
4.9
76.2
DARTS Ye
0.01
97.49
3.8
83.48 0.003
5.4
75.8
A-DARTS Alw,0)
lavg)
3.5:10.13
97.1820.111
3,6100133
03:47:10:00
DARTSS A(azz) (best)
97.59
$3.64
A-DARTS (w,,))
lavg)
3.6++0.13
07.57
3.6+-0.1
83.85 4-.38
A-DARTS (g(wr)) (best)
97.65
84.20
5.2
75.7
search epoch. Comparing the results to DARTS, we see that the performance of the discovered architecture declines consistently, eventually reaching an extremely low point.
5.2 DARTS SEARCH SPACE
In order to show that A-DARTS is effective in mitigating the performance collapse problem in complex search space, we used the original search space of DARTS proposed in (Liu et al. (019) for our second experiment. In Table-Z we can see the results of A-DARTS compared to the baselines. A-DARTS able to achieve sate-of-the-art results on the CIFAR-10 10 and CIFAR-100 datasets, surpassing the baseline models by a large margin on the more complex dataset CIFAR- 100. Furthermore, A-DARTS achieves comparable results ImageNet, with about 0.1% difference in accuracy compared to the state-of-ther the-art results on the same macro-ardee arcbro-ccchttcture Furthermore, compared to (Ye et al. 2022). we see that A-DARTS achieves better performance two of the three datasets, supporting our claim that 3-DAATTS may be converging to sub-optimal architectures due to pre-mature convergence. Interestingly, compared to most baselines, A-DARTS does not select unnecessarily complex architectures, with the number of parameters averaging around 3.5 on A(w,a) and 3.6 on A+ (w,a) on CIFAR-I0. This means that the architectures are much more efficient terms of performance, further proving the superiority of A-DARTS. This is especially the case on the ImageNet search space, where our model is well within the boundaries of the mobile setting introduced in (Liu et al. 2019).
Comparing A(w,a) with A (w.a), we see that the second function achieves a much better performance. Comparing the discovered cells, we noticed that A (w,a) a)) usually discovers deeper cells with more parametric operations, quality that highly correlates with the performance of the model. This observation can be attributed to the large dimension of a on this search space.
5.3 REDUCED SEARCH SPACES
In order to show the effectiveness of A-DARTS in alleviating the performance collapse problem even in the most extreme circumstances, we used the reduced search spaces proposed in (Zela et al. (020) for further experimentation. In Table we can see the results of A-DARTS compared to the baselines. Out of 12 different experiments performed, A-DARTS performs better than the baselines in 10 of them, with its performance in the other two experiments being comparable to the baselines. Following these observations, we can claim that our model robustly stepping out of the performance collapse trap without any modifications to the original formulation of DARIS.
8

---

Published as a conference paper at ICLR 2023
Table 3: Comparison with sfate-of-the the the reduced DARTS search spaces. The test rate (%) of
the best performing architecture from independent runs is reported. The best performance boldface.
Dataset
Search Space
DARTS
PC-DARTS
R-DARTS
SDARTS-RS
DARIS+PT
A-DARTS
Si
3.84
3.11
2.78
2.78
3.50
2.83
4.55
3.02
3.31
2.75
2.79
2.56
CIFAR-10
3.34
2.51
2.51
2.53
2.49
2.38
S4
7.20
3.02
3.56
2.93
2.61
2.16
24.25
23.51
S2
26,05
22.18
22.41
22.28
23.16
21,68
CIFAR-100
28.90
21.69
23.90
21.08
22.03
21.03
22,85
21.50
21.94
21.46
20.80
20.65
S1
4.58
2.47
4.79
2.35
2.62
2.30
SVHN
2.42
2.51
2.39
2.53
S3
3.11
2.41
2,18
2.36
2.12
2.31
3.05
2.43
2.50
2.46
2.42
2.34
C100 10
44
0.0023 000
0.21
0.0023
0.125
0.20
1.23
15.
200
Number Soarch Spocts
(a) Effects of co. Figure An analysis of the effects of hyper-parametert on A-DARTS, We report the mean and 95% confidence interval of CIFAR-10 validation accuracy, calculated over runs. (a) Effects of eo over the accuracy the approximation and the performance of discovered architectures. (b) Effects of À over the performance of the discovered architectures. (c) Effects of the number of search epochs the performance of the discovered architectures. The search is performed on CIFAR-10.
(b) Effects of A.
(c) Effects of number of epochs.
Comparing the baselines, we can see that while DARTS+PT performs better the S4 search space, SDARTS performs mostly better on the other search spaces. Furthermore, we see similar inconsistencies in PC-DARTS and R-DARTS, where they both perform well S3 but usually not well on the other search spaces. On the other hand, A-DARTS consistently performs well across all search spaces and datasets, which further shows the robustness of method.
5.4 ABLATION STUDY
In order to assess how the hyper-parameten affect A-DARTS, conducted ablation study 6o and X on the NAS-Bench-201 search space. Furthermore, to show that A-DARTS has solved the performance collapse issue, we perform the search for longer epochs (starting from 50, up to 400 epochs) following (Chu
al. 2021 Ye et al. 2022). In Figure 4al we can see the effect of En over the performance the discovered architecture. We observe that smaller values of correlate positively with the quality of the discovered architecture. Considering that lower values of co correspond to lower approximation error in (13), this expected behavior. But we note that due the problem of round-off error expect this trend to reverse for smaller values of (Shen et al. 2020). In fact, we can see that the variance of the performance is already starting to increase for the smallest value of eg-
Figure shows the effect of over the performance of the discovered architecture. Clearly, the model very sensitive the value of A, with the best performing value and the worst performing value having a difference of about 2% in their mean accuracy. Furthermore, the best performing value at A = 0.125 corresponds an average length normalized dot-product of about 0.7, while the worst performing value x=0.5 is around 0.9. So we are well within the realm of diminishing returns for larger As, and as result of increased emphasis on the regularization term the Pareto-optimal point, we observe negative effects from the main objective being sidetracked in the form of lower quality architectures.
We can see the performance of A-DARTS for different numbers search epochs in Figure Unlike other methods that try to address the issue of performance collapse for larger number of epochs (Bi) cet al. 2019 Chu et al. 2021), we see a negligible drop performance (about 0.4%) when comparing the best performing and the worst performing setting. This is clear indication that the models have achieved convergence at around 100 epochs, reaching a local-minima. that corresponds to near-optimal architectures. In fact, clearly the absolute changes in the softmax normalized architecture weights has reached a plateau in 50 epochs of search, resulting in concave curvature.
observe further proof for the convergence of A-DARTS in Figure where

---

Published as a conference paper at ICLR 2023
6 CONCLUSION
In this paper, we use a set of analytical and empirical tools to show that the main reason behind performance collapse DARTS is the lack of correlation between the gradient of its layers, caused by the multi -ulti-layer structure and the use of weight-sharing for cell-search. Based on the aforementioned analysis, we propose A-DARTS which manages to achieve better or comparable performance when compared to state-of-the-art baselines. Here, we focused our discussion and experiments on DARTS, however the analysis and ideas are extendable to any other type of differentiable NAS model that performs cell search using weight sharing, which plan to study in future works. Furthermore, we point out to the cost of estimating the gradient of the regularization terms proposed here as another potential subject for further research.
ACKNOWLEDGMENT
We thank colleagues and students at the University of Tehran for all the useful discussions. We would like also thank researchers Google for feedback, in particular Hanxiao Liu, Xuanyi Dong, and Avital Oliver for their detailed and valuable comments on the paper. We also thank the reviewers for their constructive and helpful feedback, which helped improving the quality of the paper.
REFERENCES
Bowen Baker, Otkrist Gupta, Nikhil Naik, and Ramesh Raskar. Designing neural network architectures using reinforcement learnin. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL
net https:// http:r/wwwweee.eee/ooooooeeeeedo wow.moidduuiiiooo Slc2vveeee
Bowen Baker, Otkrist Gupta, Ramesh Raskar, and Nikhil Naik. Accelerating neural architecture search using performance prediction. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Workshop Track Proceedings. OpenReview.net, 2018. net
URL htt.lttpps//ppereeeeee wttp://www.eeeeouunneeeeeee. httr://orummiidmdmm HI-HG
Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, and Quoc V. Le. Understanding and simplifying one-shot architecture search. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning. ICML 2018, Stockholnsmässann Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 549-55995 PMLR, 2018. URL
ngs. ml www.s/ressvvee /tt//beedrrr 8a html
Kaifeng Bi, Changping Hu, Lingxi Xie, Xin Chen, Longhui Wei, and Qi Tian. Stabilizing DARTS with amended gradient estimation on architectural parameters, CoRR, abs/1910.11831, 2019. URL
http://arxiv wttpoorggga///o/// 1910.
11831 11881
Stephen P. Boyd and Lieven Vandenberghe. Convex Optinization. Cambridge University Press, 2014. ISBN 978-0-521-83378-21 doi: 10.10177CH89978051 1804441. URL. htt.tps:/wweb..ttaaff...
edu/ & 87bboyd/ cvxbook/
Andrew Brock, Theodore Lim, James M. Ritchie, and Nick Weston. SMASH: one-shot model architecture search through hypernetworks. In 6th International Conference Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018 Conference Track Proceedings. OpenReview.net, 2018. URL http://pppreeeeeeeeeeeeeee net /ane//for wtr/moormmm////m intoormriiiiioo iini2idd-iiee rededdd
Han Cai, Tianyao Chen, Weinan Zhang, Yong Yu, and Jun Wang. Efficient architecture search by network transforation. In Sheila A. Mcllraith and Kilian Q. Weinberger (eds.), Proceedings of the Thecoy- AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (1),I-18), and the 8th AAAI Symposium Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pp. 2797-2794. AAAI Press, 2018. URL https: : www. aaai org/ ocs index.n.n.. php/ AAAI AAA.AAAAAA................. http:/AAAA111////////// wwapee//eeiiiee 16755
Xiangning Chen and Cho-Jui Hsieh. Stabilizing differentiable architecture search via perturbation- based regularization. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 Proceedings of Machine Learning Research, pp. 1554-1565. PMLR, 2020. URL UR.ptttp:: httt::://proceedi ngs. pre.s / 19 wwc.en00...... httl
10

---

Published as a conference paper at ICLR 2023
Xiangning Chen, Ruochen Wang, Minhao Cheng, Xiaocheng Tang, and Cho-Jui Hsieh. Drns: Dirichlet neural architecture search. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021a. forum? i d=9FWass 6YbmB33
Xin Chen, Lingxi Xie, Jun Wu, and Qi Tian. Progressive DARTS: bridging the optimization gap for NAS in the wild. Int. J. Comput. Vis., 129 635, 2021b. doi: 10.1007/s1 1263-020-01396- URL
https://doi.com .wrg D. 1007/ s 0tt.:///00-00111112200000----110
Xiangxiang Chu, Xiaoxing Wang. Bo Zhang, Shun Lu, Xiaolin Wei, and Junchi Yan. DARTS- robustly stepping out of performance collapse without indicators. In 9th International Conference on Learning Reprexentations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https://opentevw ew
(it.mmmmdiiHHHH. d= 6.I.
Xuanyi Dong and Yi Yang. Searching for robust neural architecture in four GPU hours. In IEEE Confer- ence on Computer Vision and Putemn Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 1761-1770. Computer Vision Foundation IEEE, 2019. doi: 10.1 100110////...01111 2019.001811 URL http: /ttp:://penaccess..t thecvf..eccccccccm com
Searching For a Caaouut_Neuuraee Architeerrrreere........r..e. in Four GPU Hours CVPR 2019 paper. ht.l
contot/eennooomootteeoeoo content. CVPR
Xuanyi Dong and Yi Yang. Nas-bench-201: Extending the scope of reproducible neural architecture search. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL /tt.itttpp.///eeeeeeeeeee net. htr///ormm? id=Hoxy2kBKDr
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. J. Mach. Learn. Res., 20:55:1 2019. URL http:/t::///....////////////////
html
Bolin Gao and Lacra Pavel. On the properties of the softmax function with applicatiion in game theory and reinforcement learning. CoRR, ths/1704.0099)) 2017. URI.
00805
Klaus Greff, Rupesh Kumar Srivastava, and Jürgen Schmidhuber. Highway and residual networks learn unrolled iterative estimation. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL
ht htttp://openreviieet net/ wwtp::eeee//ee.eeeeeeeee Eorum?id Swr.idd--SnSShxeeeee
Yuchao Gu, Lijuan Wang, Yun Liu, Yi Yang. Yu-Huan Wu, Shao-Ping Lu, and Ming-Ming Cheng. DOTS: decoupling operation and topology in differentiable architecture search. In IEEE Confer- ence on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pp.
12311-12320. Computer Vision Foundation / IEEE, 2021. doi: 10.1109/CVPA44437.222 2022.444333.21111113
URL https: http:////ppcccccssceo thwaaacccccsss..t..ecceeeee com/ /ccooottett////r2222//taaallllla
DOTS Decoupl ing Operation and Topolocy lil_iinniiiieee Differentiablee Architecture. Bearch__vPP2 2021 wwwpr. html
Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uniform sampling. Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision ECCV 2020 16th European Conference, Glasgow; UK, August 23-28, 2020, Proceedings, Part XVI, volume 12361 of Lecture Notes in Computer Science, pp. 560. Springer, 2020. doi: 10. 11000/97--30000-5555555 --2. BtLptttss://ee org/10 10000..0079770
030 58517-4414441 32
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Patiern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 770-778. IEEE Computer Society, 2016. URL htthst/:: doi wrw./og....... wwt.99/////2222 2016.
doi: 10. 10.11099CVR.22016.900
Shoukang Hu, Sirui Xie, Hehui Zheng, Chunxiao Liu, Jianping Shi, Xunying Liu, and Dahua Lin. DSNAS: direct neural architecture search without parameter retraining, In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp. 12081 102899 Computer Vision Foundation / IEFE, 2020. doi: 10.1 WBW.CCCVR4220..0000000001
11

---

Published as a conference paper at ICLR 2023
URL httpa://openacceee thecff content ceppsteee____e____ee 2020/htm1 Hu. DSNAS Parameter Retraining
Direct Neural Architecture Beerch
CVPR 2020 paper. paper.html
Stanislaw Jastrzebski, Devansh Arpit, Nicolas Ballas, Vikas Verma, Tong Che, and Yoshua Bengio. Residual connections encourage iterative inference. In 6th International Conference on Learning Repre- sentations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. net wttf//ormmmdddooo
OpenReview.net, 2018. URLLhttps://openrevvi
HgAZ
Yanxi Li, Zhaohui Yang, Yunhe Wang, and Chang Xu. Adapting neural architectures between domains. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-FForina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020. NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL https::/prroeedingg http:://pprrreediinnneeee......s http:/ppps.cppppp///2200/0 httpa///r//222/aahh/ 08 (at866444444222888eeaee666113111 Aatsseett.....t--aatattt.....c. html
Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (eds.), Computer Vision ECCV 2018 15th European Conference, Munich, Germany, September 8- 14, 2018, Proceedings, Purt I, volume 11205 of Lecture Notes in Computer Science pp. 19-35. Springer, 2018. doi: 10.1007/978-3-030-02266--- httLhhtp::///t ptstps://doi.. org 10. 1 0C 7/ 978 3 030- 01246-122660 2
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: differentiable architecture search. In 7th International Conference on Learning Representaations, ICLR 2019, New Orleans, LA, USA. May 6-9, 2019. OpenReview.net, 2019. htLttt:::
net htt.://ww...t.rrommmooccceeeeeoo
Luyan Liu, Zhiwei Wen, Songwei Liu, Hong- Yu Zhou, Hongwei Zhu, Weicheng Xie, Linlin Shen, Kai Ma, and Yefeng Zheng. Mixsearch: Searching for domain generalized medical image segmentation architectures. CoRR, abs/2102.13288, 13280, 2021. URL http://aaivvvv/.. org/ abs / 221/22222..02 13280
Carl Dean Meyer. Matrix Analysis and Applied Linear Algebra. SIAM, 2000. ISBN 978-0-898714. 54-8. doi: 10.1137/199308988199522 URL hhttps://my/ s sia..aaaaa.....mmaaaae...a wwt.:oorr//oteeeooucec.....e wilewprohuct/PProddccccee
Hieu Pham, Melody Y. Guan, Barret Zoph, Quoe V. Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockhelmsmüssan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 4092-4101. PMLR, mlr.
2018. URL B.RL/httpp//prrceeeeii htwp://gg..m...eee///e/eess
porees/v80. hhtp/phamll..m 8a html
Esteban Real, Sherry Moore, Andrew Selle. Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoc V. Le. and Alexey Kurakin. Large-scale evolution of image classifiers, In Doina Precup and Yee Whye Tch (eds.). Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 2902-2911. PMLR, 2017. URL
ngs mlr. www.s/wwwsss.oemm wwwp//////aa1l0 7a. html
Pengzhen Ren, Yun Xiao, Xiaojun Chang, Poyao Huang, Zhihui Li, Xiaojiang Chen, and Xin Wang. A comprehensive survey of neural architecture search: Challenges and solutions. ACM Comput. Surv, 54 (4):76:1-76:34, 2021. doi: 10.1445/3447582 URL ltttt::///ooii
3444822
Siyuan Shen, Tianjia Shao, Kun Zhou, Chenfanfu Jiang, Feng Luo, and Yin Yang. Second-order neural network training using complex-step directional derivative. CoRR, abs/2009.07098, 2020. URL
https://arxiv. org. aal////ss//000 /ttp///0.00000000000
Kenneth O. Stanley and Risto Mikkulainen. Evolving neural network through augmenting topologies. htlphttts:://a
Evol. Comput. 10(2):99-12) 2002. doi: 10. 10.1162/10636550022206999
10. 00.1162//666666622211168
Ruochen Wang, Minhao Cheng, Xiangning Chen, Xiaocheng Tang, and Cho-Jui Hsieh. Rethinking architecture selection in differentiable NAS. In 9th International Conference Learning Rep- resentations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL
12

---

Published as a conference paper at ICLR 2023
Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. SNAS: stochastic neural architecture search. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA. May 6-9, 2019. OpenReview.net, 2019. utttttt:://
Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Guo-Jun Qi, Qi Tian, and Hongkai Xiong. PC-DARTS: partial channel connections for memory-efficient architecture search. In 8th International Conference on Learning Representaaiinss, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
net http::/rwweeeeeeeeeeeroeoaaaaaaa
Song Xue, Runqi Wang, Baochang Zhang, Tian Wang. Guodong Guo, and David S. Doermann. IDARTS: interactive differentiable architecture search. In 2021 IEEF/CVF International Conference Com- puter Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pp. 1 1143-1152, IEEE, 2021. doi: 10. (0000//CCV44922222100002222 URL // //tpi///oiioooo org/ 10 1109 00120
Peng Ye, Baopu Li, Yikang Li, Tao Chen, Jiayuan Fan, and Wanli Ouyang. 3-darts: Beta-decay regular- ization for differentiable architecture search. CoRR, abs/2203.00 2t0.//22.....55. 2022. doi: 10.485500/rXXvv.223.. 01665. httpttss:: //tttps::/dd...o. www.oioorg//... wttp://...r........////.../ 48550/85551i1 2203. 61665
Arber 7ela, Thomas Elsken, Tonmoy Saikia, Yassine Marrakchi, Thomas Brox, and Frank Hutter. Under- standing and robustifying differentiable architecture search. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https:://pperrevile
net /eree/fforuue forum? i id=Hi hit::dd=Hllhhi gDS
Miao Zhang, Steven W. Su, Shirui Pan, Xiaojun Chang, M. Ehsan Abbasnejad, and Reza Haffari. idarts: Differentiable architecture search with stochastic implicit gradients. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 12557-12566. PMLR, 2021. URL http:::/proceed.nngeee
mlr.. press/v139/ htwp1/////haaaggll.. whwng2ls. html
Pan Zhou, Caiming Xiong, Richard Socher, and Steven Chu-Hong Hoi. Theory-inspired path-regularized differential network architecture search. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL htt httttpp:://p.oocediggg..ooo neur www.eeips...pppppe///////.. /hash./ Setb:
.ttll
Barret Zoph and Quoe Le. Neural architecture search with reinforcement learning. In Sth International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Confer- ence Track Proceedings, OpenReview.net, 2017. htttpstts:: //pporpeeevieeeeeeeee net net. forum?irummm id
r1Je8Hcxg
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V. Le. Learning transferable ar- chitectures for scalable image recognition. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 20/8, Salt Lake City, UT, USA, June 18-22, 2018 pp. 8697- 8710. Computer Vision Foundation / IEEE Computer Society, 2018. doi: 10. 1011 00907. htattpp//peeaccesss.eeee.....oo
2018 2011 Searria._Taaaneeeoooc.cttttttrrr 201_ paper.
A APPENDIX
A.1 REVIEW ON DIFFERENTIABLE NEURAL ARCHITECTURE SEARCH (DARTS)
A. CONTINUOUS RELAXATION
For simplicity, let's assume we are searching for single cell. Let the vector E R!El-1O1 be the set of parameters corresponding to the architecture, where E is the set of all the possible edges in the computational graph of the cell, and O is set of dre-defined operations. The goal is to find the optimal topology and operations in the continuous space using gradient-descent and then transfer it from the continuous space to the discrete space using the argmax operator get a near-optimal architecture.
13

---

Published as a conference paper at ICLR 2023
Specifically, let 2, be the output of the jth node in the computational graph; We can write zy as:
ΣE pb o(xr))
(14)
IEJOEO
where Po (1.)) is the softmax normalized architecture weights: po (6:)
expla (1)
DARTS then solves
Eoro Exxp(aaa
the following bi-level optimization problem find the optimal value for az: min - (a))aa)
(15)
s.t. w' (o)aa)=aggn) (trasn (w.a),
where (train(..)) and L (...) correspond to the loss function calculated on the train and validation set, and is the set of parameters corresponding the operations. There are two methods proposed by the original paper to solve the problem above using gradient descent. The first method dubbed first-order: performs gradient descent on and an interleaved fashion, while the second method dubbed second-order considers the dependency between w (a) and when calculating the descent direction corresponding a. this paper, we focus on the first-order method since it more widely used (Elsken al.) 2019
For further information, we direct the reader to the original paper (Liu al. 2019).
A.1.2 WEIGHT-SHARING FOR CELL-SEARCH
In order to perform cell-search, DARTS utilizes weight-sharing framework. Specifically, in a cell-search scheme, L layers of the same cell stacked on top of each other according pre-defined macro architecture. These cells share the same - i.e. share the same architecture parameter but have different parameters corresponding to the operations. So they have the same structure, but different operation weights. As result, we can write the gradient corresponding to the architecture parameter as the sum of the gradients received by each layer.
In order to define this weight-sharing framework concretely, let 'z correspond to the output of the jth node of the architecture graph produced by the tt layer. Then we can write the gradient received by
po (i.j) from the 8th layer as:
Oc OC ppb-))
Øxz oo(
(16)
In other words, the gradient received by pe from the 4th layer corresponds to the dot product of the output of the 7th node in 9th layer and the gradient of the jth node in 4t layer. Since we have L layers in the macro-architecture, we can write the gradient corresponding po as the sum of the gradients received by each layer:
ac
L
Oc
(17)
Op.
(15)
(15)
Now, by writing the gradient for all edges and operations for p in vector form, we have:
VpL(w,a)= = (vpc(w
(18)
A.2 THE ROOT CAUSE OF LOW LAYER ALIGNMENT
An important matter consider the root cause of low layer alignment, which can help further understand the reason behind the performance collapse, and the superior performance of A-DARTS. Here, we will consider two likely candidates that may have caused this issue:
High level of approximation in the formulation of first-order DARTS Zhang et al. 2021
Learning unrolled iterative estimates in the ResNet family of networks (Geff et al. 2017
In the following, will further investigate these candidates and their effects performance collapse.
14

---

Published as a conference paper at ICLR 2023
A.2.1 HIGH LEVEL OF APPROXIMATION IN FIRST-ORDER DARTS
In (15). the optimality of w (a) w.r.t the train loss is added to the formulation of the optimization problem of DARTS in the form of a constraint. Using the implicit function theorem, we can write the gradient corresponding to the outer optimization problem as (Zhang et al. (021):
VaL (w (a).a) Vartaal V2
Ctrain (w,a)) (V2 Ct (w,a)) " (w C,
(19)
where w = (a) and the inverse Hessian is estimated using the identity matrix, while in iDARTS the inverse Hessian is estimated using Neumann series. Note that it is entirely plausible for these approximations to be the cause of low layer alignment, and by extention, the performance collapse problem. So it may be the case that by eliminating at least one of these approximations in DARTS, the layer alignment may get improved.
(w) is the inverse of the Hessian w.I. w. In second-order DARTS,
In order investigate this claim, we apply the layer alignment function A(w;,) over the layer gradients with the approximations removed:
Ve Vep (w'
(w,o) V Ctrain (w ( Ctrain (w.a))) (troiin))
VV Loal (w.a), (20)
"p,
where can equivalently see wn (a) function of wn (p) with the softmax function removed from the abstraction. Since we need to calculate (20) for each layer, we estimate the inverse Hessian with the identity function similar to second-order DARTS. Note that estimating (20) with this simplification
can be performed similar to estimating the gradient corresponding A(w,a), by using finite difference approximation and perturbing the value of p for each layer separately.
Comparing the average value for layer alignment for first- order and second-order estimation, we did not notice any meaningful differences. In both cases, the value of A(w.a) would vary between 0.06 to 0.08 on the MAS--Bench2201 search space, as can be observed in Figure with the varia- tion resembling the noise caused by mini-batching. Consid- ering the performance of A-DARTS doesn't It improve signif- icantly until the layer alignment reaches a value larger than 0.2 (as observed in [ig we can confidently assert that the low layer alignment is not caused by the approximation of the gradient corresponding to architecture parameters.
Fras: Drder Est mation Senend Order Estimattio
0.6
0.5
0.3
0.2
0.0
Epoch
Figure 5: The average layer alignment for the first and second-order estimation of gradients of the layers, compared to A- DARTS.
A.2.2 THE ITERATIVE ESTIMATION THEORY
An interesting observation note is that the value of layer alignment is much lower for parametric operations compared to non-parametric operations. So it is completely plausible that the low layer alignment problem somehow caused by the parametric operations (specifically, convolution operations) and their dynamic with the network in some way. Here, we will show that at according to the iterative estimation theory ResNets (Greff et al. 2017). this indeed the case.
According to (16). we can write the gradient corresponding to each operation on each edge for cach layer as the dot product between the gradient of the node receiving the edge in that layer and the value of the operation itself. So we can write the correlation between the gradients of the layers & and E for operation o on edge (i.j) as:
Oc OC OC T (1.))
OL T of x.)
OC Oc T x.).
G'z, x.)=0(())))) xz)""
(21)
(1.) ppp po
of
Øx,
0'z, o x
which implicitly assumes the two layers work on the same number of height, width, and channels. Assuming ac and SC n are very similar (an assumption we will confirm empirically), then their outer product is 37y
very close to a positive semi-definite matrix. As a result, the correlation between layer gradients mainly depends on whether the value of the operations of 2,) and o( x5)) highly correlate or not.
Interestingly, the iterative estimation theory provides framework for us to answer this question. According to this theory, the layers of a ResNet model try to iteratively refine an estimate of an ideal output that will ultimately result in an accurate prediction at the classification level (Greff et al. 2017 Jastrzebski
15

---

Published as a conference paper at ICLR 2023
al. 2018). As result, see the output of each ResNet block representation in the same vector-space with each block trying to move this representation a direction that better estimates the ideal representation using its corrssonndng convolution operations.
Note that this casc, an optimal iterative estimation would correspond to set of orthogonal updates, since undoing the work of previous blocks in undesirable. Furthermore, since most of the search spaces used for DARTS (and generally, NAS) are from the family of ResNets, we can apply these findings to DARTS well. Therefore, under the iterative estimation theory, we can claim that the reason behind low layer alignment in convolution operations is the orthogonality of their output. This is inline with the motivation behind DARTS-PP -TT in (Wang et al., 202T). which showed direct connection between performance collapse and the iterative estimation theory. In this paper, we provide further evidence for this claim, but also provide a solution the problem that does not require significant changes the DARTS framework.
DARTS
Stage
ADARTS Stage #1 Stage
0.14
A-DAITS
Stage 82
0.6
0.12
A-DAITS-
A-BARTS- Stage #1
Stage
Cosine
0.10
DARTS Stage #2
-Stge
Cosine
0.5
Stage
0.08
0.4
0.06
0.04
0.02
0.2
0
10
20
30
40
50
10
20
30
40
50
Epoch
# Epoch
(a) Cosine Similarity, Convolution Operations
(b) Cosine Similarity, Node Gradients
Figure 6: The cosine similarity between the convolution operations, (b) gradients received by each node, averaged over each pair of layers and each edge. We report the mean and the 95% confidence interval for four experiments on the WAS-Bench-201 search space and the CIFAR- 10 dataset.
In order to confirm our assumptions and provide empirical evidence for our analysis, here we will provide visualization of the cosine similarity between 82, n , and the convolution operation outputs
a2 and 0C
o(fx;) and of" zz), averaged over each pair of layers on the NAS-Bench-201 search space. Since DARTS works in three stages with respect to imput scale, we will provide the aforementioned values for the three stages separately. You can observe the results in F
The first thing note is the large correlation between node gradients, which increases in higher stages. On the otherhand, we can see that the correlation between the output of the convolution operations is incredibly low. This observation confirms low correlation between the output the convolution operation, which can be attributed to the iterative estimation theory. Furthermore, we can observe that in order increase the value of layer alignment, A-DARTS focuses on increasing the correlation between the output of convolutions, and does not have noticeable impact on the correlation between node gradients.
assumption that the main reason behind low layer alignment
Given the analytical and empirical evidence provided here, we can confidently assert that as pointed in (Wang et al. (202]), one the main reasons behind performance collapse the residual connections and the ResNet-lne like structure of most DARTS search spaces. But unlike (Wang et al., (021). we attribute performance collapse to the discrepancies caused by the residual connections on the optimal architecture from the perspective of each layer, and the resulting problem in the convergence point of DARTS. The superior performance of A-DARTS compared to DARTS-PT confirms the validity of this claim, and we hope these findings can provide new directions for further research.
A.3 RELATIONSHIP TO PRIOR WORKS
In this section, we will provide an analysis on the relationship between our work and some the prior works that aim to solve the performance collapse problem. More specifically, will show that most of the papers that reduce the severity of performance collapse to some degree can be explained through our analytical framework, suggesting that they may be alleviating the symptoms of low layer alignment indirectly.
16

---

Published as a conference paper at ICLR 2023
Table 4: The test performance of architectures discovered by PC-DARTS with batch size of 256 (PC-
DARTS- and 64 (PC-DARTS--A) on NAS-Bench-201.
Method
CIFAR-10 Acc (%)
CIFAR-100 Acc (%)
ImageNet Acc (%)
PC-DARTS-256
93.41
67.48-0.889
01.30
PC-DARTS-64
93.76=00.00
71.11+0.000
41.44
A.3.1
THE CONVERGENCE OF DARTS AND ITS RELATION TO Es REGULARIZATION
In (Ye al. 2022). it has been suggested that one of the possible solutions to performance collapse is l reg-
ularization performed on the iog-sum-exp of the architecture values: (a(a)) h log log(( ( (f))))
The regularization term is then added to the loss function corresponding to the outter optimization problem. changing it to: min, L(w" (a),.)) Ca)), where y is scheduled linearly, starting from 0 and increased to a very large value (usually around 25 to 100).
Note that in the case of l regularization, the optimality condition in | can be re-written as:
V VaLual (w,a" ==>V=
(22)
where we have V voL)(a)=o(a)= Given that is normalized over each edge, we have [[{(a]]] = F So we can given lower-bound on the e norm of (aC)(a) using the Sauchy-Schwarz inequality:
| (VoL)())122 [VaCo(a)]1 0
E
(23)
O
Since the number of edges and operations are usually comparable in most search spaces (for example, in DARTS it is equal to 14 and 7. and in NAS-Bench-201 it is and 5, respectively), the lower-boum proposed above is very close to 1. Therefore, as 7 increases to very large value, the convergence point of 3-DARTS no longer correspond to the stationary point of the primary loss function. Moreover, in our experiments noticed that (w.""( usually has a very small value (in the order of O(1--))) This means that the convergence point doesn't happen on the saturation point of the softmax function, but instead, becomes dominated by the regularization term, which corresponds to the current value of p. So similar to the case of DARTS, the convergence point does not directly depend on the loss function, but is directed by the current value of a which based on previous information from the gradient of the loss, which may be inaccurate at the current point.
This analysis is supported by the experiment results provided by (Ye et al. 2022) its Figure-4c, where at the convergence point the value of a increases monotonically and linearly, while the value of does not change at all, and indeed does not correspond to the saturation point of softmax. Note that similar analysis can be done on implicit gradient and second-order methods of DARTS as well, where the convergence point no longer corresponds to a stationary point of the explicit gradient the architecture parameters
A.3.2 VANISHING GRADIENT AND THE LOW LAYER ALIGNMENT ISSUE
In (Zhou et al. 2020) it has been observed that the rate of the convergence of the search model with respect to the operation parameters is directly impacted by the architecture weights corresponding to the skip-connection operations. This observation then used incentive to propose auxiliary skip-connections by (Chu et al. (021)) dubbed DARTS-, where the effect of auxiliary connections is gradually reduced by a linearly scheduling their magnitude to approach zero. Independently, in order to reduce the memory consumption of the search model, PC-DARTS proposes performing the candidate operations on subset of the channels of the input, thereby implicitly providing an auxiliary skip-connection on a subset of the input channels (Xu et al.] 2020). Here, we will argue that both of these methods are implicitly trying to solve symptom of the lower layer alignment issue by up- weighting the gradients corresponding to the lower layers.
Firstly, in order show that the increased batch size doesn't 't benefit PC-DARTS greatly, we performed the search using PC-DARTS on a batch size of 64 and compared the results to performing the search with batch size of 256 the MMS-Bench-201 search space in Table 4 Clearly, we claim that the batch size is not the main contributer to the better performance of PC-DARTS, and instead, it is the partial-channeling that appears to be alleviating performance collapse.
Note that according to (16), we can write the gradient received by each architecture weight as the dot product of the gradient of the receiving node and the corresponding operation. While gradient vanishing
17

---

Published as a conference paper at ICLR 2023
0.02
0.02
0.00
0.01
connecttooonn
-0.02
0.00
-0.01
-0.08
3x3 Corw Gradient- first Layes
3x3 Carn Gredient Last 5 Layers
-0.10
Selp-canmetiinn Gaatnn-- trii Layers
cap-cconeattoon Gradiett
5000
10000 Mini-betch
15000
20000
5000
10000
15000
20000
Mini-batch
(a) DARTS-
(h) PC-DARTS
Figure 7: The exponential average of the gradient of the first five layers and the last five layers for DARTS- and PC-DARTS, similar to F
does not affect the operation, it does have an impact over the gradient of the loss w.n.t the receiving node. We can see this effect in Figure-Z] wherein the gradients corresponding to the first 5 layers generally have smaller magnitudes. Therefore, one way of alleviating the symptoms of low layer alignment can be reducing the effects of gradient vanishing by providing stronger skip-connections. This is effectively what is done by the auxiliary skip-connection proposed by DARTS- and the partial-channel structure of PC-DARTS, which causes limited improvements in the performance of DARTS.
In order to make our argument stronger, also provide the gradients corresponding to the first five layers and the last five layers similar to Figure both DARTS- and PC-DARTS, which you can see in Figure 7 The first thing notice is the striking similarity between the two figures, where the gradients corresponding to layers closer to the output have been extremely down-weighted companed to the gradients corresponding to layers closer to the input. This observation proves the validity of our claim that auxiliary skip-connection and partial-channeling significantly up-weights the gradients corresponding to the layers closer the input. Since these layers prefer parametric operations, we observe better performance by these models. But note that despite the improvements, these methods are merely circumventing the problem of low layer alignment, which is the reason behind their poor performance compared to A-DARTS.
A.3.3 HIGH LAYER ALIGNMENT AS A PRIOR BY ELIMINATING WEIGHT-SHARING IN HAMPLE-BASEL NAS
One of the popular methods of cell-based NAS the use of sampling candidate architectures from a pre-determined distribution, wherein the sufficient statisties of the distribution are optimized according to the performance of the sampled architecture. Among these methods, we can mention DrNAS which utilizes a Dirichlet distribution for the search (Chen al. [021a), and SNAS which utilizes concrete distribution (Xie et al. (2019). In both of these cases, the sufficient statistics are optimized according to gradient estimation technique that approximates the gradient corresponding to the sufficient statisties based on the gradients received by the sampled operations.
We argue that the mere of using sample-based NAS is a way circumvent the problem of low layer alignment. Firstly, because such formulation of the NAS problem completely eliminates the issue of convergence at softmax saturation points, thus preventing the optimal architecture be completely dictated by the layers closer to the input. Secondly, because the sampled architecture in a cell-based NAS method is used on all of the cells, sampling itself comes with an implicit prior that the optimal architecture according to each layer must be the same. Therefore, sample-based NAS can be also viewed as way to enforce high layer alignment in an priori fashion. Therefore, we can see sampled-based differenntaalee NAS as way of solving the problem of performance collapse. But we should note that since in these methods only a subset of the supernet is optimized, they usually come with the caveat of low sample efficiency and longer search time (Ren et al. 2021).
A.4 REGULARIZATION OF THE OUTER OBJECTIVE
An interesting point to consider is whether we can apply the regularization term of A-DARTS over the outer objective, which will directly optimize the architecture parameters a. Here, we will first investigate
18

---

Published as a conference paper at ICLR 2023
Table The test performance of architectures discovered by A-DARTS with the regularization term applied to the inner objective (inner), and the regularization tern applied to the outer objective (outer). The search is performed on CIFAR- 10 and the Be-BBenh--01, and the mean and standard deviation of three runs is reported.
Model
CIFAR-10 Ace (%)
CIFAR-100 Acc (%)
ImageNet Acc (%)
A-DARTS (inner) A-DARTS (outer)
93.15
72.75=0.42
45.49:10.08
91.37+1.559
67.87+1.57
41.6+-2.11
Table 6: The test performance of P-DARTS, P-DARTS- (P-DARTS with auxiliary connections), and P-DARTS with the proposed regularization term. In all experiments, the prior assumptions of P-DARTS are removed. The search is performed on CIFAR-10 and the DARTS search space, and the mean and standard deviation of three runs reported.
Model
CIFAR-10 Acc (%)
P-DARTS (w/o priors) (Chen et al. 2021b)
06.5
P-DARTS- (wo priors) (Chen et al. (022bb
97.2
A-P-DARTS (w/o prios)
97.34
this matter from a theoretical standpoint, and provide two points that shows the sub-optimality of such approach. Then provide empirical results to confirm our analytical findings.
The first problem with regularizing the outer objective has to do with the fact that the gradient of (iJ) does not directly depend on pe itself, but the output the corresponding operation o(.) and the gradient of the receiving node zy. as observed in (16). Therefore, increasing the layer alignment must be done by changing the composition of the operations of the layers in the higher layers to adjust ac while aaa
changing the composition of the operations in the lower layers to adjust o( (.) Considering the small size of a, and the weight- sharing framework, this may an impossible task that may result in overfitting on the regularization term as well.
Another issue we need to take note the connection between the problem of low layer alignment and the iterative estimation theory, noted in Appendix-A.2 Assuming the problem of low layer alignment is caused by the orthogonality of the output of convolution operations in each layer, then resolving this problem is only possible through regularizing the convolution operations themselves. While it may be possible to resolve the issue through adjusting the input, we note that as pointed the previous paragraph, this adjustment may lead overfitting due to the low complexity of the model with respect to
In order to provide empirical evidence for our analysis, we tried to perform the regularization on the outer objective. You can the results of 50 epochs of search in Table where we compared A-DARTS with inner objective regularized with A-DARTS with outer objective regularized. Clearly, we observe that regularizing the outer objective performs poorly compared the case where we regularized the inner objective. This observation further supports our analytical reasoning, and shows that performing the regularization on w yields more superior and robust performances compared regularizing a.
A.5 COMBINATION WITH ORTHOGONAL VARIANTS OF DARTS
In this section, we will empirically demonstrate that our method can be extended to orthogonal variants of DARTS to yield better and more robust results. Specifically, we will discuss two popular variants of DARTS, P-DARTS and SDARTS, and how A-DARTS can eliminate the need for strong prior assumptions or costly regularization in these methods.
A.5.1 PROGRESSIVE-DARYE AND THE PROBLEM OF STRONG PRIOR ASSUMPTIONS
Progressive-DARTS (P-DARTS) is low-cost variant of DARTS that is based performing the search in a progressive manner. P-DARTS performs the search in three stages, with each stage eliminating some of the candidate operations while also making the search model deeper and more similar to the evaluation architecture (Chen et al. (021b). One the issues with PC-DARTS is its reliance on two very strong priors in the search process: 1) limiting the number of skip-connections to two, which severely reduces the size of
19

---

Published as a conference paper at ICLR 2023
Table 7: The test performance of SDARTS-RS, SDARTS-ADV, and SDARTS-RS with the proposed regularization term. The search and evaluation is performed on CIFAR-IO and the DARTS search space, and the mean and standard deviation of three runs is reported. The search cost is reported based on GPU days on 1080 Tī.
Model
Test Acc
Search Cost
(%)
(GPU days)
SDARTS-RS (Chen & Hsieh| (020))
97.33 0.03
0.4
SDARTS- (Chen al. 2022b)
97.39
1.3
A-SDARTS-RS
97.42++0.04
0.8
the search space, 2) using dropout on skip-connections to limit their utility during search. Following (Chu ct al. 202T). here will show that A-DARTS can eliminate the need for both of these strong priors, which further validates the effectiveness of our method. You can see the results in Table-6
We can see that our method successfully removes any need for strong prior assumptions auxiliary connections in the progressive search format, achieving superior performance to both cases. We note that these results attained without any adjustment to the hyperparameters of A- A-DARTS, which is likely going to further benefit the results, considering that the progressive framework changes the macro-aree aacro-architecture and the search space significantly in each stage.
A.5.2 SDARTS AND THE PROBLEM OF COSTLY REGUI ARIZAATINN
SDARTS is variant of DARTS that aims to improve generalization by regularizing the Hessian of the loss function corresponding to the architecture parameters (Chen & Hsich 2020). The regularization is done through additive noise to the architecture parameters during the optimization of operation parameters
(w). There are tow methods proposed by (Chen & Hsich 2020) based on this idea: 1) SDARTS-RS and 2) SDARTS-ADY, SDARTS-RS is based random perturbations sampled from a uniform distribution, while SDARTS-ADV utilizes the costly method of projected gradient descent to produce adversarial perturbations. One of the issues with SDARTS-RS is that is not as effective the costly method of SDARTS-ADV, which needs more than four forward-backe bark--bbawwrrddd propagations to estimate an effective perturbation. Here, we will show that by combining our proposed method with SDARTS-RS. we will achieve better results than both SDARTS-RR RS and SDARTS-ADD ADV, at the cost of only two
propagations.
In Table-7 you can see the performance of SDARTS with and without adversarial perturbations, and compare them to that of A-SDARTS-RS, which utilizes our proposed regularization term along with the random perturbation regularization of SDARTS-RS. Clearly, our proposed regularization completely eliminates the need for costly adversarial training, reducing the cost to almost half of what SDARTS- ADV requires. This observation further proves the importance of increasing the layer alignment in DARTS, regardless of the type of DARTS variant used during search. We note that these results are attained without any adjustment to the hyperparameters of A- A-DARTS, which is likely going to further benefit the results, considering that the Hessian regularization is likely change the optimal eo due to changes in the loss landscape.
A.6 PERFORMANCE COLLAPSE WITH MORE EPOCHS
Recently, it has been shown that performance collapse happens more consistently and frequently the longer the search process is performed (Bi et al. 2019). This observation is inline with our analysis, which shows the negative impacts of low layer alignment becomes more pronounced with longer epochs of search, which is attributed to the softmax saturation point and vanishing gradients. In order to show that A-DARTS has successfully alleviated this problem even the most complex search spaces, we performed the search on the DARTS search space for 200 epochs following (Bi al. 2019). You can see the results in Table-8
Clearly, our method successfully mitigates performance collapse even in the most complex search spaces. Among the baselines, the only method with comparable performance to A-DARTS is Amended-DARTi which performs the search at 50% more search cost due to its second-order nature. Furthermore, comparing the number of parametric operations, we architecture that are not unnecessarily over peee-parameteriee.. On the otherhand, Amended-DARTTS perform poorly despite the fact that it has two more parametric operations. Furthermore, see that the
that our method is much better at discovering efficient
20

---

Published as a conference paper at ICLR 2023
Table 8: The test performance of Random Search, DARTS (first and second order). P-DARTS, PC-DARTS, Amended-DARTS, and A-DARTS after 200 epochs of search on CIFAR-10. All results, except for A-DARTS, are borrowed from (Chen et al. [02 The search cost reported based on GPU days on 1080 Ti, and inferred from reported costs for the baselines. Note that here, (#P) corresponds to the number of parametric operations in the normal cell.
Model
Test Acc (%)
#P
Search Cost
(%)
(GPU days)
Random Search (Bi 2019)
96.71
DARTS (1st) (Liu al. 2019)
93.82
1.6
DARTS (2nd) (Liu et al.
2019
94.85
0
4.0
P-DARTS (Chen et al. (021bb
94.62
0
1.2
PC-DARTS (Xu et al. 2020)
96.85
3
0.4
Amended-DAKIS (Bi
2019)
97.29
4.8
A-DARTS
97.34
5
3.2
Table 9: The test performance of A-DARTS, compared with baselines, with the search and evaluation performed on ImageNet and the DARTS search space.
Model
ImageNet Top-1 Acc
ImageNet Top-5 Acc
(%)
(%)
NASNet-A Zoph et al. 2018)
74.0
91.6
DARTS (2nd) (Lu et a
2019
73.3
91.3
SNAS Xie et al. 2019
72.7
90.8
GDAS (Dong & Yang
2019)
74.0
91.5
P-DARTS (Chnn et al.
202Tb
75.3
92.5
PC-DARTS (Xu et
2020)
75.8
92.7
DrNAS (Chen et al.
2021a)
75.8
92.7
SDARTS-ADV Chen & Hsieh 2020
74.8
92.2
DOTS (Gu et al.
2021)
75.7
92.6
DARTS+PT (Wang et al.
2021
74.5
92.0
3-DARTS (Ye et al. 2022
75.8
92.9
A-DARTS-A(w.a)
75.9
93.0
only models performing better than random baseline are PC-DARTS, Amended-DARTS and A-DARTS. Considering the connection between A-DARTS and PC-DARTS, as pointed out Appendix this observation further proves the role of vanishing gradient and low layer alignment in performance collapse.
A.7 SEARCH RESULTS ON IMAGENET
In this section, we provide the search and evaluation results for the ImageNet dataset. The search was performed on the setting proposed by PC-DARTS (Xu et al. (2020), on a search model with 8 layers (6 normal and 2 reduction cells) and 16 initial channels. Following PC-DARTS we randomly sampled 10% and 2.5% of the ImageNet dataset for optimizing w and respectively. We used a batch size of 384 and learning rate of 0.5 which linearly decayed 0.0. The evaluation was performed with similar setting, for 250 epochs with model with 14 layers and 48 initial channels. You can see the results in Table
We can see that our proposed method improves upon baselines both terms of top-1 and top-5 accuracy. We improve upon the performance of DARTS by 2.5% in terms of top-1 accuracy and 1.7% in terms of top-5 accuracy. Furthermore, we beat the best-performing baseline, namely 3-DARTS, by a margin of 0.1% in terms of top-1 accuracy and 0.1% in terms of top-5 accuracy. We also note that DrNAS utilizes a slightly different setting for search, which performs the search on a search model with 14 layers and 48 initial channels, similar to the evaluation model. Despite this advantage, proposed method beats the performance of DrNAS by a margin of 0.1% on top-1 accuracy and 0.3% on top-5 accuracy. The discovered architectures can be seen in Figure-22
21

---

Published as a conference paper at ICLR 2023
Table 10: The test performance of architectures discovered by A-DARTS with central finite difference approximation (ctr) and forward/backward finite difference approximation (f/b) for 50 epochs of search on CIFAR-10.
Method A-DARTS (ctr)
CIFAR-10 Acc (%)
CIFAR-100 Acc (%)
ImageNet Ace (%)
94.04=0.03
7..4
4.04
A-DARTS (f/b)
93.95=0.15
72.33
04.98
A.8
SEARCH COST
One of the important factors in NAS is the cost of performing the search on a large-scale dataset, which is usually measured in terms of GPU hours GPU days on the DARTS search space and the CIFAR-10 dataset (Flsken et al. 2019). Since A-DARTS requires two extra forward-backwant passes to estimate the gradients of the regularization term, it costs twice as much as DARTS to perform the search on the same dataset. Therefore, the cost of our method in terms of GPU days is 0.8 days on a GTX 1080 Ti GPU on the DARTS search space and CIFAR-IO 10 dataset, which is 20% less than the second-order DARTS (at 1.0 GPU days) and about 40% less than SDARTS-ADV (at about 1.3 GPU days), both of which perform inferior to our model.
In order to reduce the search cost and make our method comparable to most baselines, we can per- form forward or backward finite difference approximation instead of a central difference approxima- tion (Shen et al. 2020):
-ted)-
V wdd A(sa))
+O(e),
(24)
6
1 ((.l:))))VVl..P-a-.. +0(c).
budd bhudd A(w,a)=
(25)
( 2
Note that since V A(w,a) and fashion, where we perform forward estimation for one mini-batch, and backward estimation in the next. This will reduce the cost by 25%, putting our method at around 0.6 GPU days, which is only 50% more than the original first-order DARTS (at 0.4 GPU days), and comparable DrNAS with progressive search. You can see a thorough comparison with state-of-the-art NAS models in terms of performance on CIFAR-10 and search cost in TT
) are biased estimations, we can use them in an interleaved
In order to show the reduction in the accuracy of the estimation does not reduce the effectiveness of A-DARTS, we have provided the results of discovered architectures by this low-cost method compared to the central method for 50 epochs the KAS-Bench-201 search space on CIFAR-10 in Table We can see that in all three datasets, the drop in average performance is negligible (less than 0.1% in CIFAR-10 10 and less than 0.5% in CIFAR- 100 and ImageNet). Therefore, it is clear that using forward/backward finite difference approximation can be a good technique to reduce the cost of searching on large datasets, which makes our method comparable to DrNAS in terms of cost for large scale applications.
A.9 EXPERIMENT SETTINGS
A.9.1 NAS-BENCH-201 SEARCH SPACE
MAS-Bench-2011 provides DARTS- cell-search framework, where we search for a single cell containing 3 nodes and 5 operations. The performance of the discovered architecture on CIFAR-10, CIFAR- 100, and ImageNet16- 120 datasets can be attained by querrying the database provided by (Dong & Yang 2020). We used the implementation of (Dong & Yang 2020) for our search. The operation parameters (w) are optimized using stochastic gradient descent with Nestrov momentum. The learning rate is set to 0.025, gradually reduced to 0.001 using cosine scheduling. The weight decay and momentum set to 0.0005 and 0.9, respectively.
For architecture parameters (a), we use Adam with the learning rate set 14m4 and the weight decay rate set to 0.001. The momentum values 3 and B3 are set to 0.3 and 0.999, respectively. We perform the search CIFAR-10 for 100 epochs, and set 6o = -0.001 and À =0.125 for this experiment.
22

---

Published as a conference paper at ICLR 2023
Table 11: Comparison with state-o--theaar NAS models on the CIFAR-IO dataset. We provide the search cost based on GPU days 1080 Ti.
Method
Test Acc
Params
Search Cost
(%)
(M)
(GPU days)
NASNet-A Zoph et al.
2018
97.35
3.3
2000
ENAS (Pham al. 2018)
97.11
4.6
0.5
DARTS (1st) [Lu et al.
2019
97.00++0..14
3.4
0.4
DARTS (2nd) Liu et al
2019
97.24--0.09
1.0
SNAS (ii et al. (2099)
97.15
2.8
1.5
GDAS (Dong & Yang
2019)
97.07
3.4
0.3
P-DARTS (Liu et al. 2018)
97.50
3.4
0.3
R-DARTS Zela et al. 2020
97.05=+0.21
1.6
PC-DARTS (Xu et al. 2020
97.43++0.07
3.6
0.1
DrNAS (Chen et al. 2021a)
97.46-L0.03
4.0
0.4
SDARTS-ADV (Chen & Fisich
2020
97.39
3.3
1.3
DARTS- (Chu et al. (2021)
97.4
3.5
0.4
DARTS+PI (Wang al. 2021
97.39
3.0
0.8
A-DARTS M(w,ar
97.4
3.5
0.8
A-DARTS (w,a,
97.57
3.6
0.8
A.9.2 DARTS SEARCH SPACE
The search space of DARTS contains two types of cells: Normal and Reduction, corresponding to layers with stride and 2, respectively. Each cell has intermediate nodes and operations. Here, we only apply the regularization term to the normal cell, since the performance collapse not usually associated with the reduction cell (Zela et al. (020). We used the implementation provided by (Wang et al.] [021]] Chen & Hsieh 2020). The operation parameters (w) optimized using stochastic gradient descent with momentum. The learning rate is set to 0.025, gradually reduced to 0.001 using cosine scheduling. The 0.0003 and 0.9, respectively.
weight decay and momentum are
For architecture parameters (a), we use Adam with the learning rate set to 3 10 and the weight decay rate 0.001. The momentum values 31 and 82 the search on CIFAR-10 for 50 epochs, and we set the value for À to 0.25 for A(w,a) and 0.125 for (w) In both cases, we the value of eg to 0.0001.
0.5 and 0.999, respectively. We perform
A.9.3 REDUCED SEARCH SPACES
The four search spaces proposed by (Zela et al. 2020) contain some subset of the operations used in the DARTS search space, except for the fourth search space which contains convolutions and random noise operations. Otherwise, the cells have the
structure in the DARTS search space.
We perform the search and evaluation on three different datasets: CIFAR-10 10, CIFAR-100, and SVHN. For CIFAR-I0. 10, the search setting is the same in Section For SVHN and CIFAR-100. increase the batch size and the value of en to 96 and 0.001, respectively. The evaluation setting for these datasets the same as in (Zela al. 22200). For all experiments, we only report the results of A(w,a), since both methods perform similarly.
A.9.4 ABLATION STUDY
For this experiment, we used the exact same setting as Section-5.1 except for the architecture learning rate. For the experiments 6 and X, the architecture learning rate is to 3 10-4, For the experiments on the number of epochs, the architecture learning rate is set to 3 10 when performing the search for 50 epochs, and reduced to 10-4 when performing the search for more than 50 epochs.
For experiments corresponding to 6n. we set À= 0.152,, while for experiments corresponding to X, we set =00000,, and for experiments corresponding the number of epochs, we set 6n =0.0001 and =
23

---

Published as a conference paper at ICLR 2023
A.10 PROOF OF PROPOSITION |
Proof: Using simple linear algebra, we can re-write the squared norm of the gradient V. (uL(w as:
- J. (a)a) J
(26)
Now using the assumption that apf(w,a) is the sum of vectors orthogonal to the null-space of the Jacobian matrix and the fact that Jo (a) Ja(a) is a positive semi-definite matrix, we can give the following lower-bound for this term Meyer 2000):
vpl(w,a) Jo J. (aa)) VpD(w.))
min V
(27)
(a)
Since we assume the gradient vpC(w,,) can written as the sum of orthogonal layer gradients, its norm can be lower bounded by the smallest gradient (Meyer 2000):
V [p[(wa))
V. ((w
(28)
1=1
((((wa)
(29)
L L-min V Ve, ((wa)))
(30)
where 29 follows from the orthogonality. So using 27 and 30
we can get:
min (a))))
X min
min V (
(31)
A.11 DISCOVERED ARCHITECTURES
3k3
Mp,. mon. 3x3
Stp. cony
skip
(kk
[k
cop, 2pp_ooov_3x3
di coan 565
11p. pool
(b) Reduction Cell
(a) Normal Cell
Figure 8: Best normal and reduction cells discovered by A(w.a) on DARTS search space and CIFAR-10 dataset.
(a) Normal Cell
(b) Reduction Cell
Figure 9: Best normal and reduction cells discovered by A+ (w,a) on DARTS search space and CIFAR-10 dataset.
24

---

Published as a conference paper at ICLR 2023
pool 3et
dLeen 2e5
1-21
ALaan
3x3
dii
[_.44
c.kl
max 2ho
coi 5x5
[k-2]
d_oex
max moolpooo__0 3x
(a) Normal Cell
(b) Reduction Cell
Figure 10: Best normal and reduction cells discovered by A(w,a) S1 and CIFAR-10 dataset.
sep cany. 3c3
stp
Ae Coneet
2k
sip_cmsy_ 3x3
C_
skp.oeest
cop_cooV_363 313
up come 3ed
(a) Normal Cell
(b) Reduction Cell
Figure 11: Best normal and reduction cells discovered by A(w,ar) on S2 and CIFAR- 10 dataset.
scp copy 3kd
(l-2)
skip connect
sep_coow_3d
cony. 363
nos
mox
skip Aipuramict
C_(k)
Ik11
xp_comw_3va
sep.cow 3x3
(a) Normal Cell
(b) Reduction Cell
Figure 12: Best normal and reduction cells discovered by A(w,a) on S3 and CIFAR-10 10 dataset.
sep_ sep_conv_3x3
@pp_conv_3xx
sep_come conv 3x3
sep_com conv 3x3
sep cop_conv__c 3x3
sep conv 3x33
(k-2)
(k-2)
sep conv 3x3
sep conv 3x3
(k)
(k)
sep conv 3x3
sep conv 3x3
c (k-1)
c (k-1)
sep sep_cone conv _33
sep cep_conv__0 _x3
sep_ conv _onv_3x3 3x3
sep_ conv _onv_3x3 3x3
sep conv 3x3
sep conv 3x3
(a) Normal Cell
(b) Reduction Cell
Figure 13: Best normal and reduction cells discovered by A(w,a) on S4 and CIFAR-10 10 dataset.
25

---

Published as a conference paper at ICLR 2023
[k:1] diLemo 5555
sep_coev_303
comv
comy
dil
(_(k2
<kip connect
sep_com
(b) Reduction Cell
(a) Normal Cell
Figure 14: Best normal and reduction cells discovered by Mwaax on SI and CIFAR-IO 100 dataset.
(a) Normal Cell
(b) Reduction Cell
Figure 15: Best normal and reduction cells discovered by Aw,a, on S2 and CIFAR-100 100 dataset.
spp paan 3e
[b-2)
<sp_comw_5x3
sp..
xp_comy__e2 363
[k]
c_lk 1)
3x3
sqpuoven 3x3
(b) Reduction Cell
(a) Normal Cell
Figure 16: Best normal and reduction cells discovered by Muxaa) on S3 and CIFAR-100 dataset.
100 comY.
spp sop_sosv__63
MCp
[-22
3x3
ssp_coss popp.cyy..u Bx3
3x.3
epp cop_corv__0 3t3
up cony he3
xp 3t3
c.kk)
sep_cotv_ hx3
sop sop_soev.ck
scp_com J33
tep
(a) Normal Cell
(b) Reduction Cell
Figure 17: Best normal and reduction cells discovered by A(w,a) on S4 and CIFAR- 100 dataset.
SCRI
(b) Reduction Cell
(a) Normal Coll
Figure 18: Best normal and reduction cells discovered by A(w,ai) on SI and SVHN dataset.
26

---

Published as a conference paper at ICLR 2023
k_/k2)
skip connect
skip connect
sep comn
sep com 363
c.
skip_comeect
1b13
skip connect
skip counect
sep_eom 3x3
(a) Normal Cell
(b) Reduction Cell
Figure 19; Best normal and reduction cells discovered by A(w,a)) on S2 and SVHN dataset.
1d3 up Dox
Lb4
MqLe 201
IKI
(b) Reduction Cell
(a) Normal Cell
Figure 20: Best normal and reduction cells discovered by Aw,aa S3 and SVHN dataset.
sep_ comv 3x3
sep_conv _3p_conv33x3 3x3
sep conv 3x3
c_
sep_conv _o33
sep_conv _33333333333_ _3x3
C_ (k)
2009
[k-1]
sep conv 3x3
sep conv 3x3
(b) Reduction Cell
sep conv 3x3
(a) Normal Cell
Figure 21: Best normal and reduction cells discovered by Auu,,rr) on and SVHN dataset.
Ei coer
MELPON Stc
mULDE 33
Aip.
Xp
xp.oee
(a) Normal Cell
(b) Reduction Cell
Figure 22: Best normal and reduction cells discovered by A(w,a) on DARTS search space and the ImageNet dataset.
27


Reviews:
[{"review_id": "xn5t2n1PVld", "reviewer": "Reviewer_iyeQ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes two new regularization terms to prevent performance collapse by harmonizing operation selection via aligning gradients of layers. The extensive experiments demonstrate its superiority. ", "summary_of_the_review": "I give 6 due to several concerns in the weakness part. I may change the score according to the author's response. ", "strength_and_weaknesses": "Strength:\nThe observation and analysis of performance collapse in darts is from a novel perspective.  \nAnd the regularization terms used in bi-level darts are solid and useful. \n\nWeaknesses:\n1. In table 3, CIFAR10 and SVHN dataset on S1 search space, the performance of the proposed method is worse than previous methods.  Seem there are no detailed analyses of this phenomenon.  It is better to show the visual comparisons of searched architectures between previous methods and the proposed method (if possible), and compare the values of layer alignment metric that these previous methods can obtain, on S1 search space (if possible). \n2. What's the effect when putting the proposed regularization term for upper-level optimization on  $\\alpha$  in bi-level Darts?  The proposed metric is currently only used in lower-level optimization for $w$.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "lfYcaiGt4w", "reviewer": "Reviewer_2qFj", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new differentiable architecture search (DARTS) variant to alleviate the performance collapse problem in DARTS. The authors first give theoretical and empirical analysis of the convergence of DARTS, which shows that the weight-sharing framework causes the undesirable skip-connection dominance problem in posterior layers. Based on the analysis, the paper introduces two regularization terms to alleviate the performance collapse problem via aligning gradients of layers.\n", "summary_of_the_review": "Overall, I think this paper is a borderline paper. My major concern of this paper is the limited contribution of novelty as well as the performance to the community.", "strength_and_weaknesses": "Strengths:\n1. This paper introduces a function named layer alignment to measure the correlation between the gradient of layers in DARTS, theoretical and empirical results show that this function might be a good indicator of the performance collapse situation. This could be a benefit for future works to better understand DARTS.\n\n2. The proposed regularization terms are simple to adopt, and they can alleviate the performance collapse problem according to the experimental results.\n\nWeaknesses:\n1. The performance collapse problem in this paper is not novel, which has been widely discussed and solved by many works since 2019.\n\n2. The performance improvements are marginal. On NAS-Bench-201, the performance of the proposed method is the same as previous work $\\beta$-DARTS. In Table 2, the improvements on CIFAR-10 and CIFAR-100 are marginal, the results on ImageNet are not state-of-the-art. \n\nQuestions:\n1. Can the authors provide the search cost of the proposed method? Besides, extensions of the method to other DARTS variants (e.g., some variants that can directly search on large-scale datasets) are preferred to better validate the method.\n\n2. In Figure 4 (c), why the performance drops after 100 epochs? Does the proposed method just delay the performance collapse process?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "PMo93vYKyqa", "reviewer": "Reviewer_iL42", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a differentiable NAS approach based on DARTS. The main discovery is that \"DARTS suffers from a specific structural flaw\" so that it \"gives an unfair advantage to layers closer to the output\". To solve this issue, a new regularization method is proposed by \"harmonizing operation selection via aligning gradients of layers\". Experiments look good on a few benchmarks.", "summary_of_the_review": "I think the efforts in solving the collapse problem are good and the explanation is interesting, but the empirical results are insufficient to fully justify the statements.", "strength_and_weaknesses": "Strengths\n1. The paper is well written.\n2. The studied problem (why DARTS often collapses) is important for the NAS community.\n3. The provided explanation is somewhat new.\n\nWeaknesses\n1. One of the most significant issues for all recent research in DARTS (not only for this paper) is the limited ability to justify the statements by experiments. Let me explain. The search space of DARTS is relatively small and well studied. The community knows a few tricks (e.g. increasing the depth, or constraining the number of skip-connections) that can easily generate strong architectures in the space. On the opposite, the gap among recognition accuracy of different methods is very small, e.g. 0.1% on CIFAR10 and 0.2-0.3% on ImageNet-1K. This makes it quite difficult to judge whether the improvement indeed comes from the proposed method (or simply an implementation trick).\n\nGoing deeper, we know that DARTS underwent a few approximations, e.g. the 1st-order method discards higher-order gradients, and the computation of the inverse Hessian matrix is **largely** approximated -- see the analysis in [Zela et al., ICLR20] and [Bi et al., arXiv19]. The question is that, the proposed method used another strong approximation to alleviate the issue caused by an existing approximation. Integrating it with the above concerns, I cannot believe that the marginal improvement (the authors may argue that the improvement on CIFAR100 is relatively large, but this is because the CIFAR100 is not well tuned by other methods) indeed comes from the proposed method.\n\nTo further validate the effectiveness, the authors may try to (i) use a more complex search space (e.g. the spaces constructed in [Bi et al., arXiv19] -- although the authors investigated the spaces offered by [Zela et al., ICLR20], but these spaces are actually even simpler than the original DARTS space) so that the design tricks have weaker impacts, or (ii) report search results with more epochs (e.g. 200 or more), since the collapse issue happens more frequently.\n\nIt is also good to discuss the relationship to other works that discussed the stability of DARTS, e.g. DARTS+ [Liang et al., arXiv], XNAS [Nayman et al., NeurIPS19], Fair DARTS [Chu et al., ECCV20], GOLD-NAS [Bi et al., arXiv20].", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Oi5JxECTCE", "reviewer": "Reviewer_hXod", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper finds the main reason behind performance collapse in DARTS. A new algorithm named  lambda-DARTS is proposed, which is able to generate better performance compared with other methods. Specifically, two new regularization terms are proposed to prevent performance collapse by harmonizing operation selection via aligning gradients of layers. The lambda denotes the dubbed layer alignment, which is a measure to show the correlation between the gradient of each layer corresponding to the architecture parameters.", "summary_of_the_review": "Overall, I think it is a good paper with a good starting point and clear writing. I mentioned 2 doubts in the weakness part.\n\nAlso, there are many existing works on the performance collapse of DARTS. For example, the MS-DARTS which leverages the mean shift to prevent performance collapse at the final discretization step of DARTS.", "strength_and_weaknesses": "Strength: \n1. It has a good starting point and the results of proposed methods can solve the mentioned issues to some extent. \n\n2. The writing is good and easy to understand. A good analysis of lambda w.r.t. the DARTS performance collapse is provided. Low value for lambda means the optimal architecture corresponding to each layer varies wildly. This work seems to be the first to investigate DARTS from the weight-sharing framework and convergence conditions.\n\nWeakness: \n\nFor DARTS tasks, the search cost (GPU days) is a crucial evaluation. But the authors did not mention it in the experimental results.\n\nAlthough the test accuracy is better than other methods, the number of parameters seems to be larger than most of the previous methods. Since the parameters are one of the factors in Table 2, I wonder how to explain this factor.\n\nTables 1 and 2 reports test accuracy, while Table 3 reports test error rate. I wonder if there is any cherry pick?\n\nAlso the performance of ImageNet is not reported in these tables. Is is known that NAS runs slow and how best to scale NAS to run on large datasets such as ImageNet is crucial for real-world applications. And as I know many DARTS methods can only search on toy datasets such as CIFAR-10. Can you report if your NAS method can be trained on large-scale dataset?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster