SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Shen Jiang*
Zipeng Ji*
Guanghui Zhu†
Chunfeng Yuan
Yihua Huang
State Key Laboratory for Novel Software Technology, Nanjing University {jiangshen, jizipeng}@smail.nju.edu.cn, {zgh, cfyuan, yhuang}@nju.edu.cn
Differentiable NAS (DARTS) is a simple and efficient neural architecture search method that has been extensively adopted in various machine learning tasks. Nevertheless, DARTS still encounters several robustness issues, mainly the domination of skip connections. The resulting architectures are full of parametric-free operations, leading to performance collapse. Existing methods suggest that the skip connection has additional advantages in optimization compared to other parametric operations and propose to alleviate the domination of skip connections by eliminating these additional advantages. In this paper, we analyze this issue from a simple and straightforward perspective and propose that the domination of skip connections results from parametric operations overfitting the training data while architecture parameters are trained on the validation data, leading to undesired behaviors. Based on this observation, we propose the operation-level early stopping (OLES) method to overcome this issue and robustify DARTS without introducing any computation overhead. Extensive experimental results can verify our hypothesis and the effectiveness of OLES.
Neural architecture search (NAS) aims to automatically design high-performance neural architectures Zoph and Le [2016] for given tasks in a data-driven manner. Deep learning models designed by NAS have achieved state-of-the-art performance in a variety of tasks, such as image classification Termritthikun et al. [2021], Real et al. [2018], Pham et al. [2018], semantic segmentation Liu et al. [2019], graph learning Zhu et al. [2022a, 2023], and recommender systems Chen et al. [2022], Zhu et al. [2022b]. Among various NAS methods, differentiable architecture search (DARTS) Liu et al. [2018] significantly improves the efficiency of NAS through parameter-sharing and continuous relaxation. DARTS converts discrete architecture selections into continuous architecture parameters  $\alpha$  and optimizes them via gradient-based optimizers.
Despite its simplicity and efficiency, DARTS still has several issues that make it difficult to outperform other NAS methods, even a simple random search Li and Talwalkar [2020], Yu et al. [2019], in many cases. The first issue is the performance gap. Due to the discrepancies between discrete architectures and continuous architecture parameters, the derived model shows non-negligible performance degradation Liu et al. [2018]. Additionally, DARTS suffers from serious robustness problems Nayman et al. [2019], Chen et al. [2019], Liang et al. [2019], Chu et al. [2020], Zela et al. [2019], Wang et al.
[2021]. In particular, during the search phase of DARTS, as the number of training epochs increases, the searched architectures will fill with parametric-free operations Chu et al. [2019a], such as skip connections or even random noise, leading to performance collapse. These issues waste a lot of computational resources and seriously hinder the application of DARTS.
The domination of skip connections is the major robustness issue of DARTS Wang et al. [2021]. Many studies have investigated why the domination of skip connections happens and how to alleviate this issue. Most of them believe that skip connections exhibit unfair advantages in gradient-based optimization during the search phase Liang et al. [2019], Chu and Zhang [2020], Chu et al. [2019a] or stabilize the supernet training by constructing residual blocks Chu et al. [2020]. To address this problem, they propose eliminating these unfair advantages Chu et al. [2019a] or separating the stabilizing role Chu et al. [2020], mainly drawing inspiration from ResNet He et al. [2015]. However, the generic optimization in ResNet differs fundamentally from the bi-level optimization in DARTS. Therefore, observations and theories based on uni-level optimization are insufficient to support the claim that skip connections still have unfair advantages in DARTS. The work Wang et al. [2021] analyzes this problem from the perspective of architecture selection and proposes that the magnitude of the architecture parameter may not reflect the strength of the operation. Based on this assumption, a perturbation-based architecture selection method is proposed. However, this idea conflicts with the fundamental motivation of DARTS, which utilizes continuous relaxation to solve the combinatorial optimization problem approximately. Xie et al. [2021] suggests that overfitting in bi-level optimization is the primary reason for the catastrophic failure of DARTS and alyzes it from a unified theoretical framework, which is similar to our conjecture.
In contrast to the above sophisticated assumptions, in this paper, we take a simpler and more straightforward perspective to consider the reasons for the domination of skip connections. We observe that the robustness problem of DARTS is caused by overfitting. Specifically, the parameters of operations (e.g., convolution) in the supernet overfit the training data and gradually deviate from the validation data. Meanwhile, the architecture parameters are trained on the validation data, causing the advantages of parametric operations to weaken progressively and ultimately leading to the domination of parametric-free operations. It is evident that the supernet is always overparameterized. Additionally, the architecture parameters also affect the supernet training, with operations that exhibit advantages in the early stage of search being more prone to overfitting.
Based on the above observations, we further propose an operation-level early stopping (OLES) method to address the domination of skip connections. The OLES method monitors each operation in the supernet to determine if it tends to overfit the training data and stops training the operation when overfitting occurs. Specifically, we utilize the gradient matching (GM) method Hu et al. [2022] to compare the gradient directions of operation parameters on the training data and the validation data. If the difference between them remains large in multiple consecutive iterations, the operation will stop training. This approach not only alleviates the domination of skip connections and but also sharpens the distribution of architecture parameters with more search iterations, thereby reducing the performance gap caused by discretization. Moreover, our method only needs to maintain the directions of operation gradients. Due to only minimal modifications to the original DARTS, the additional overhead is negligible.
- We analyze the robustness issue of DARTS from a new perspective, focusing on the overfitting of operations in the supernet.
- We propose the operation-level early stopping method to address the domination of skip connections in DARTS with negligible computation overheads.
- Extensive experiments demonstrate the effectiveness and efficiency of our method, which can achieve state-of-the-art (SOTA) performance on multiple commonly used image classification datasets.
<sup>1</sup>Open-source code can be found at https://github.com/PasaLab/oles.
2.1 Neural Architecture Search
Over the years, a variety of NAS methods have been proposed, including evolutionary-based Termitrithikun et al. [2021], Real et al. [2018], Guo et al. [2020], reinforcement-learning-based Zoph and Le [2016], Pham et al. [2018], and gradient-based Liu et al. [2018], Xie et al. [2018], Cai et al. [2018] methods. They have achieved great success in many computer vision (CV) and natural language processing (NLP) So et al. [2019] tasks. As a pioneering work, NAS-RL Zoph and Le [2016] uses an RNN controller to generate architectures and employs the policy gradient algorithm to train the controller. However, this method suffers from low search efficiency. To improve the search efficiency, the cell-based micro search space Zoph et al. [2017] and weight-sharing mechanism Pham et al. [2018] have been proposed. DARTS Liu et al. [2018], which also takes advantage of weight-sharing, utilizes the gradient-based method to optimize architecture parameters and model weights alternatively. Recently, train-free NAS methods Chen et al. [2021a], Li et al. [2023] that utilize proxy metrics to predict the ranking or test performance of architectures without training have been proposed to further promote search efficiency.
2.2 Preliminaries of Differentiable Neural Architecture Search
We will now provide a detailed review of DARTS. The original DARTS Liu et al. [2018] relies on a cell-based micro search space, where each cell contains  $N$  nodes and  $E$  edges organized in a directed acyclic graph (DAG). Each node represents a feature map  $x^{(i)}$ , and each edge is associated with an operation  $o \in \mathcal{O}$ , where  $\mathcal{O}$  denotes the set of candidate operations (e.g., skip connect, sep conv 3x3, etc.). During architecture search, instead of applying a single operation to a specific node, continuous relaxation is applied to relax the categorical choice of a specific operation to a mixture of candidate operations, i.e.,  $\bar{o}^{(i,j)}(x) = \sum_{o \in \mathcal{O}} \frac{\exp(\alpha_{o}^{(i,j)})}{\sum_{o' \in \mathcal{O}} \exp(\alpha_{o'}^{(i,j)})} o(x)$ .  $\alpha = \{\alpha^{(i,j)}\}$  serves as the architecture parameters. DARTS then jointly optimizes architecture parameters  $\alpha$  and model weights  $w$  with the following bi-level objective via alternative gradient descent:
$$
\min _ {\alpha} \mathcal {L} _ {v a l} (w ^ {*}, \alpha)
$$
$$
\text {s . t .} \quad w ^ {*} = \underset {w} {\arg \min } \mathcal {L} _ {t r a i n} (w, \alpha).
$$
Since DARTS follows the weight-sharing mechanism, we refer to the continuous relaxed network as the "supernet". At the end of the search phase, a discretization phase is performed by selecting operations with the largest  $\alpha$  values to form the final architecture.
2.3 Robustifying DARTS
Despite its simplicity and efficiency, the undesired domination of skip connections is often observed in DARTS. Previous works have attempted to eliminate the domination of skip connections by diminishing their unfair advantages. DARTS+Liang et al. [2019] uses a fixed value to constrain the number of skip connections. Progressive DARTSChen et al. [2019] adds a Dropout after each skip connection operation. DARTS—Chu et al. [2020] introduces an auxiliary skip connection to separate the unfair advantage of the skip connection by analyzing it from the perspective of ResNet He et al. [2015]. Another strategy is to remove the exclusive competition of operation selection in DARTS. FairDARTSChu et al. [2019a] replaces the Softmax relaxation of DARTS with Sigmoid and offers an independent architecture parameter for each operation. NoisyDARTSChu and Zhang [2020] inject unbiased noise into the candidate operations to ensure that the good ones win robustly. However, DARTS+PT Wang et al. [2021] believes that the magnitude of the architecture parameter is not a good indicator of operation strength and proposes a perturbation-based architecture selection method.
Our work is also related to GM-NAS Hu et al. [2022], which falls in the field of Few-Shot NAS Zhao et al. [2021]. Although our method and GM-NAS both employ the GM score as a mathematical tool, they are entirely distinct lines of work. GM-NAS argues that due to coupled optimization between child architectures caused by weight-sharing, One-Shot supernet's performance estimation could be inaccurate, leading to degraded search results. As a result, GM-NAS proposes to reduce the level of weight-sharing by splitting the One-Shot supernet into multiple separated sub-supernets.
Figure 1: The motivational experiment to verify the domination of skip connections is due to the overfitting of operations.
GM-NAS utilizes GM scores to make splitting decisions, determining whether a module should be shared among child architectures. The GM scores in GM-NAS are computed based on the gradient information of different child architectures on shared parameters. In contrast, we aim to address the problem of skip connection donimation from a totally new perspective and employ the GM score as an indicator for early stopping, preventing operation parameter overfitting. The GM scores, in our approach, are calculated using the gradient information of parameters on training and validation data.
3 Motivation and Methodology
In this section, we argue that the domination of skip connections is due to the overfitting of operations in the supernet and demonstrate this hypothesis through a simple motivational experiment. Based on our observations, we propose an operation-level early stopping method to address this issue.
3.1 The Domination of Skip-Connection is Due to Overfitting
In DARTS, we observe that skip connections dominate due to the overfitting of operations in the supernet to the training data, causing them to increasingly deviate from the distribution of the validation data. However, the architecture parameters are trained on the validation data, thus operations are selected based on their validation performance. As the number of training iterations increases, the parametric operations overfit more seriously, and the parametric-free operations gradually dominate.
To verify the above conjecture, we present empirical evidence on the relationship between the overfitting of operations in the supernet and architecture parameters. We retain only architectural parameters on one edge in DARTS' search space and set all other architectural parameters to 0 to ensure that they will not be trained. In Figure 1, we record the architecture parameter and the validation loss with the corresponding output of each operation in this edge. From Figure 1a, we can observe that the validation losses of all parametric operations grow even faster as the training steps increase, and the architecture parameter gradually decreases, exhibiting an obvious negative correlation with the validation loss. Moreover, Figure 1b shows that the skip connection easily obtains higher architecture parameters, exhibiting advantages over other parametric-free operations. This is mainly due to the fact that the skip connection can always retain more information from the input.
Most existing works suggest that the domination of skip connections is due to their unfair advantage. To investigate this claim, we follow the settings of FairDARTS Chu et al. [2019a] and replace the Softmax relaxation in DARTS with the Sigmoid relaxation. The unfair advantage of skip connections will be eliminated by removing the exclusive competition among operations. We then rerun the experiment and present the results in Figure 2, which shows that even after removing the exclusive competition, the architecture parameters still exhibit a negative correlation with the validation losses, similar to that in Figure 1a. It illustrates that the unfair advantage of skip connections is not the fundamental reason for their domination. Instead, it is the effect of the overfitting of operations in the supernet on the training of architecture parameters.
3.2 Operation-Level Early Stopping
Based on the above observations, we propose an operation-level early stopping method to alleviate the domination of skip connections and enhance the robustness of DARTS. During the search phase, the architecture parameters have an effect on the supernet training, resulting in different operations with different architecture parameters being trained differently. Operations that exhibit advantages in the early stage will get more attention and are more prone to overfitting. Moreover, the operation is the basic unit of
choice in DARTS, which selects operations to form the final architecture. Therefore, we perform early stopping for each operation. If early stopping can be performed for each operation during the supernet training, when the operation tends to overfit, the overfitting problem can have less impact on the training of architecture parameters, and the supernet can also be fully trained.
Figure 2: The architecture parameters and validation losses of different operations in FairDARTS.
Overfitting occurs when a model is too complex and fits the training data distribution too closely, resulting in poor performance on the validating data. When a model is overfitting, the training loss continues to decrease while the validation loss increases, indicating that the optimization direction (i.e., the direction of gradients) of the model parameters on the training data will be inconsistent with that on the validation data. To identify overfitting, we propose using the gradient direction of the operation parameters. If the gradients on the training and validation data differ significantly in direction, we consider the operation to tend to overfit the training data.
Algorithm 1 DARTS with operation-level early stopping
Require: Supernet weights  $w$ ; architecture parameters  $\alpha$ ; number of search iterations  $I$ ; number of consecutive iterations for GM  $M$ ; threshold of GM score for verifying overfitting  $\xi$ .
Ensure: Searched architectures.
1: Construct a supernet  $A$ ;
2: for each  $i\in [1,I]$  do
3: Get a batch of training data  $B_{train}$ ;
4: Update weights  $w$  by  $\nabla_w \mathcal{L}_{train}(w, \alpha, B_{train})$ ;
5: Record the training gradient of each operation  $g_{train}^{o}$ ;
6: Get a batch of validation data  $B_{val}$
7: Update parameters  $\alpha$  by  $\nabla_{\alpha}\mathcal{L}_{val}(w,\alpha ,B_{val})$
8: Record the validation gradient of each operation  $g_{val}^{o}$ ;
9: Compute the gradient matching score over consecutive  $M$  iterations based on Eq. 1.
0: Stop the gradient of an operation  $o$  if  $GM(g_{train}^{o}, g_{val}^{o}) < \xi$
11: end for
12: Derive the final architecture based on learned  $\alpha$ .
Concretely, for an operation  $o$ , let  $P_{o}$  denote the parameters of  $o$ , and  $g_{train}^{o}$  and  $g_{val}^{o}$  denote the gradients of  $P_{o}$  on a training batch and a validation batch, respectively. We calculate the cosine similarity between  $g_{train}^{o}$  and  $g_{val}^{o}$  to quantify the differences in the direction of the operations' gradients as follows:
$$
G M (g _ {t r a i n} ^ {o}, g _ {v a l} ^ {o}) = \frac {\sum_ {m = 1} ^ {M} \mathrm {C O S} (g _ {t r a i n} ^ {o} , g _ {v a l} ^ {o})}{M},
$$
$$
\text {w h e r e} \quad \operatorname {C O S} \left(g _ {\text {t r a i n}} ^ {o}, g _ {\text {v a l}} ^ {o}\right) = \frac {g _ {\text {t r a i n}} ^ {o} \cdot g _ {\text {v a l}} ^ {o}}{\left| g _ {\text {t r a i n}} ^ {o} \right| \left| g _ {\text {v a l}} ^ {o} \right|}, \tag {1}
$$
$$
g _ {t r a i n} ^ {o} = \frac {\partial \mathcal {L} _ {t r a i n} (w , \alpha , B _ {t r a i n})}{\partial P _ {o}}, g _ {v a l} ^ {o} = \frac {\partial \mathcal {L} _ {v a l} (w , \alpha , B _ {v a l})}{\partial P _ {o}},
$$
where  $B_{train}$  and  $B_{val}$  denote batches of training data and validation data, respectively. The GM score is computed as the average cosine similarity between training gradients and validation gradients
over  $M$  consecutive steps. The gradient matching score here is similar to that in GM-NAS Hu et al. [2022]. When the GM score falls below a predefined threshold of  $\xi \in (0,1)$  within  $M$  steps, the operation is considered to be overfitting. Consequently, if  $GM < \xi$  for an operation  $o$ , the parameters of this operation will not be updated during subsequent supernet training. And the remaining operations and architecture parameters will be trained as usual.
Through the operation-level early stopping method, the overfitting of operations in the supernet can be alleviated. Benefiting from the attribute that the DARTS search process involves alternating between training the supernet on the training data and training architecture parameters on the validation data, we only need to maintain the gradient of each operation on a training data batch and a validation data batch in one iteration, with negligible additional computational overhead. As a result, OLES can alleviate the domination of skip connections, making DARTS more robust. This allows for training DARTS with more iterations to obtain better architectures and sharper architecture parameters, resulting in a smaller discretization gap and improving performance. The algorithm for DARTS with operation-level early stopping can be found in Algorithm 1.
4.1 Search Spaces and Experimental Settings
To verify the effectiveness of OLES, we conduct experiments on three different search spaces: the standard DARTS' search space, NAS-Bench-201, and MobileNet-like search space. We keep all experimental settings the same as the original in each search space and only use the first-order optimization, as our method only needs to freeze the operation parameters during the supernet training. In addition to the standard hyperparameters in DARTS, our method introduces an additional hyperparameter, i.e., the overfitting threshold  $\xi$ , to stop operation training. We determine the threshold  $\xi$  by averaging the cosine similarity over 20 iterations for 30 randomly initiated architectures in each search space. And the gradient matching (GM) score is dynamically computed by averaging over every 20 iterations throughout the entire training process to reduce variance. Following Algorithm 1, we stop updating parameters of the corresponding operation  $o$  when  $GM(g_{train}^{o}, g_{val}^{o})$  is lower than the threshold  $\xi$ .
4.2 Performance on DARTS' CNN Search Space
4.2.1 CIFAR-10
We conduct NAS on CIFAR-10 using DARTS' search space and then transfer the searched architectures to CIFAR-100. The threshold  $\xi$  is set to 0.3. As shown in Table 1, our method OLES consistently outperforms other recent SOTA NAS methods. OLES achieves a top-1 test error of  $2.30\%$  on CIFAR-10 with nearly the same search cost as DARTS of 0.4 GPU-days. Moreover, the robustness of OLES is also guaranteed, as evidenced by the excellent average result of three independent runs. When the architectures searched on CIFAR-10 are transferred to CIFAR-100, our method still achieves a competitive test error of  $16.30\%$ . This indicates that the architectures searched by OLES are transferable and expressive, and the architecture parameters do not overfit the CIFAR-10 dataset.
4.2.2 CIFAR-100
We conduct NAS on CIFAR-100, and the results are presented in Table 2. The threshold  $\xi$  is set to 0.4. OLES still exhibits significant advantages, improving the test error of DARTS from  $20.58\%$  to  $17.30\%$  without incurring extra search costs. It demonstrates that our method can adapt to datasets of different sizes and consistently improve the performance of DARTS. OLES not only consistently outperforms other methods that focus on the skip-connection domination issue, but also surpasses more sophisticated NAS methods (e.g., GM+DARTS Hu et al. [2022]) on CIFAR-10 and CIFAR-100 without requiring extensive code modifications or introducing additional computational overheads.
4.2.3 Transfer to ImageNet
To verify the transferability of architectures discovered by OLES, we transfer the best architecture derived from CIFAR-10 to ImageNet. Following the setting of DARTS, we train the searched