Improving Differentiable Neural Architecture Search by Encouraging Transferability
Authors: Parth Sheth
Pengtao Xie
Venue: ICLR 2023 poster (2023)

Abstract: Differentiable neural architecture search  methods are increasingly  popular due to their computational efficiency. However, these methods have unsatisfactory generalizability and stability. Their searched    architectures  are often  degenerate with a dominant number of skip connections and perform unsatisfactorily on test data. Existing methods for solving this problem   have  a variety of limitations, such as cannot prevent the happening of architecture degeneration, being excessively restrictive in setting the number of skip connections,  etc. To address these limitations, we propose a new approach for improving the generalizability and stability of differentiable NAS, by  developing a transferability-encouraging tri-level optimization  framework  which improves the  architecture  of a main model  by encouraging  good transferability to an auxiliary model. Our framework  involves three stages performed end-to-end: 1) train network weights of  a main model; 2) transfer knowledge from the  main  model  to an auxiliary model; 3) optimize the architecture of the main model by maximizing its  transferability to the  auxiliary model. We propose a new knowledge transfer approach based on matching quadruple relative similarities.  Experiments on several datasets demonstrate the effectiveness of our method.

Published as a conference paper at ICLR 2023
IMPROVING DIFFERENTIABLE NEURAL ARCHITEC- TURE SEARCH BY ENCOURAGING TRANSFERABILITY
Parth Sheth"
Pengtao Xie University of California San Diego pl plxiel plxieθeng. www.eeng.usdd..em uusd.edu
University of Pennsylvania pa parthfour@gmai com
ABSTRACT
Differentiable neural architecture search methods are increasingly popular due to their computational efficiency. However, these methods have unsatisfactory generalizability and stability. Their searched architectures are often degenerate with dominant number of skip connections and perform unsatisfactorily on test data. Existing methods for solving this problem have variety of limitations. such as cannot prevent the happening of architecture degeneration, being exces- sively restrictive in setting the number of skip connections, etc. To address these limitations, we propose new approach for improving the generalizability and sta- tri-level
bility of differentiable NAS, by developing a waanferability-encourraa optimization framework which improves the architecture of main model by en- couraging good transferability to an auxiliary model. Our framework involves three stages performed end-to-end: 1) train network weights of main model; 2) transfer knowledge from the main model to an auxiliary model; 3) optimize the architecture of the main model by maximizing its transferability the auxil- iary model. We propose new knowledge transfer approach based on matching quadruple relative similarities. Experiments on several datasets demonstrate the effectiveness of our method.
INTRODUCTION
Neural architecture search (NAS) (Zoph & Le 2017 Liu et al., 2018b; Cai al. 2019 Liu al.
2019a Pham et al.) 2018 Real et al. 2019). which aims to search for highly-performan performant neural architectures automatically, finds broad applications. Among various NAS methods, differentiable search methods (Liu et al. 2018b
| Cai et al. 2019 Chen et al. 2019 Xu et al. (020) gain increasing popularity due to their computational efficiency. In differentiable NAS, architectures are represented as differentiable variables and are learned using gradient descent.
While differentiable NAS computationally efficient, its generalizability and stability has been challenged in many works (Zela et al., [019 Chu et al. [2020a; 2019; Zhhou et al.) 2220aa] Chen & Hsieh 2020): the searched architecture is degenerate with a dominant number of skip connections: while having good performance
test data. For example, Zela et al. (2019) identified NAS benchmarks based on four search spaces where archi- tectures searched by standard DARTS (Liu al. 2019a) differentiable NAS method) have poor performance on test data of CIFAR-10, 10, CIFAR- 100, and SVH. A variety of approaches (Zela et al.,
validation data, it performs unsatisfactorily
Zhou al.) 2020a Chen & Hsieh 2020) Chen al. 2019 Liang et al. 2020 Wang et al. 2021 have been proposed to improve the generalizability and stability of differentiable NAS methods. These methods have various limitations, such cannot improve
2019 Chu al., 2020a 2019
search algorithms to prevent degenerate architectures from occurring (Zela et al., 2019). cannot ex- plicitly maximize the generalization performance of architectures (Chu et al. [202aa] 2019), cannot broadly explore search spaces (Zhou et al.) 2020a Chen & Hsieh, 2020), requiring extensive tuning of the number of skip connections (Chen et al.) 2019 Liang et al. 2020), etc. As result, their effectiveness in improving differentiable NAS is less satisfactory.
To address these limitations, we propose a new approach for improving the generalizability and sta- bility of differentiable NAS methods. Specifically, we develop transferability-cccourag tri-level
"The work was done while visiting UCSD. Corresponning author

---

Published as a conference paper at ICLR 2023
optimization (TETLO) framework, which improves the architecture of main model by encouraging effective knowledge transfer to an auxiliary model. Intuitively, to train a better auxiliary model, the main model needs to generate accurate knowledge (which is used train the auxiliary); to generate accurate knowledge, the main model needs to improve its architecture (which is used to generate knowledge). Combining these two steps together, we conjecture that improving the auxiliary drives the main model to improve its architecture (empirical evidence is provided in Figure 2). Our method also motivated by the theoretical analysis in (Liu et al. 2019b) that good transferability improves generalization performance.
In our framework, main model and an auxiliary model have architectures and network weights to learn. Learning consists of three stages, In the first stage, we train the network weights of main model while temporarily fixing its architecture. trained in the first stage to help train an auxiliary model via transfer learning. To capture high-order data relationships, we propose a new knowledge transfer approach based on matching quadruple relative similarities (QRS), where the main model generates QRS relationships, e.g., the similarity between data example x and ys larger than that between w and z. Then the auxiliary model is trained by fitting these QRS relationships. In the third stage, model as a measure of transferability from the main model to the auxiliary model and update the architecture of the main model by maximizing the transferability. The three stages are performed end-to-end in a three-level optimization framework.
the second stage, we leverage the main model
validation performance of the auxiliary
The major contributions of this work include:
tri-level optimization (TETLO) framework improve the generalizability and stability of differentiable NAS methods.
We propose
We propose a new knowledge transfer approach based on matching quadruple relative similarities.
We perform various experiments which demonstrate the effectiveness of our method.
2 RELATED WORKS
2.1
NEURAL ARCHITECTURE SEARCH
The goal of neural architecture search (NAS) is to automatically identify highly-performing neural architectures that can potentially surpass human-designed ones. The research of NAS has made considerable progress in the past few years. Early NAS approaches (Zoph & Le 2017: Pham al.) 2018 Zoph et al. 2018) are based on reinforcement learning (RL), where a policy network learns to generate high-quality architectures by maximizing validation accuracy (used as reward). These approaches are conceptually simple and can flexibly perform search in any search space. In differen- tiable search methods (Cai et al. 2019 Liu et al.. [019a] Xie et al. 2019), each candidate architec- ture is a combination of many building blocks. Combination coefficients represent the importance of building blocks. Architecture search amounts to learning these differentiable coefficients, which can be conducted using differentiable optimization algorithms such as gradient descent. Another paradigm of NAS methods (Liu et al., 2018b; Real al., 2019) are based on evolutionary algo- rithms (EA). In these approaches, architectures are considered as individuals in population. Each architecture is associated with a fitness score representing how good this architecture is. Architec- tures with higher fitness scores have higher odds of generating offspring (new architectures), which replace old architectures that have low-fitness scores. Our proposed method can be applied to any differentiable NAS methods. Recently, meta-NAS methods (Elsken al., 2020 Lian et al., 2019) have been proposed for fast adaptation of neural architectures. Our method differs from meta-NAS approaches in the following aspects. First, method aims at improving architecture by encour- aging it to have good transferability to an auxiliary model, while meta-NAS focuses on adapting a meta architecture to different task-specific architectures. Second, method transfers knowledge from a main architecture to an auxiliary model using our newly proposed quadruple relative similar- ity matching mechanism while meta-NAS adapts a meta architecture task-specific architectures via gradient descent update. Third, our method searches for architectures on a single dataset while meta-NAS operates on a collection of meta- veta-training tasks (each with a training and validation set).
2.2 IMPROVE THE GENERALIZABILITY AND STABILITY OF DIFFERENTIABLE NAS
Various methods have been proposed for improving the generalizability and stability of differen- tiable NAS methods. Zela et al. (2019) proposed early stopping approach based on eigenvalues of validation loss' Hessian matrix. This method early stops search algorithms when degenerate ar- chitectures occur, but can not improve search algorithms to prevent degenerate architectures from oc-
2

---

Published as a conference paper at ICLR 2023
curring. Chu et al. (2020a) leverage auxiliary skip connections to encourage fairer competition for all operations. CChu et al. (2019) encourage each operation ' architectural weight to be independent of others. These methods focus reducing the number of skip connections, but lack mechanism of encouraging searched architectures to learn representations that can be well transferred to other models. In contrast, our method explicitly improves the transferability of representations learned by searched architectures via minimizing the validation loss of an auxiliary model trained on transferred representations. [Zhou et al. (2020a) proposed a path-depth-wise regularization approach. Chen & Hsieh 2020) developed perturbation based regularization method. In these regularization-bare based methods, it difficult to choose proper level of regularization strength. As theoretically analyzed in (Xie et al., 2017), an excessively strong regularization will lead to underfitting while an insuf- ficient regularization will lead to overfitting. Finding the sweeping point between underfitting and overfitting needs extensive tuning, which is time-consuning and computationally expensive. Our method does not have this limitation since it is not regularization- based and therefore does not need to find the optimal level of regularization strength. Some methods in (Chen et al.] 2019; Liang et al. 2020) constrain the number of skip connections to a fixed number (typically 2). which might be too restrictive and consequently hurts performance. In contrast, constraints on the number of skip connections. [Wan et al. (2021) identified that magnitude-based architecture selection approaches hurt generalization and proposed a perturbation-
method does not have hard
architecture selection method to address this problem. Our proposed method, which focuses on training better weights of the supernet, orthogonal (Wang et (02I) and can be applied together with (Wang et al. 202T) to improve the generalizability and stability of differentiable NAS.
2.3
TRANSFER LEARNING
Existing transfer learning (TL.) methods (Pratt 1993 Mihalkova et al. 2007, Viculescu-Mizii &
Caruana 2007 Pan & Yang 2009 Luo et al. 2017 1 Zhuangg et al. (2020) aim to improve performance of a target task by transferring knowledge from a source task. Various transfer strategies have been
proposed, based on reweighting source data to better match target distribution (Huang et al., 2006
Jiang & Zhai, |007 Sugiyama et al.] [007]] Foster et al.) 2010; Moore & Lewis, 2010; Axelrod ctt al., [011] Wang et al. 2017b] Nggiam et al. 2018), transforming source and target data into a
common feature space (Borgwardt et al. |006 Pan et al. 2008: 2010 Duan et al., 2012 Long
cet al. 2013: Wang et al. (2017a). extracting comainn-nvvarian
representations based on adversarial learning (Ganin et al. 2016 Long et al.] 2017; Tzeng et al. 2017; Hoffman et al.] 2018 Shen
ctt al. 2018 Zhang et al. 20019), regularizing target models using source models (Luo et al.) 2008
Duan et al.
2009 Zhuang et al., 2009 Tommasi al., 2010; Duan et al. 2012), etc. Existing
methods improve a target model by absorbing knowledge from a source model. In our work, we
take an opposite direction, which improves main model by encouraging good transferability to an auxiliary model.
Train main encoder and main head
Main encoder generates QRS relations
Train auxiliary encoder by fitting QRS relations and train auxiliary head
3 METHODS
In this section, we present the transferabllty-eccouragggg tri-level optimization framework (Figure). Given a main task m (c.g., image classification (He et al., 621) with a main dataset Dm, we aim learn a nigh-performance main model to conduct this task. The main model has learnable architecture M, a data encoder Em. and a task-specific head Hm. To prevent M from being degenerate, we encourage it to have good transferability to an auxiliary model. Via transfer learning, we leverage the main model to train an auxiliary model in an auxiliary task with an auxiliary dataset D.. The auxiliary model has learnable architecture A, encoder Eg. and task-specific head Hg. We use the validation performance of the auxiliary model to measure main-to-au transferability. Since the auxiliary model is trained with the help of the main model, good validation performance of the auxiliary model indicates strong generalization power of the main model. We optimize the architecture of the main model by maximizing the auxiliary's validation performance.
Update architectures
Validation main model and auxiliary model
Figure 1: Overview of the TETLO framework.
3.1 A THREE- OPTIMIZATION BASED FRAMEWORK
Learning consists of three stages which are performed end-to-end. In the first stage, the main model trains its network weights with its architecture temporarily fixed. In the second stage, the auxiliary model is trained with the help of the main model via transfer learning. In the third stage, we measure
3

---

Published as a conference paper at ICLR 2023
main-to-auxiliary transferability, which is the validation performance of the auxiliary model, and update the architecture of the main model by maximizing transferabilityy
Stage I. In this stage, perform train the main encoder Em and main head Hm on D with the
architecture M tentatively fixed, where D is the training set of Dm- Let Lo denote a training loss (e.g., cross
cross entropy for classification). This stage amounts to solving the following problem: (I. = argming..... (M( Em, Hm. )
(1)
M is needed to calculate L.- However, it should not be updated via minimizing Lm Otherwise, trivial solution will be produced where M can perfeetly overfit D but will yield poor performance on test examples. F. F, (M) and H (M) denote that optimal solutions E and H" are functions of M: different M will result in a different training loss, which renders ES and Hz to change accordingly.
Stage II. In the second stage. we use transfer learning to train the auxiliary model with the help of the main model. Previous knowledge transfer approaches (Romero et al.) 2014: You et al.) 2017, Pham et al. 2000) Chen et al., (2200) are limited capturing low-order (<=3) relationships among data examples during knowledge transfer and fail to explore high-order relationships. To address this problem, we propose a new transfer learning approach, which can explore fourth-order data relationships, based on matching quadruple relative similarity. Given two randomly sampled data
pairs (x,y) and w) from D we use the main model to determine which pair has more similar data examples. For each pair (x,y), the main encoder (including E (M) and M) trained in the first stage is used to encode the two examples in this pair; then cosine similarity is calculated between the two encodings. Given two pairs (x,y) and (z,w), a quadruple relative similarity relation is about whether E M) > c(z, w; Em MM). Given quadruple relative similarity relations generated by the main encoder, and A) to match these relations. If (M) c(x,y; (a.AA; > c(z, A..,,,,, where Ea.A) is the cosine similarity between x and y which are encoded using the auxiliary encoder. Since each quadruple involves four examples, our method is able to capture data relationships with an order of four. In this stage, we solve the following problem:
(M),M)
train the auxiliary encoder (including Eg > c(z, E. (M), then it is required that
EN (M,A), (A) = argmin Ey La(A, Ea, Ho. D S.I. (xx,,,,, [c (M). > c(z, w: (M) J M)] + (c(x, > c(z, w; A)]
Stage III. In the third stage, we measure "transferability" from the main model to the auxiliary model using the validation performance of the auxiliary model. Since the auxiliary model is trained using knowledge generated by the main model, the auxiliary's validation performance reflects the quality of main-to--uuiliar "transferability". A smaller validation loss indicates the knowledge generated by the main model has better quality, which implies that the main model has better gen- eralization ability. We update the architecture of the main model by minimizing the auxiliary's validation loss. In addition, we can also measure the validation loss of the main model and update its architecture by minimizing the main model's validation loss as well. This amounts to solving the following problem:
minMA (M, 7L.
where (val) and ((vad) is the validation set of Dm and D, respectively. is a tradeoff parameter.
Three-level optimization framework. Putting these pieces together, we have the following transferabllty--eccourag tri-level optimization (TETLO) problem:
minM, Lm (M, (M), (M), D(r"))) YLa E (M, s.f.
(M,,), H =A)= argming, UI La Ha, D(")) s.d. V(x,
(3)
c(z. w: (M) (c(x,y: > c(z,w:
(M), H (M) argming,.,H. Lm (M, Em, Hm, D (r))
Continuous relaxation. The constraints in Eq. are not amenable for optimization. We perform a relaxation of them to make optimization easier. To satisfy the constraint (x > c(z, E, (M), - (c(x,y;E,,A)) > Eq. E(,) in
we can encour-
age y; (M.M) c(z, w; Et (M),M) relaxation, the optimization problem at stage II becomes as follows, where 2 is tradeoff parameter:
- wiz, to be positive. Under this
E (M,A), (A) = argming, H, La(A, H,, D D
(4)
2 Σ Eyenn max(O, - (c(x.y. M). c(2, (M)
w. A),))
4

---

Published as a conference paper at ICLR 2023
Table 1: Test errors of CIFAR-10 (C10) and CIFAR-100 (C100) under four search spaces. t: the number of initial channels is 16 and the number of cells is 8. 4: the number of initial channels is 36
and the number of cells is 20. [1] (Liu al. 2019a). [2] (Zela al. 2019). [3] (Zela et al. 2019)
[4] (Chu et al. 2020a [5] (Chen & Hsieh 2020), [6] Maninis et al.
2019
Data
Spuce
DARTS [1]
RDARTS-L2 [2]
DARTS [3]
DARIS [4]
SDARTS-ADE [5]
MIL
Ours
4.69
3.46
3.34
3.26
SI
3.84
2.78
3.01
2.68
2.73
3.07
2.51
C10
S2
5,54
3,31
4.07
4.03
3.11
4.11
2.94
S3
3,92
3,55
2.95
3.07
2.44
$4
8.33
3,56
4.69
4.14
3.49
6.05
3.32
SI
29.46
24.25
28.37
22.41
22.33
27.85
21.63
$2
26.05
23.25
20.56
19.72
C100
S3
28.90
23.99
23,73
21.13
21.08
21.40
20.35
54
22.85
21.94
21.26
21.55
21.25
22.97
20.51
Differentiable parameterization of architecture variables. Following (Liu et al. 2019a),
parameterize architectures in a differentiable way. Each candidate block an architecture is as- sociated with importance weight. Architecture search amounts to learning these weights using gradient methods. After learning, blocks with top weights compose an architecture.
Optimization algorithm. To solve the problem in Eq. (3) (with relaxation), we used a standard algorithm developed in (Liu al., 2019a), which broadly used in many previous works and demonstrated to be effective in the literature. The convergence analysis of this algorithm has been given many works (Ghadimi & Wang 2018, Grazzi et al.) 2020 Ji al. 2202I] Liu al., [021] Yang et al. 2021). The optimization algorithm is not the focus or contribution of our work. Please
Section
H for the algorithm.
Reduce computational and memory costs. To reduce the computational and memory costs, we leverage the following methods. First, we perform architecture tying and weight parameter tying between the main encoder and auxiliary encoder. For both the main and auxiliary model, we set them to be stack of 7 cells, sharing the same architecture as the first 7 cells in DARTS. Parameter tying is performed by making the weight parameters the first six cells the two models be the same. For the seventh cell, the main and auxiliary models have different weight parameters. Second, following (Liu et al. 2019a), we use one-step gradient descent update to approximate optimal solu- tions and finite difference approximation to approximate matrix-vector multiplications. Third, we reduce the frequencies of updating (including calculating hypergradients) the architecture M. updated every 5 mini-batche (i.e., iterations) instead of every mini-batch. We empirically found doing this greatly reduced computational costs without significantly sacrificing accuracy. The rest of parameters
updated every mini-batch. 4 EXPERIMENTS
In this section, we present experimental results. We compare our method with state-of sate--of-theart meth- ods 1) on search spaces designed by (Zela al. 2019) for measuring generalizability and stability of differentiable NAS methods; 2) on CIFAR-T00 CIFAR-10, and ImageNet under search spaces of DARTS, P-DARTS, PC-DARTS, and PR-DARTS. Please refer to the appendix for detailed hyper- parameter settings and additional results.
The main and auxiliary tasks are both image classification, but are conducted using different models. Following (Liu et al. 2019a), each experiment consists of 1) an architecture search phrase which searches for an optimal cell by minimizing the loss on validation set, and 2) an architecture eval- uation phrase which composes multiple copies of the searched cell into larger network and trains the network on the combination training and validation data, and evaluates on a test set.
4.1 DATASETS
Three datasets are used in the experiments, including ImageNet (Deng et al., 2009), CIFAR- 100 (Krizhevsky et al. 2009). and CIFAR-10 (Krizhevsky & Hinton 2010). ImageNet is split into training set and a test set with 1.2M and 50K images respectively. CIFAR-100 is split into training, validation, and test sets, with 25K, 25K, and 10K images respectively. So is CIFAR-10. These three datasets have 1000, 100, and 10 classes respectively.
4.2 EXPERIMENTAL SETTINGS
The head of the main model for CIFAR-10, CIFAR-100, and ImageNet is set to a 10-way, 100-way, and 1000-way linear classifier respectively. For each dataset, its training set is used D(ir) and
5

---

Published as a conference paper at ICLR 2023
Tradeoff parameters y and â are set to 1 and 0.1 respectively. We compare with the following baselines: 1) multi-task learning (MTL) (Maninis et al. 2019): main and auxiliary models trained by minimizing the weighted sum of their losses; 2) encoder sharing (ES) (Kokkinos] 2017): main and auxiliary models share the same encoder; 3)
D its validation set is used Diaall and maa
auxiliarytt--main transfer (ATT) (He al. 6b) first train an auxiliary model on D and (val) then train a main model by transferring knowledge from auxiliary. We report mean and standard deviation of 10 random runs following (Liu al. 2019a).
4.3 EVALUATION OF GENERALIZABILITY AND STABILITY FOUR SEARCH SPACES
We evaluate method on four archi- tecture search spaces designed by (Zela) et al. 2019) for measuring the gen- cralizability and stability of differ- entiable NAS methods. Following RobustDARTS-L2 (Zela et al., 2019). set the number of cells to 8 and the number of initial channels to 16 for CIFAR-10, and set cell number 8 and initial channel number to 16 for CIFAR-100 100. Following DARTS (Liul et al. 2019a) and RobustDARTS (Zela
Table 2: Classification error (%) on CIFAR-100 test set, num- ber of model weights (millions), and search cost (GPU days). TETT.O-darts TETLO is applied to DARTS- Simi- lar meanings hold for other such notations. Results marked with * are obtained from DARTS (Chu et al. 2020a) DARTS (Ye et al. (2022). Methods marked with f were rerun 10 times. Search cost is measured by GPU days on Nvidia 1080Ti.
B-
Method
Emron())
Param
Cost
"ResNet (t
2016e
22.10
1.7
"DenseNat Hunng |
2017
17.18
*PNAS (Lu al. 201
19.53
3.2
150
et al. 2019), the search in each method was run 4 times with different random seeds independently. Each searched ar- chitecture is retrained from scratch for a few epochs. The one (after retrain- ing) yielding the best validation accu- racy is selected. Architecture evaluation is performed ture to generate the final result. Table [ shows the results. Our method achieves lower test errors on CIFAR 100/ 10 under these four spaces than baselines, which demonstrates our method has better gen- cralizability and stability than baselines. The reason is: our method explicitly en- courages the architecture M of main model to have good transferability to an auxiliary model; this way, M is learned be effective not only for the main task, but also for the auxiliary task, which results in better generalizability.
"ENAS Pham et al 2018
19.43
4.6
0.5
*AmoebhNNt Real
2019
18.93
3.1
3150
*GDAS Dong & Yang
2019
18.38
3.4
0.2
*R-DARTS Zela (Zel et
20220
18.01+0.226
"DARTS" (Ciu el
20200
17.51=0.25
3.3
0.4
"DrupNAS [Hong el sl
2020
16.95.10.41
0.7
*DOTS (Gu
16.48.10.13
4.1
*B- 3-DARTS (Te 2l
2022
16.24 +0022
3.8
0.4
"Duri22nd (Liu
(2019
20.5811.44
3.1
4.0
the selected architec-
ATT dariss [Tl el 20166
19.62.10.17
3.2
ES-dart<2nd (Kokkieee)
19.34++0.21
MTL. dTrtsdarts2ni MMamnns cl 2019
18.71.10.26
3.1
4.1
ddd.t-ddddddd (murs)
17.2140.10
*Ptartss Chen al.
2019)
17.4210.14
0.3
ATT-pdarTs (He al
2011h
17.3740.16
3.6
0.5
ES-pdarts (Kokkinos
17.3540.21
MILL-daari (Mauuiis 21.
2019
17.22:4..44
3.6
0.4
TETLO-pdars (ours)
16.05-+0.08
Podarts Xu ct 2020
17.96+0.15
Al'I-podaits [He el 20166
17.7120.19
4.2
0.2
pedarts
17.82.10.16
MTI. -pddarrs Maninis ci
2019)
17.85++0.23
4.0
0,2
TETLO-podart (ours)
16.4240.07
0.1
*Priartss (7hou 2020hh
16.48-10.06
AlTT-prdarts (Tg el
A0160
17.35:10.14
3.4
0.4
podurts
16.73-10.27
MTL-prdant Mañinis et al:
2019)
17.46++0. 10
3.3
0,3
TETLO-prdart (ours)
15.81=00066
0,2
4.4
CIFAR-100 AND CIFAR 10 RESULTS UNDER SEARCH SPACES OF DARTS, P-DARTS, PC-DARTS, AND PR-DARTS
For the main encoder architecture, we experimented with the search spaces in DARTS (Liu et al. 2019a), P-DARTS (Chen et al. 2019).
MMin Auxillrry
Cassitcationn enrer
PC-DARTS (Xu et al. (020). and PR-DARTS (hhou et al.) 2020b).
8.34
During architecture search on CIFAR- 100 and CIFAR-10 10, the ar- chitecture is a stack of 8 cells, each consisting of 7 nodes. Initial channel number was set to 16. The rest of hyperparameters follow
0.2
Error of QRS
those in DARTS, P-DARTS, PC-DARTS, and PR-DARTS.
Figure How validation errors of the auxiliary and main models change with the error of QRS.
In Table 2 we compare different NAS methods on CIFAR-100. From these results, we observe the following. First of all, when our method TETLO is applied to DARTS, P-DARTS, PC-DARTS.
and PR- RRRDAR,S, classification errors test data significantly reduced (significance test results are in Appendix R which shows that encouraging main-te tilt-to-uuuu -iiii---uuillaryy transferability can improve the generalizability of the main model. Table 18 shows that architectures searched by our method have

---

Published as a conference paper at ICLR 2023
fewer skip connections than baselines, which demonstrates that method has better stability. By minimizing the validation loss of the auxiliary trained on knowledge (quadruple relative similarities) generated by the main model, more accurate knowledge can be generated. By promoting the fidelity of generated knowledge, the generalizability and stability of the main model improved.
For the above analysis, provide pirical evidence. To show that minimiz- ing auxiliary's validation loss can yield more accurate quadruple relative sim- ilarity (QRS) relationships, we create QRS relationships with different levels of accuracy. Given (x.y) and (2, if x and have the same class label and and w have different class labels, then the similarity y(x,y) between x and y is considered to be larger than the simi- larity c(z, w) between and w. First, we create a set fully accurate QRS relationships. Then randomly sam- ple 25%, 50%, and 75% quadruples and make their relative similarity relation- ships incorrect, i.e., changing c(x,y) c(z,w) c(x,y) < c(z, w). By doing this, we generate three sets of QRS with decreasing accuracy. Using the three sets as constraints, we train three aux- iliary models. Figure shows that clas- sification errors crease as the error of QRS relationships increases. These results show that the fidelity of knowledge is positively cor- related to auxiliary's 's classification accu- racy, Therefore, minimizing auxiliary's validation loss can yield more reason- able knowledge.
Table 3: Results on CIFAR-10, including classifica-
tion error (%) on the test set, number of model weights (millions), and search cost (GPU days). denotes that the results are taken from DARTS (Chu al. 2020a). NoisyDARTS (Chu et al. 2020b), DrNAS (Chen el al, 2020b). B-DARTS (Ye al. 2022), and AGNAS (Sun)
et al. 2022). The rest notations are the same as those in Table the main paper.
Method
Emorr()) (%)
Param(M)
Cost
*DenseNet (Hunng et al.
2017
3.46
25.6
"ENAS [Pham ot al. 2018
2.89
4.6
0.5
*NASNeLA Zoph
2018
2.65
3.3
1800
AAmocbaNetB Keal ot
2019
2.55++0.05
2.8
3150
*R-DARTS Zela
2.954-0.21
*ASAP Nov al 20200
2.68+0.11 11
2.5
0.2
*SARRTS (C lonn & Hitchh
2020
2.61:0.022
*DARTS+ Wang
2021
2.61100.08
3.0
0.8
*DropNAS Hring et al 221
2.58-40.1
0.6
*FirDARTS
2.54
*DiNAS Chen el 20221
2.54.10.03
4.0
0.4
TISTA-NAST (ang of 20220
2.54+00.05
"MiLeNAS (HS al 2020)
2.51+0.11
3,9
0,3
*GAEA ét [
2.5010.006
0.1
"PDART' (Chen & lisieh 2020
2.48.10.22
validation set in-
*GTN (FT (28) (Sach ct 2019
2424-0033
97.9
0.67
*DOTS (Gu Ar
2.49+0.06
*B- DARDARTS of al 20122
2.53+-0.08
3.8
0.4
*AGNAS (Sun al
(200
2.53+.003
Darisaad (Li el
2019a
2.76+00.09
3,3
4.0
AaT-darts2ai
201
2.78:-0.05
ES-daris2nd Kookinno
2017
2.76++0.14 14
3,4
4,0
MTL--durt2hn [Maiiiis ct
2019
2.75+0.06
TETLO-darts22n (ours)
2.62:00.6
3.2
4.0
Pdarts (Chen et 2019
2.35=00.06
3.5
0.3
Al'T-pdart [He at
201bb
2.6310.10
ES-pdarts (Kokkinos 2017
2.59+0005
3,6
0,3
To show that promoting knowledge' fi- delity can improve the generalizabil- ity of the main model, we train main models by fitting QRS relationships that have different accuracy. Then we mea- sure classification errors of these mod- els. Figure 2 shows that main model trained using more accurate QRS rela- tionships achieves lower classification error. This indicates the fidelity of knowledge is positively correlated with the classification accuracy of the main model. Therefore, promoting knowledge" fidelity can im- prove the generalizability of the main model.
MILL-pdaris [Nammis cl
2019
2.57:0.07
0.4
TET1O pilutts (ours)
2.47
0.3
*Pedarts (Xu et 20220
20220)
2.3720.07
3.6
0.1
ATT-ppdaais a.
2016b
2.68.10.06
ESS Pedppettts Kokkinn 2017
2.63-10.10
3.6
0.1
MILL-pedaris (Mamnnn cl cl.
2019
2.6010.05
0.1
TET10. peslarts
2.48
3.5
0,1
*Prdarts (Zhou et (0020b)
2.3720.03
3.5
0.2
prdarts [[le
2.40
ES FS--prdrrts KKlkkinot
2117
2.60+-0.1 14
3.6
0,2
MTL-prdarts (Maiiii cl
2019
2.55+00.8
TETLO-prdats (ous)
2.17+0.02
0.2
Second, Tablee our method performs better than MTL. In MTL, main and auxiliary tasks leamed simultancously by minimizing the weighted sum of their training losses, which results in competition between these two tasks: more decrease of auxiliary loss incurs less decrease of main loss; vice versa. In contrast, our method performs these two tasks sequentially. The auxiliary task helps the main task to learn instead of competing with Third, our method performs better than ATT. ATT is classic way of performing transfer learning, which pretrains a model in auxiliary task, then transfers this model to perform the main task. Our method performs transfer learning in reverse direction: transfer is performed from main model to auxiliary model; quality of the main model is evaluated by its transferability to the auxiliary model. Results show our proposed direction of performing transfer is more effective. Fourth, method outperforms ES. ES requires the main
7

---

Published as a conference paper at ICLR 2023
model and auxiliary model to have the same encoder, which might be too restrictive and prohibit the learning of rask-specific representations. Main and auxiliary tasks are different. A good representa- method allows different
tion beneficial for one task may not be suitable for the other. In contrast, tasks to have different encoders, which facilitates task-specific representation learning. Fifth, while TETLO framework can greatly help improve the quality of searched architectures, it has sim- ilar model parameter number and search cost to baselines. The memory costs (Table 21 of our methods are also similar to those of baselines.
Table show results on CIFAR-IO. From this table, we make similar observations as those in outperforre
Table Our methods outperform vanilla DARTS, P-DARTS, PC-DARTS, PR-DARTS, and achieve lower emrors than ATT, ES, MTL. The analysis of reasons is similar to that for results in Table
4.5 IMAGENET RESULTS UNDER SEARCH SPACES OF DARTS, P-DARTS, PC-DARTS
For ImageNet, following (Xu et al. 2020). 10% of the 1.2M images randomly sam- pled as a training set and another 2.5%
Table 4: Top-1 1 top-5 classification emoor on ImageNet test set. SSTLO-darts2nd by applying TETLO to DARTS 2nd on CIFAR- 10. Results marked with
10: an architecture searched
are used for validation. Table 4 shows top-1 and top-5 test errors on ImageNet, where our methods outperform DARTS, P-DARTS, and PC-DARTS. These results again demonstrate the effectiveness of our method in improving the main encoder ar- chitecture by encouraging good transfer- ability to an auxiliary model. Our method performs better than MTL, ATT, and ES, which further demonstrates the benefits of performing main and auxiliary tasks se- quentially than jointly, transferring knowl- edge from the main model to an auxiliary instead of in the opposite direction, and learning task-specific representations.
taken from DARTS (Chu al. 2020a),
DrNAS (Chen al. (020b). OFA CCai al. (020aa) FB-
Net (Wu al. 2019).
B-DARTS (ee et al. 2022), and AG-
NAS (Sun et al. 2022)
The other notations are the same as
those in Table 2
Top-1
Top-5
*Inception v1 Szegedy et 2015
30.2
10.1
*ShuffleNet 2x (V2)
2018)
25.1
7.6
*NASNet-A (Zoph et al
2018
26.0
8.4
*AmoehaNes
*SDARTS-ADV-CIAAP
Hsich
2020
25.2
7.8
*PC-DARTS-CIFAR (Xu et 2020
25.1
7.8
*HBNettCC
*Pruxyless NAA ImugeNel CCai et
2019
24.9
7.5
FairDARRSS ImageNNe
*PR-DARTS ZZoou et 20.00
24.1
7.3
DDARTS" -CIFAAKI 1 1ing oral.
2019
23.7
7.2
OFA (Cai
20200
20.0
DOTS (Gu 2t al 202
|
24.3
7.4
4.6 ABLATION STUDIES
B.-DARIS e
*AGNAS (Sin et all (1744
23.4
6.8
In this section, we perform ablation studies understand the contributions of individ- ual components in our framework.
Darts2nd-cifar1
2019g
8.7
daartsnnd-cifan Iur et cu
26.3
8.2
daris2nd cifarl0 (Kokkinos 2017
ATTL-dart2nd-ciff 10 (Maninis ml.
2019
26.4
8.3
Ablation on knowledge transfer ap- proaches. We compare our quadruple rel- ative similarity based knowledge transfer
TETL TETLO-darts2i O-do-r-2nd--cffaa (ouro
25.4
7.8
"Fdarts-cifar10 (Chen [019
2019
ATT pdarts cifurTU (1le et
2016b
24.4
7.2
2S--parts-cifar10) (Kokkinos
approach with: 1) L. regularization on embeddings (RegEmbed) (Romero et al. 2014]: encouraging embeddings gener- ated by auxiliary encoder to have small 1.2 distance with those generated by main encoder; 2) pseudo-labeling (PL) (Zhang et al. 2017 Lan et al. 2018: [Pham et al. 2020 : the main model generates pseudo- labels. which are used to train the auxil-
ITTL-paats-cifarti Maninis
2019)
24.3
7.2
TETL TETLO O-pdarts-ciffr (ours))
23.6
7.0
*Fdarts-cifarI00 (Chen
2019)
24.7
7.5
ATT pdarts cifur cifur el
20166
24.6
7.5
ES-pdarts-cifar (Kokkinns
htl--patt-iffaaltrrrryy (Mlaiii et
2019
24.7
7.7
TETLO O-t.--aat--iifaa100000000 (ours)
24.1
7.3
Pedarts- Imallett--mmgeNerr (Xu
[202]
24.2
7.3
ATT-pecaat--naggee.
al
2016b)
24.0
7.1
(Kokkimos
HTLL-pddtts-mmage/ Maninis cl
2019
23.9
7.3
O-iTat-sd poddarts-ImageeN (ours)
23.3
6.2
iary; 3) pairwise similarity (PS) (Chen al.]
2020a): the main model annotates whether two images are similar or dissimilar and the auxiliary fits these similarity labels; and 4) triple-wise similarity (TS) ( Youu et al. 2017): the main model compares similarity among three instances (c.g., x is more similar to y than to 2) and the auxiliary fits these triple-wise similarities. Please refer to Appendix for detailed setup of these baselines.
Figure shows the results. We observe the following. First, our method works better than RegEm- bed and PL. These two baselines transfer knowledge individual data instances without consider- ing the relationship between instances. In contrast, our method considers relative similarity relation- ships among four data instances during knowledge transfer. Second, method works better than
8

---

Published as a conference paper at ICLR 2023
PS and TS. PS and TS are limited to capturing second-order and third-order relationships among instances while our method can capture fourth-order relationships.
Figure 6 shows 4-ncarest neighbors retrieved by different methods, for some randomly sampled CIRAR-100 test images. We use the trained main encoder to generate image encodings and calculate cosine similarity on these encodings. Nearest neighbors are retrieved based on cosine similarity. Compared with baselines, nearest neighbors retrieved by QRS are more semantically similar to query images. By capturing 4th-order similarity relationships among instances, QRS can more effectively group similar images together, which is good for classification.
Query
MNs by QRS
NAs by PL
NNA by PS
Nik by
Figure 3: Nearest neighbors retrieved by different methods.
Ablation on removing main validation loss. Next, we perform another ablation study "auxiliary val- idation only": main encoder architecture is updated by minimizing auxiliary validation loss only, without considering main validation loss. TETLO is applied to DARTS-2nd. Table E shows the results. On both datasets, minimizing the sum of main and aux- iliary validation losses ("main+auxiliarr") results in better architectures than minimizing auxiliary valida- tion loss only. The reason is that auxiliary valida- tion loss indirectly measures the quality of architectures. In contrast, the main validation loss directly depends on main encoder architectures and can serve as direct (hence stronger) signal guide architecture search. In the end, combining the direct signal (main validation loss) and indirect signal (auxiliary validation loss) together is more beneficial than using the indirect signal only.
RegEmbed
TS Ours
18
17
DARTS(2nd)
Figure 4: Comparison of knowledge transfer approaches. The y-axis is test error on CIFAR-100.
Ablation on tradeoff parameters. Finally, we in- vestigate how performance changes with tradeoff pa- rameters y and 2 in E3. TETLO is applied to 5K held-
Table 5:
Test errors in the ablation of "auxiliary validation only". C denotes CI- FAR.
DARTS-2nd. Figure 5 shows the results out validation set of CIFAR-100. A value of y in the middle ground that properly balances validation losses of the main and auxiliary models yields the optimal performance. This is also the case for à.
Method
Error (%)
Auxiliary only (C100)
19.5440.13
Main + auxiliary (C100)
17.21 10
Auxiliary only (C10)
2.88.10.09
Main + auxiliary (C10)
2.62+00.06
Other ablations. We conduct other ablations, such the following one: ignoring the auxiliary task entirely, defining QRS loss the validation data, and using this QRS loss together with the classification loss on the validation set to optimize architectures. Please refer to Appendix [or details.
CHAR-300
5 CONCLUSIONS
223
25
30
22.32
21.34
213
20.09
We propose transferabliity-nccorrgggg tri- level optimization framework which aims to improve the generalizability and stability of dif- ferentiable neural architecture search methods by encouraging good transferability to auxiliary models. Our framework performs three learning stages end-to-end. First, we train main model. Second, the knowledge of the main model is transferred to an auxiliary model based on match- ing quadruple relative similarities. Third, the main-to -uxxiiiar transferability is measured and the main model is improved by maximizing this transferability. Experiments on a variety of datasets demonstrate the efficacy of our methods.
Dombda
Figure 5: How errors change with y (left) and A (right).

---

Published as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
Datasets used in experiments are publicly available and are cited and described S.ction Major
training details including hyperparameters, parameter tuning strategies, data splits. etc. are specified
in Section 4.2. Section M Section M Section [ Section P] Section Q Section [T and Section V
Standard deviations of results (after running experiments multiple times with random seeds) are
reported in Table 2 Table []] Table [6] Table 3] and Table [2 Statistical significance tests are
reported Section R
Computation time reported Table Table Table 19
Table 20 and
Table 25
GPU type is reported in Section M.
ACKNOWLEDGEMENTS
We thank all the reviewers for invaluable comments and feedback.
REFERENCES
Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingfod,, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. arXiv preprint arXiv: 1606. 04474, 2016.
Antreas Antoniou, Harrison Edwards, and Amos Storkey, How to train your maml. arXiv preprint arXiv:1810.09502, 2018.
Amittai Axelrod, Xiaodong He, and Jianfeng Gao. Domain adaptation via pseudo in-domain data selection. In Proceedings of the 2011 conference on empirical methods in natural language processing, pp. 355-362, 2011.
Atilim Gunes Baydin, Robert Cornish, David Martinez-Rubi,, Mark Schmidt, and Frank D. Wood. Online learning rate adaptation with hypergradient descent. CoRR, abs/1703.04782, 2017. URL
httt:://axivv: org/
Joachim Bingel and Anders Søgaard. Identifying beneficial task relations for multi-task learning in deep neural networks. arXiv preprint arXiv: 2017.
Karsten M Borgwardt, Arthur Gretton, Malte J Rasch, Hans-Peter Kriegel, Bernhard Schölkopf, and Alex J Smola. Integrating structured biological data by kernel maximum mean discrepancy. Bioinformatics, 22(14):e49-c57, 2006.
Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search target task and hardware. In ICLR, 2019.
Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once for all: Train one network and specialize for efficient deployment. In International Conference on Learning Representations, 2020a.
Han Cai, Chuang Gan, Tianzhe Wang. Zhekai Zhang, and Song Han. Once-for-all: Train one network and specialize it for efficient deployment, 2020b.
RA Caruana. Multitask learning: A knowledge-basee source of inductive bias, ser. Learning to Learn. US: Springer, 1998.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709 2020a.
Xiangning Chen and Cho-Jui Hsieh. Stabilizing differentiable architecture search via perturbation- based regularization. CoRR, abs/2002.05283, 2020.
Xiangning Chen, Ruochen Wang. Minhao Cheng. Xiaocheng Tang, and Cho-Jui Hsieh. Drnas: Dirichlet neural architecture search. CoRR, abs/2006. 10355, 2020b.
Xin Chen, Lingxi Xie, Jun Wu, and Qi Tian. Progressive differentiable architecture search: Bridging the depth gap between search and evaluation. In ICCV, 2019.
10

---

Published as a conference paper at ICLR 2023
Yi-Wei Chen, Qingquan Song, Xi Liu, PS Sastry, and Xia Hu. On robustness of neural architecture search under label noise. Frontiers in big Data, 3:2, 2020c.
Zhao Chen, Vijay Badrinarayanan, Chen- Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In International Conference on Machine Learning, pp. 794-803. PMLR, 2018.
Xiangxiang Chu, Tianbao Zhou, Bo Zhang, and Jixiang Li. Fair DARTS: eliminating unfair advan- tages in differentiable architecture search. CoRR, abs/1911. 12126, 2019.
Xiangxiang Chu, Xiaoxing Wang, Zhang, Shun Lu, Xiaolin Wei, and Junchi Yan. DARTS-: robustly stepping out of performance collapse without indicators. CoRR, 0tss/2009.010277 2020a.
Xiangxiang Chu, Bo Zhang, and Xudong Li. Noisy differentiable architecture search. CoRR, abs/2005.03566. 2020b.
Yin Cui, Menglin Jia, Tsung-Yi Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.
2019.
Jifeng Dai, Kaiming He, and Jian Sun. Instance-awand semantic segmentation via multi-task network cascades. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 3150-3158, 2016.
Stephan Dempe. Foundations of bilevel programming. Springer Science & Business Media, 2002.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
Carl Doersch and Andrew Zisserman. Multi-task sulf-supervised visual learning. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2051-2060, 2017.
Xuanyi Dong and Yi Yang. Searching for robust neural architecture in four GPU hours. In CVPR, 2019.
Lixin Duan, Ivor W Tsang, Dong Xu, and Tat-Seng Chua. Domain adaptation from multiple sources via auxiliary classifiers. In Proceedings of the 26th Annual International Conference Machine Learning, pp. 289-296, 2009.
Lixin Duan, Dong Xu, and Ivor Wai-Hung Tsang. Domain adaptation from multiple sources; A domain-dependent regularization approach. IEEE Transactions on neural networks and learning systems, 23(3):504-518, 2012.
Long Duong, Trevor Cohn, Steven Bird, and Paul Cook. Low resource dependency parsing: Cross- lingual parameter sharing in neural network parser. Proceedings of the 53rd annual meeting of the Association for Computational Linguistics and the 7th international joint conference natural language processing (volume 2: short papers), pp. 845-850, 2015.
Thomas Elsken, Benedikt Staffler, Jan Hendrik Metzen, and Frank Hutter. Meta-learning of neural architectures for few-shot learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 12365-12375 2020.
Matthias Feurer, Jost Springenberg, and Frank Hutter. Initializing bayesian hyperparameter opti- mization via meta-learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1126-1135. JMLR. org, 2017.
George Foster, Cyril Goutte, and Roland Kuhn. Discriminative instance weighting for domain adap- tation statistical machine translation. In Proceedings of the 2010 conference on empirical methods in natural language processing, pp. 451-459, 2010.

---

Published as a conference paper at ICLR 2023
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Romain-adverseee training of neural net- works. The journal of machine learning research, 17(1):20962000 2030, 2016.
Saeed Ghadimi and Mengdi Wang. Approximation methods for bilevel programming. arXiv preprint arXiv:1802.0f22444 2018.
Spyros Gidaris and Nikos Komodakis. Dynamic few-shot visual learning without forgetting, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4367- 4375, 2018
Ting Gong, Tyler Lee, Cory Stephenson, Venkata Renduchintala, Suchismita Padhy, Anthony Ndi- rango, Gokce Keskin, and Oguz H Elibol. A comparison of loss weighting strategies for multi task learning in deep neural networks. JEEE Access, 7:141627-141632, 2019.
Riccardo Grazzi, Luca Franceschi, Massimiliano Pontil, and Saverio Salzo. On the iteration com- plexity of hypergradient computation. In International Conference on Machine Learning, pp- 3748-3758. PMLR, 2020.
Yu-Chao Gu, Li-Juan Wang, Yun Liu, Yi Yang, Yu-Huan Wu, Shao-Ping Lu, and Ming-Ming Cheng- Dots: Decoupling operation and topology in differentiable architecture search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12311 1-12320, 2021.
Chaoyang He, Haishan Ye, Li Shen, and Tong Zhang. Milenas: Efficient neural architecture search via mixed-level reformulation, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016b.
Judy Hoffman, Eric Tzeng, Taesung Park, Jun- Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. In International conference on machine learning, pp. 1989-1998. PMLR, 2018.
Weijun Hong, Guilin Li, Weinan Zhang, Ruiming Tang, Yunhe Wang, Zhenguo Li, and Yong Yu.
Dropnas: Grouped operation dropout for differentiable architecture search. IJCAI, 2020.
Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: survey, arXiv preprint Xix:20004.05439 2020.
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. In CVPR, 2017.
Jiayuan Huang, Arthur Gretton, Karsten Borgwardt, Bernhard Schölkopf, and Alex Smola. Correct- ing sample selection bias by unlabeled data. Advances in neural information processing systems, 19:601-608 2006.
Kaiyi Ji. Junjie Yang, and Yingbin Liang. Bilevel optimization: Convergence analysis and enhanced design. In International Conference on Machine Learning, pp. 4882 4892. PMLR, 2021.
Jing Jiang and ChengXiang Zhai. Instance weighting for domain adaptation in nlp. ACL, 2007.
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7482-7491, 2018.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
lasonas Kokkinos. Ubernet: Training universal convolutional neural network for low-, mid- and high-level vision using diverse datasets and limited memory. In Proceedings of the IEEE Confer- ence on Computer Vision and Pattern Recognition, pp. 6129-6138, 2017.
12

---

Published as a conference paper at ICLR 2023
Alex Krizhevsky and Geoff Hinton. Convolutional deep belief networks on cifar-10. 10. Unpublished manuscript, 1-, 2010.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Himanshu Kumar and PS Sastry. Robust loss functions for learning multi-class classifiers. In 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp. 687-692, IEEE, 2018.
Xu Lan, Xiatian Zhu, and Shaogang Gong. Knowledge distillation by on-the-fly native ensemble, 2018.
Isabelle Leang, Ganesh Sistu, Fabian Bürger, Andrei Bursuc, and Senthil Yogamani. Dynamic task weighting methods for multi-task networks in autonomous driving systems. In 2020 IEEE 23rd International Conference Intelligent Transportation Systems (FTSC), pp. 1 1-8. IEEE, 2020.
Yoonho Lee and Seungjin Choi. Gradient-basee meta-learning with learned layerwise metric and subspace. In International Conference Machine Learning, pp. 2927-2936. PMLR, 2018.
Kiran Lekkala and Laurent Itti. Attentive feature reuse for multi task meta learning. arXiv preprint
arXiv:2006.0744 07438, 2020.
Ke Li and Jitendra Malik. Learning to optimize. arXiv preprint arXiv: 2016.
Liam Li, Mikhail Khodak, Maria-Florina Balcan, and Ameet Talwalkar. Geometry-aware gradient algorithms for neural architecture search, 2021.
Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for few- shot learning. arXiv preprint arXiv: 2017.
Dongze Lian, Yin Zheng, Yintao Xu, Yanxiong Lu, Leyu Lin, Peilin Zhao, Junzhou Huang, and Shenghua Gao. Towards fast adaptation of neural architectures with meta learning. In Interna- tional Conference on Learning Representations, 2019.
Hanwen Liang, Shifeng Zhang, Jiacheng Sun, Xingqiu He, Weiran Huang, Kechen Zhuang, and Zhenguo Li. DARTS+: improved differentiable architecture search with early stopping. CoRR, abs/1909.06035. 06005, 2019.
Hanwen Liang, Shifeng Zhang, Jiacheng Sun, Xingqiu He, Weiran Huang, Kechen Zhuang, and Zhenguo Li. Darts+: Improved differentiable architecture search with early stopping, 2020.
Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan L. Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In ECCV, 2018a.
Hanxiao Liu, Karen Simonyan, Oriol Vinyals, Chrisantha Fernando, and Koray Kavukcuoglu. Hi- erarchical representations for efficient architecture search. In ICLR, 2018b.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: differentiable architecture search. In ICLR, 2019a.
Hong Liu, Mingsheng Long, Jianmin Wang, and Michael Jordan. Towards understanding the transferability of deep representations. arXiv preprini arXiv: 12031, 2019b.
Risheng Liu, Yaohua Liu, Shangzhi Zeng, and Jin Zhang. Towards gradient-based bilevel optimiza- tion with non-convex followers and beyond. Advances in Neural Information Processing Systems,
34, 2021.
Shikun Liu, Edward Johns, and Andrew J Davison. End-to-end multi-task learning with attention. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp- 1871-1880 2019c.
13

---

Published as a conference paper at ICLR 2023
Mingsheng Long, Jianmin Wang, Guiguang Ding, Jiaguang Sun, and Philip S Yu. Transfer feature learning with joint distribution adaptation. In Proceedings of the IEEE international conference computer vision, pp. 2200-2207, 2013.
Mingsheng Long, Zhangjie Cao, Jianmin Wang. and Philip Yu. Learning multiple tasks with multilinear relationship networks. arXiv preprint
Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael Jordan. Conditional adversarial domain adaptation. arXiv preprint arXiv: 10667, 2017.
Yongxi Lu, Abhishek Kumar, Shuangfei Zhai, Yu Cheng, Tara Javidi, and Rogerio Feris. Fully- adaptive feature sharing in multi-task networks with applications in person attribute classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5334
5343, 2017.
Ping Luo, Fuzhen Zhuang, Hui Xiong, Yuhong Xiong, and Qing He. Transfer learning from multiple source domains via consensus regularization. In Proceedings of the 17th ACM conference Information and knowledge management, pp. 103-112, 2008. 2008.
Zelun Luo, Yuliang Zou, Judy Hoffman, and Li Fei-Fei. Label efficient learning of transferable representations across domains and tasks. arXiv preprint 172iv:17/2.00123, 2017.
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet V2: practical guidelines for efficient CNN architecture design. In ECCV, 2018.
Kevis-Kokitsi Maninis, Ilia Radosavovic, and lasonas Kokkinos. Attentive single-tasking of multi- ple tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 1851-1860, 2019.
Lilyana Mihalkova, Tuyen Huynh, and Raymond Mooney. Mapping and revising markov logic networks for transfer learning. In Auai, volume pp. 608-614, 2007.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta- 2017.
learner. arXiv preprint arXiv:1707.033
Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. Cross-stitch networks for multi-task learning. In Proceedings of the IEEE conference on computer vision and pattern recog- nition, pp. 3994 4003, 2016.
Robert C Moore and Will Lewis. Intelligent selection of language model training data. 2010.
Jiquan Ngiam, Daiyi Peng, Vijay Vasudevan, Simon Kornblith, Quoc Le, and Ruoming Pang. Do- main adaptive transfer learning with specialist models. arXiv preprint arXiv:1811.07056 2018.
Alexandru Niculescu-Mizil and Rich Caruana. Inductive transfer for bayesian network structure learning. In Artificial intelligence and statistics, pp. 339-346. PMLR, 2007.
Asaf Noy, Niv Nayman, Tal Ridnik, Nadav Zamir, Sivan Doveh, Itamar Friedman, Raja Giryes, and Lihi Zelnik. ASAP: architecture search, anneal and prune. In AISTATS, 2020.
Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on knowledge and data engineering, 12(10):1345-1353 2009.
Sinno Jialin Pan, James Kwok, Qiang Yang, et al. Transfer learning via dimensionality reduction. In AAAI, volume 8, pp. 677-682, 2008.
Sinno Jialin Pan, Ivor W Tsang, James T Kwok, and Qiang Yang. Domain adaptation via transfer component analysis. IEEE Transactions on Neural Networks, 2292):199-210, 2010.
Hieu Pham, Melody Y. Guan, Barret Zoph, Quoe V. Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. In ICML., 2018.
Hieu Pham, Qizhe Xie. Zihang Dai, and Quoc V Le. Meta pseudo labels. arXiv:2003. 10580, 2020.
arXiv preprint
14

---

Published as a conference paper at ICLR 2023
Lorien Y Pratt. Discriminability- based transfer between neural networks, Advances in neural infor- mation processing systems, pp. 204, 1993.
Siyuan Qiao, Chenxi Liu, Wei Shen, and Alan Yuille. Few-shot image recognition by predicting parameters from activations. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 7229-7238, 2018.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. 2016.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. Regularized evolution for image classifier architecture search. In Proceedings of the aaai conference on artificial intelligence, volume 33, pp. 4780-4789 2019.
Zhongzheng Ren, Raymond Yeh, and Alexander Schwing. Not all unlabeled data are equal: Learn- ing to weight data in semi-supervised learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 21786-21797, Curran Associates, Inc., 2020. URL https:://proceeednnnne neurisg.........s...o..........s c0/ h1tp:///0000111fffaccaaaa3222211
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and
Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550, 2014.
Sebastian Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint 1705007706.00098. 2017.
Ozan Sener and Vladlen Koltun. Multi-task learning multi-objective optimization. arXiv preprint arXiv: 2018.
Jian Shen, Yanru Qu, Weinan Zhang, and Yong Yu. Wasserstein distance guided representation learning for domain adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. Meta-weight net: Learning explicit mapping for sample weighting. In Advances in Neural Information Processing Systems, pp. 1919-1930, 2019.
Trevor Standley, Amir Zamir, Dawn Chen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Which tasks should be learned together in multi-task learning? In International Conference Machine Learning, pp. 9120-9132. PMLR, 2020.
Felipe Petroski Such, Aditya Rawal, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. Genera- tive teaching networks: Accelerating neural architecture search by learning to generate synthetic training data. CoRR, abs/1912.077688, 2019.
Masashi Sugiyama, Shinichi Nakajima, Hisashi Kashima, Paul Von Buenau, and Motoaki Kawan- abe. Direct importance estimation with model selection and its application to covariate shift adaptation. In NIPS, volume pp. 1433-1440. Citeseer, 2007.
Qianru Sun, Yaoyao Liu, Zhaozheng Chen, Tat-Seng Chua, and Bernt Schiele. Meta-transfer learn- ing through hard tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
Zihao Sun, Yu Hu, Shun Lu, Longxing Yang, Jilin Mei, Yinhe Han, and Xiaowei Li. Agnas: quuided micro and archtterche search. In International Conference on Machine Learning. pp. 20777-20789. PMLR, 2022.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, 2015.
Aleksandr Timofeev, Grigorios G Chrysos, and Volkan Cevher. Self-supervised neural architecture search for imbalanced datasets. arXiy preprint (09.::2000008888 2021.
Tatiana Tommasi, Francesco Orabona, and Barbara Caputo. Safety in numbers: Learning cate- gories from few examples with multi model knowledge transfer. In 2010 JEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 3081 300888088. IEBE, 2010.
15

---

Published as a conference paper at ICLR 2023
Eric Tzeng, Judy Hoffmann, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7167-766,, 2017.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems, 29, 2016.
Jindong Wang, Yiqiang Chen, Shuji Hao, Wenjie Feng, and Zhiqi Shen. Balanced distribution adaptation for transfer learning. In 2017 IEEE international conference data mining (ICDM), pp. 1134-1 IERE, 2017a.
Rui Wang, Masao Utiyama, Lemao Liu, Kehai Chen, and Eiichiro Sumita. Instance weighting for neural machine translation domain adaptation. In Proceedings of the 2017 Conference on Empirical Methods Natural Language Processing, pp. 1482-1488, 2017b.
Ruochen Wang, Minhao Cheng, Xiangning Chen, Xiaocheng Tang, and Cho-Jui Hsieh. Rethinking architecture selection in differentiable nas. In International Conference on Learning Representa- tion, 2021.
Yulin Wang. Jiayi Guo, Shiji Song, and Gao Huang. Meta-semi: A meta-learning approach for semi-supervised learning. CoRR, abs/2007.02394, 2020. URL
2.00702300
Chris Williams, Edwin V Bonilla, and Kian M Chai. Multi-task gaussian process prediction. Ad- vances in neural information processing systems, pp. 153-160, 2007.
Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang. Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10734 10742, 2019.
Pengtao Xie, Yuntian Deng, Yi Zhou, Abhimanu Kumar, Yaoliang Yu, James Zou, and Eric P Xing. Learning latent space models with angular constraints. In International Conference on Machine Learning, pp. 3799-3810. PMLR, 2017.
Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. SNAS: stochastic neural architecture search. In ICLR, 2019.
Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Guo-Jun Qi, Qi Tian, and Hongkai Xiong. PC- DARTS: partial channel connections for memory-efficient architecture search. In ICLR, 2020.
Junjie Yang, Kaiyi Ji, and Yingbin Liang. Provably faster algorithms for bilevel optimization. Ad- vances in Neural Information Processing Systems, 34, 2021.
Yibo Yang, Hongyang Li, Shan You, Fei Wang, Chen Qian, and Zhouchen Lin. Ista-nas: Efficient and consistent neural architecture search by sparse coding, 2020.
Yongxin Yang and Timothy M Hospedales. Trace norm regularised deep multi-task learning. arXiv preprint arXiv: 1606:.666..0000 00038, 2016.
Huaxiu Yao, Xian Wu, Zhiqiang Tao, Yaliang Li, Bolin Ding, Ruirui Li, and Zhenhui Li. Automated relational meta-learning. arXiv preprint arXiv:2001.00745 2020.
Peng Ye, Baopu Li, Yikang Li, Tao Chen, Jiayuan Fan, and Wanli Ouyang. b-darts: Beta-decay regularization for differentiable architecture search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 01874-10883, 2022.
Shan You, Chang Xu, Chao Xu, and Dacheng Tao. Learning from multiple teacher networks. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD '17. pp. 1285-1294, New York, NY. USA, 2017. Association for Computing Machinery. ISBN 97819908383 450344874. doi: 10.1 10.1145/3097983330988188
URL hhttps://dot
1T1/10.1145/309 3098115
16

---

Published as a conference paper at ICLR 2023
Amir R Zamir, Alexander Sax, Nikhil Cheerla, Rohan Suri, Zhangjie Cao, Jitendra Malik, and Leonidas Guibas. Robust learning through cross-task consistency. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11197-11206, 2020.
Arber Zela, Thomas Elsken, Tonmoy Saikia, Yassine Marrakchi, Thomas Brox, and Frank Hutter. Understanding and robustifying differentiable architecture search. arXiv preprint arXiv:1909.088 09656, 2019.
Arber Zela, Thomas Elsken, Tonmoy Saikia, Yassine Marrakchi, Thomas Brox, and Frank Hutter. Understanding and robustifying differentiable architecture search. In ICLR, 2020.
Yabin Zhang, Hui Tang, Kui Jia, and Mingkui Tan. Domain-symmetric networks for adversarial domain adaptation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5031-5040, 2019.
Ying Zhang, Tao Xiang, Timothy M. Hospedales, and Huchuan Lu. Deep mutual learning, 2017.
Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu. Deep mutual learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4320- 4328, 2018
Yu Zhang and Qiang Yang. A survey on multi-task learning. IEEE Transactions on Knowledge and Data Engineering, 2021.
Yu Zhang and Din Yeung, A regularization approach learning task relationships in multitask learning. ACM Transactions on Knowledge Discovery from Data (TKDD), 8(3):1331, 2014.
Guoqing Zheng, Ahmed Hassan Awadallah, and Susan T. Dumais. Meta label correction for learn- ing with weak supervision. CoRR, abs/1911.03809, 2019. URL http::///////
1911. 03809
Pan Zhou, Caiming Xiong, Richard Socher, and Steven C. H. Hoi. Theory-inspired path-regularized differential network architecture search, 2020a.
Pan Zhou, Caiming Xiong, Richard Socher, and Steven C. H. Hoi. Theory-inspired rath-regularized differential network architecture search. CoRR, abs/2006.16537, 2020b. URL https://
htt.v/orggabbb//0oooo 16537
Fuzhen Zhuang, Ping Luo, Hui Xiong, Yuhong Xiong, Qing He, and Zhongzhi Shi. Cross-domain learning from multiple sources: consensus regularization perspective. IEEE Transactions on Knowledge and Data Engineering. 22(12):1664-1677 2009.
Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He. A comprehensive survey on transfer learning. Proceedings of the IEEE, 109(1): 43-76, 2020.
Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. In ICLR, 2017.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In CVPR, 2018
17

---

Published as a conference paper at ICLR 2023
Table 6: Test emorr (%) CIFAR-10, when validation data has label noise and loss function is cross-entropy.
Method
Error(%)
Darts2nd (Liu et al. 2019a)
21.35++0:40
TETL -orTt--ddrtt2ndd (ours)
18.71++0.33
Table 7: Test error (%) on CIFAR-10, when validation dataset is
Method
Error(%)
Darts2nd (Liu et al. 2019a)
16.02-1:025
BETLO-darts22n (ours)
14.29::0.17
APPENDIX
A LABEL NOISE AND CLASS IMBALANCE IN VALIDATION DATASET.
In this section, we discuss the case where validation data has label noise class imbalance. In any differentiable neural architecture search method, there is always a validation set used to update architecture variables. This validation is used the auxiliary model's validation set in method. As long as differentiable NAS method has access to a class-balanced and non-noisy validation set, the auxiliary model in our method has access such a validation cases where the validation dataset has issues of class imbalance and label noise, there have been NAS methods (Chen al., 2020cc) Timofeev et al.] 2021) developed cope with these issues. Our framework can be combined with these NAS methods to learn tigh-performance architectures from class-imbalanced and noisy validation data.
well. In
We performed experiments to investigate how our method performs when the validation dataset has label noises. Following (Chen et al., 22200c) Kumar & Sastry (008)), we added symmetric label noise (with parameter 17 = 0.6) to validation data: the probability that one class is corrupted to be another class is equal. Training and test sets are clean. The experiments were conducted on CIFAR- 10. Training and validation losses cross-entropy. All methods perform architecture search using the same noisy validation dataset. Table shows test errors (%) on CIFAR-10. 10. As can be seen, our TETLoOdarrt2nd outperforms vanilla Darts2nd when validation data has label noises.
To combat label noise in validation dataset, we can combine our approach with robust NAS methods such (Chen et al., 2020c) which uses robust log losses (Kumar & Sastry 2018) training and validation losses. Table 6 shows the results (both TETLoO-drtt22d and Darts2nd use robust log loss). As be seen, TETLLO-drrtt2nd achieves better performance than vanilla Darts2nd. Compared with Table [6, we can see that using robust log loss achieves better performance than using cross-entropy loss.
We also performed experiments to investigate how our method performs when the validation dataset has class imbalance. The experiment was conducted on CIFAR-I0. In the original training, vali- dation, and test set of CIFAR-10, 10, the classes are balanced. Following (Timofeev et al.) 2021), we transformed the validation data examples for each class according to exponential function. Training and test sets are bal- anced. All methods perform architecture search using the same class-imbalanced validation dataset. Table 6 shows test errors (%). As can be seen, our TETLO-dartSnd outperforms vanilla Darts2nd when validation data has class imbalance.
CIFAR-10 10 into an imbalanced dataset by reducing the number of
To deal with class-imbalance in validation dataset, we combine our approach with class reweighting method proposed by (Cui et al. 2019). T shows the results (both TETLO-dart2nd and Darts2nd use this class reweighting method). As be seen, our TETLO-darts2 better performance than vanilla Darts2nd. Compared with Table 7 we can see that using class reweighting achieves better performance than not using class reweighting.
achieves
18

---

Published as a conference paper at ICLR 2023
Table 8: Test emrrr (%) CIFAR-10, when validation dataset is class-imbalanced and class
reweighting is used.
Method
Error( ())
Darts2nd (Liu et al. 2019a)
13.51=0.29
TETLO SETTLO-darts2nd (ours)
11.74++0.13
Table 9: Top-1 error (%) on ImageNet.
Method
Top-1 I error (%)
OFA w/o PS
27.6
OFA w/ PS
24.0
OFA w/ PS #25
23.6
OFA w/ PS #75
23.1
Ours- dartsand--diiiiiie
25.4
Ours- Ours-pdarts- -darts-cifar10
23.6
Ours- Ours-pdarts pdarts-cifar cifaar
24.1
Ours- pedarts-Imageeeee
23.3
B COMPARE OUR METHOD WITH OFA
OFA-large achieves the best performance because trains many sub-architectures progressively, from the largest sub-archieecture to gradually smaller sub-architectures. It takes 1200 GPU hours to train so many sub-architcctures. In contrast, our method trains a single searched sub-architccture method with other variants of OFA. As can be
which takes much less time. Table 9 compares seen, some variants of our method outperforms some variants of OFA.
Our method and OFA both perform knowledge transfer (from model A to model B). The difference is: in our method, model B (the auxiliary model) is leveraged to improve model A (the main model) by minimizing B's validation loss. In contrast, in OFA, model A is leveraged to improve model B.
C SEARCH COSTS UNDER DIFFERENT KNOWLEDGE TRANSFER APPROACHES.
Table 10 shows the search costs (GPU days) under different knowledge transfer (KT) approaches including QRS, pairwise similarity (PS, second order), and triple-wise similarity (TS, third order). The experiments were conducted CIFAR-100. As the number of orders increases (from 2 in PS, to in TS, to 4 in QRS), the search costs increase. But the increase is not substantial, As shown in Figure 4 in the main paper, QRS achieves much better classification performance than TS and PS. Overall, the gain in classification performance achieved by QRS over TS and PS outweighs the slightly increased search cost.
D ABLATION ON AUXILIARY MODELS.
We study how the performance of the main model is affected by different auxiliary models and perimented with ResNet- 18 (RN188 8), ResNet-50 (RN50), ResNet-101 (RN101), and simple CNN proposed (Vinyals et 2016), on DARTS, PCDARTS. and PDARTS. Table [T shows the re- sults. RN50 works better than RNI8. RN50 has more layers and is more expressive than RN18. This demonstrates that a more expressive auxiliary helps to train stronger main model. Intuitively, the predictive performance of a more expressive auxiliary is more sensitive to knowledge generated by the main model; consequently, minimizing the auxiliary's validation loss can more significantly change the fidelity of knowledge and further the predictive power of the main model. We pro- vide empirical evidence in Figure 6 by measuring the correlation between classification errors of auxiliaries and errors of QRS relationships. RN50's correlation is higher than RN18's. Other obser- vations include: 1) ResNet variants work better than the simple CNN: 2) ResNet-50 on par with ResNet- 101.
19

---

Published as a conference paper at ICLR 2023
Table 10: Search costs (GPU days) under different knowledge transfer approaches.
Search Method
KT Method
Search Cost
PS
3.8
Ours- Buns-Darts2nd
TS
3,9
QRS
4.0
0.4
0.3
Correlation
0.2
0.1
RN18
RN50
Figure 6: Correlation between classification errors of auxiliaries and errors of QRS relationships.
E APPLY TETLO TO DARTS WITH FIRST ORDER.
We apply TETLO method to DARTS with first order (DARTS-1s)) 1s) well. Table 12 and show results on CIFAR-100 and CIFAR-10. Our method outperforms vanilla DARTS-
F ARCHITECTURE TRANSFERABILITY BETWEEN CIFAR- 10 AND CIFAR-100.
Table 14 shows the results of architecture transferability between CIFAR-10 and CIFAR-100. 10
Cifar10 Cifart0->Ciffrr000 denotes that the architecture trained on CIFAR- 10 and evaluated on CIFAR-
100. 100frir0- denotes that the architecture is trained on CIFAR-I00 and evaluated on
CIFAR-10. As can be seen, our method has better architecture transferability between CIFAR-10 and CIFAR-100 than vanilla DARTS2nd and PDARTS, which further demonstrates the effectiveness of our lannsferbblity-encuraggg three-level optimization framework.
G MEASURE STABILITY
Following (Chen & Hsich 2020). we measure the stability of differentiable NAS methods on the WAS-Bench-IShenn (eela et al.) 2020) benchmark. NAS-Bench-IShhee has three search spaces for CIFAR-100. Following (Chen & Hsieh 2020), we perform architecture search for 100 epochs and measure the architectures searched different epochs on a test set. the average test accuracy for epoch 90-100 is larger than that for epoch 45-55, then the architecture considered to be degenerate. For each NAS method, we run for times and measure how many times the searched architecture degenerate. Table I5 shows the results. As can be seen, our 2adto-dart22nd has fewer degenerate times than Darts2nd, which demonstrates that our method is more stable than vanilla Darts2nd, thanks to its transferailliy--nccouaa mechanism.
20

---

Published as a conference paper at ICLR 2023
Table 11: Ablation auxiliaries.
Method
Error (%)
SCNN-darts2nd
18.06=0.11
RNI8-darts2nd
17.63=0.09
RN30-darts2nd
17.15=0.114
RN101 darts-d
17.32=00.08
SCNN-pdart
17.11=0.07
RN18 -darts
16.24=0.09
R R
16.02+00.06
RN101-pdars
15.97+0.10
Table 12:
Results of TETLO-darts on CIFAR-100, 100, including classification error (%) on the test set, number of model weights (millions), and search cost (GPU days).
Method
Error(%)
(M)
Cost
Darts st (Liu et al. 2019a)
20.49-H0.27
3.4
1.5
TETLO-darts (ours)
18.46:0.13
3.3
1.5
Table 13:
Results of TETLO-dartssst on CIFAR-10, including classification error (%) on the test set, number of model weights (millions), and search cost (GPU days).
Method
Error ())
Param(M)
Cost
Dartslst (Liu al. 2019a)
3.00++0.14
3.3
1.5
TETLO-darts 1st (ours)
2.80:10.09
3.1
1.5
Table Architecture transferability between CIFAR-10 and CIFAR-100. cation errors (%) on test sets.
The results are classifi-
Method
Cifarl0-:Cifar100
Effar100-Ciifal
Darts2nd (Liu et al. 2019a)
21.42++0.26
2.82+-0.07
"'ETIO-ddrrt/nn (ours)
18.05+0.114
2.09
Pdarts (Chen et al. 2019)
18.65=:0.10
2.67+0.009
TETLO-pdarts (ours)
16.82:t0.12
2.53:10.04
Table 15:
Times of architecture degeneration.
Method
Search space
Search space
Search space
Darts2nd (Liu et al. 2019a)
8
5
9
TETLO-dartse (ours) 2
2
21

---

Published as a conference paper at ICLR 2023
H OPTIMIZATION ALGORITHM
We use well-established algorithm developed (Liu et al.
2019a) to solve the proposed TETLO
problem. Theoretic convergence of this algorithm has been broadly analyzed in (Ghadimi & Wang 2018 Grazzi et al. 2020 Ji et al., 2021 Liu et al., 2021 Yang et al. (021). At each level of optimization problem, the optimal solution (on the left-hand side of the equal sign, marked with "), its exact value is computationally expensive to compute. To address this problem, following (Liu al. 2019ap. we approximate the optimal solution using a one-step gradient descent update and plug the approximation into the next level of optimization problem. In the sequel, denotes partial d- derivative. d. denotes ordinary derivative.
With architecture tying, the same as M. The optimization problem becomes:
Following (Liu et al. 2019a), we approximate E (M) using
dlm (M, Emg Hm, D'm!)
E = Em 5m
(5)
dem
We approximate (M) using
dl [M. Em Hms D
H = Hm
(6)
dHm
where 5m is a learning rate. Substituting E,, into the loss function at the second stage, obtain an approximated loss:
0 LA,, ) +2
max(O, - MCC Ea)
xyEd
(7)
Next, we approximate (MA)) and (A) using one-step gradient descent update of Eg and Ha w.r.t O:
E Eg 5. d0
(8)
dl d0
11 11, E. dH,
Finally, we plug these approximations into the validation loss at the third stage and get an approxi- mated validation loss P:
P= Lm Lm (M, H', ((rdd)) + (A, H ((vl)))
(10)
We can update the meta parameters M and A by descending the gradient of P w.r.t M and A:
M + M 17 dM A
dP
dP A 17 dA
(11)
where
dP
dLm [M, E H (vaf)
dL (A, E,, H', D (val)
(12)
dM
dM
dM
and
dL, H' 4 ((val)
dP
(13)
dA
dA
dlae EY bymit dM
can be calculated as:
dLm (M, E',, Hi (val)) dM
L (M, E", H² Suall)
dE', L (M, En H', (yal)) dM
dHL a (M, E',, H',, D(vaa))
2M
2E
dM
HHr
(14)
dlo (A.E. E. E,H,H,,, ( dMd
can be calculated as:
dLg (A,E,, H', (ral))
dE', (A, (ival)
(15)
dM
dM
aE
22

---

Published as a conference paper at ICLR 2023
where
dE
dE
dE dE
JVE 0
d dVE. 0
(16)
dM
dm
dM dE
dM
In Eq.
13),
dLgg dAg E,
Daally
can be calculated as:
dA
dLa (A, H. Du (nal)
aL (A, F., H (vul)
de,
aL (A, (A, Ey Da ((va))
dH
aL (A, H'g Dimll) (nal)
dA
dA
dA
dE
dA
(17)
In the above equations, we have:
dE
5m VM.
M,Emn Lmn (M, Em Hm: D
(18)
dM
dIl
5m V Hus Lo o (M, Em, Hm, (tr)
(19)
dM
JE -5 A.E, (A, Ha. )
JA
F. A,Eo
22 E ExyeD!m) max (0, - (c(x,, E'm - c(2, w: M) E(,
(20)
dH's
= - 5. La V,,, Ea, Ma,Di ((!))
(21)
dA
Similar to (Liu al. 2019a), approximate matrix-vector multiplication terms using finite differ- ences.
The gradient descent update of M and A in equation can updated, the one-step gradient-descent approximations (in Equation 8 which are functions of M and A, need to be re-updated. Then, the gradients of M and which are functions of one-step gradient-descent approximations, need to be re-calculated and are used to refresh M and A. In sum, the updates of M /A and the updates of one-step gradient-descent approximations mutually depend on each other. These updates are performed iteratively until convergence. Algorithm T shows the algorithm.
steps. After M and A
while not converged do
Update the approximation E,, of E (M) using Eq.
2. Update the approximation H", of H (M) using Eq. (6
Update the approximation E of E2 (M, A) using Eq.
4. Update the approximation H" of (A) using Eq.
Update architectures and A using [I.
end
Algorithm 1: Optimization algorithm for TETLO
Though using an approximated optimization algorithm, our framework is still valid tri-level frame- I6 18 the gradient the validation loss w.r.t to M involves the product of three gradients: gradient of the validation loss w.r.t to E gradient of E w.r.t Ey gra- dient of E', w.r.t M. The gradient information propagated from stage stage then stage 1. This shows that the tri-level optimization formulation of our framework is well preserved in the approximated algorithm.
work. As shown in equations 15
H.1 NUMBER OF GRADIENT DESCENT STEPS
In the above algorithm, at each level, we use one-step gradient descent (GD) to approximate the optimal solution. We are interested in investigating whether more (> 1) GD steps can result in better performance. We experimented with more GD steps at each level. The results are in Table [6 As can be seen, more GD steps do not differ significantly from one-step GD. This indicates that one-step gradient update yields sufficiently good approximation to the optimal solution.
23

---

Published as a conference paper at ICLR 2023
Method
Dataset
Num. GD steps
Test error (%)
TETLO-darts2nd (ours)
CIFAR100
1
17.21
TETLO-darts2nd (ours)
CIFARI00
17.16=0.13
TETLO-dartS2nd (ours)
CIFAR 100
3
17.11
TETLO-darts2nd (ours)
CIFARIO
1
2.62+00.06
TETL.O-dart22nd (ours)
CIFARIO
2.59+-0.07
TETLO-darts2nd (ours)
CIFARIO
2.57++0.06 3
TETLO-pdarts (ours)
CIFART00
16.05
TETLO-pdarts (ours)
CIFARI00
2
16.01
TETLO-pdarts (ours)
CIFARI00
3
15.98=0.007
TETLO-pdarts (ours)
CIFARIO
1
2.47
TETLO-pdarts (ours)
CIFARIO
2.47=0.04
TETLO-pdarts (ours)
CIFARIO
3
2.46+0.04
Table 16: Test errors under different numbers of GD steps.
Table 17: Top-1 and top-5 classification errors on ImageNet test set, number of parameters (mil- lions), and search costs (GPU days).
Method
Top-1
Top-5
#Parameters (M)
Search Costs (GPU days)
FBNet-A
27.0
4.3
9
FBNNttB
25.9
4.5
9
FBNet-C
25.2
5.5
9
OFA w/o
27.6
1.7
OFA w/ PS
24.0
1.7
OFA w/ PS #25
23.6
1.7
OFA w/ PS #75
23.1
1.7
OFA-large
20.0
1.7
dans--artsnd--cffrr 10
25.4
7.8
4.6
4.0
Ours- Ours-pdarts- -darts-cifar10
23.6
7.0
4.8
0.3
purs-pdarts-cifal100
24.1
7.3
5.1
0.3
Ours-pedarts- ImtageeNtt
23.3
5.3
4.0
COMPARE OUR METHOD WITH FBNET AND ONCE-FOR-ALL (OFA) NETWORKS
Table compares method with FBNet (Wu et al. 2019 and Once-for-All (OFA) (Cai al. 2020b) networks on ImageNet. The classification performance and search costs our methods are on par with those of OFA in general. The search costs of methods are lower than those of FBNet while our methods achieve classification performance similar to that of FBNet.
J NUMBER OF SKIP CONNECTIONS AND CONVOLUTIONS
Table 18 shows the number of skip connections and convolutions in searched architectures. Darts- Cifar10-Nonee denotes a normal cell searched on Cifar10 by Darts. Reduction denotes a reduction cell. The search space is that in DARTS. As be seen, in most cases, the number of skip nections in cells searched by our methods is smaller than that in baselines, which demonstrates the effectiveness of our methods in avoiding an excessive number of skip connections which causes performance degradation.
K SEARCH COSTS FOR ROBUSTNESS EXPERIMENTS ON S1-S4
Table 19 and 20 shows search costs for robustness experiments S1-S4 and CIF1R-10/100.
24

---

Published as a conference paper at ICLR 2023
Table 18: Number of skip connections, convolutions, and poolings
Cell
skip connections
convolutions
convolutions
poolings
Darts-Cifar 10-Norma
3
Sons-Dart--Ciff 10-Normal
2
Darts-Cifar (0-Reduction
3
3
0
2
Outs-Darts Sass-DDrts-Cffarr Sonts-CiffalO00 Cifar10-Redeen (0-Reduction
1
2
1
Darts Cifar NorNoaal
4
3
1
Ours-Darts-Cifal Maans-CiffaOO-NNrmmall 000NNrrml
2
6
0
Darts Cifar 100- Reduction
3
2
0
Ours- Dart-DDtts--aa Cww.raiaaaarrdiliia
1
1
0
Pedarts- 10- No-Moal
3
3
Ours- Pedarts-Cifarlln
2
5
Pedarts Citar 10- Reduction
2
4
0
Ours-Podarts
Ours Podarts Cifarl0-Ree 1-RReduciion
3
Pedarts- 100-Normal
Ours- Podarts-Cifarr CifarlOO-Noomaa
5
Pedarts- Cifar (00-Reduction
Mons-eddatt--iine C Reduction
Mdarts-CifarIO-Nooe
Ours-Pdarts-CCiffe 10-Normal
5
Pdarts-Cifar 1--eucttn
2
0
Murs-Pdarts-CCin Cifet Reduction
0
5
Edarts-Cifar1000 N0-m
5
0
2
Lans-Pdarts-Cifee (0-Norrmal
3
3
M 10-RReductoonnn
3
0
Burs-Pdarts-Ciing R0-Recucttoon
0
8
0
Prdarts-Cifan 1asss-CiaarOOOoonne
2
5
0
Ours- Burs-Prdarts- Cifisr 10-Normal
2
6
0
Prdarts Prdant--Cifrl10s Cifarl0-Reductionn
2
1
2
Ours Muns-Prdarts-Ciiie (iffarl0 10- (o-Reduction
0
3
2
3
3
Ours Pans-Prdarts-Ciie 100-Normal
4
3
3
0
Ours Ours-Prdarts- PerParrrrriieeeeee
0
3
2
Table 19: Search costs for $1-S4 on CIFAR-10. Search cost is measured by GPU hours Nvidia 1080Ti.
Method
DARTS
RDARTS-L2
6.2
9.1
10.4
8.8
DARTS-ES
10.1
18.3
15.1
16.7
DARTS-
6.0
8.9
10.5
8.9
SDARTS- ADVV
11.4
MTL.
10.6
Ours
6.1
8.7
10.3
8.7
Table 20: Search costs for S1-S4 on CIFAR-100. Search cost is measured by GPU hours on an Nvidia 1080Ti.
Method
S1
$2
S3
S4
DARTS
6.3
9.0
10.6
9,0
RDARTS-L2
6.0
9.3
10.7
8.6
DARTS-ES
10.4
17.9
14,8
16.5
DARTS-
10.7
SDARTS-ADV
7.8
11.6
13.8
11.3
MTL
6.3
9.2
10.5
9.1
Ours
6.2
9.0
10.5
8.8
25

---

Published as a conference paper at ICLR 2023
L
MEMORY COSTS
Table 21 compares memory costs of different methods. As can be seen, the memory costs of our methods are similar to those of baselines while our methods achieve lower test errors than baselines.
Method
Ptror-Cifar100
Cifor-C
Memory on Cifar100
Memory cost
on Cifar10
Darts2nd
20.48:1:0.4
2.76:10.09
11053
11008
TETLO-darts (ours)
17.21
2.62:10.06
11091
11049
Pdarts
17.42:+:.14
2.55+00.06
9659
9721
TETI.O-podarts (ours)
16.05-+0.08
2.47++0.03
9692
9745
Pedarts
17.9610.15
2.57=0.077
10058
10024
TETI.O-podarts (ours)
16.42+-0.07
2.48+00.66
10083
10066
Prdarts
16.48:10.06 15.81:0.066
2.3740.03
10159
10119
TETLO --pddrts (ours)
2.17:0.02
10192
10176
Table 21: Test errors (%) memory costs CIFAR-100 and CIFAR-10.
M EXPERIMENTAL DETAILS OF NEURAL ARCHITECTURE SEARCH
M. ADDITIONAL COMMON SETTINGS
In each iteration, we randomly sample a small minibatch (300) of pairs to calculate quadruple rel- ative similarity relations, instead of using all pairs. Calculation the small minimatch is very efficient. Following (Liu et al.) 2019a), we select the checkpoint at the final epoch to evaluate the test performance. No early stopping is involved. All GPU costs are measured 1080Ti GPUs.
M.2 DARTS BASED EXPERIMENTS
For methods based on DARTS2nd, including IETTaO-darts2nd (ours), MTLL-darts2nd, ES-darts2nd, ATT-darts2nd, the experimental settings are similar. In search spaces of DARTS, the candidate operations include: 3 x 3 and 5 X 5 separable convolutions, 3 x 3 and 5 x 5 dilated separable con- volutions, 3 3 max pooling. 3 3 average pooling, identity, and zero. The network
stack of multiple cells, each consisting of 7 nodes. The stride of all operations is set to 1. The convolved feature maps are padded to preserve their spatial resolution. The order for convolutional operations is BeLU-CCnv-BN. Each separable convolution is applied twice. The convolutional cell has 7 nodes. The output node is the depthwise concatenation of all intermediate nodes, excluding the input nodes. We create network by stacking cells. The first and second nodes of cell k are equal to the outputs of cell k - 2 and cell & - 1, respectively. Ixl convolutions are inserted when necessary. Reduction cells are located the 1/3 and 2/3 of the total depth of the network. In reduction cells, operations adjacent to the input nodes have a stride of 2.
For CIFAR-10 and 100, during architecture search, the main network model is stack of 8 cells, each consisting of 7 nodes, with the initial channel number set to 16. The search algorithm for 30 epochs with batch size of 32. Network weights are optimized using SGD, with initial learning rate of 0.025 (adjusted using cosine decay scheduler). momentum of 0.9, and a weight decay of 3c-4. The architecture variables were optimized using Adam (Kingma & Ba 2014) with a learning rate of 0.001, momentum of 0.999,, and weight decay of 0.001. The learning rate was scheduled with cosine scheduling. The architecture variables were initialized with zero initialization.
During architecture evaluation, for CIFAR-10 10 and CIFAR-100, larger network of the main model is formed by stacking 20 copies of the searched cell. The composed large network is trained on the combination of ((")) and D.vaa) The initial channel number was set to 36. We trained the network with a batch size of 96, an epoch number of 600. The SGD optimizer is used for weights training, with an initial learning rate of 0.025, a cosine decay scheduler, a batch size of 96, momentum of evaluate two types of architectures: 1) those searched on a subset of ImageNet; 2) those searched on CIFAR-10 10 or CIFAR- 100. either type, 14 copies of optimally searched cells stacked into large network, which trained the
0.9, and weight decay of 3e-4. On ImageNet,
26

---

Published as a conference paper at ICLR 2023
1.2M training images. with batch size of 1024, an epoch number of 250, an initial learning rate of 0.5, and a weight decay of 3e-5. Initial channel number was set to 48. Cutout, path dropout of probability 0.2 and auxiliary towers with weight 0.4 were applied.
M.3 PC-DARTS BASED EXPERIMENTS
For methods based on PC-DARTS, including TETLO-pedarts (ours), MTL-pcdarts, ES- ES--pdarrss ATT-pedarts, the experimental settings are similar. The search space of PC-DARTS follows that of DARTS. For architecture search on CIFAR-100 and CIFAR-IO, the hyperparameter K was set to 4. The network is a stack of 8 cells. Each cell contains nodes. Initial channel number is set to 16. The architecture variables are trained using the Adam optimizer for 50 epochs. The learning rate is set to 6e - without decay. The weight decay is set to le- 3. The momentum is set to 00.9,0.999). The network weight parameters are trained using SGD for 50 epochs. The initial learning rate is set to 0.1. Cosine scheduling is used to decay the learning rate, down to O without restart. The momen- tum is set 0.9. The weight decay is 3e 4. The batch size is set 256. Warm-up is utilized: in the first 15 epochs, architecture variables are frozen and only network weights are optimized.
The settings for architecture evaluation on CIFAR-100 and CIFAR-IO follow those of DARTS. 18 normal cells and 2 reduction cells are stacked into a large network. The imitial channel number is set to 36. The stacked network is trained from scratch using SGD for 600 epochs, with batch size 128, initial learning rate 0.025, momentum 0.9, weight decay 3e - 4, norm gradient clipping 5, drop-path rate 0.3, and cutout. The learning rate is decayed to using cosine scheduling without restart.
We combine our method and PC-DARTS to directly search for architectures on ImageNet. The stacked network starts with three convolution layers which reduce the input image resolution from 224x to 28 x28, using stride 2. After the three convolution layers, 6 normal cells and 2 reduction cells are stacked. Each cell consists of N = nodes. The sub-sampling rate was set to 0.5. The network trained for 50 epochs. Architecture variables are trained using Adam. The learning rate is fixed to 6e - 3. The weight decay is set to le-3. 3. The momentum is set to (0.5,0.999). In the first 35 epochs, architecture variables SGD. The initial learning rate is set to 0.5. is decayed to 0 using cosine scheduling without restart. Momentum is set to 0.9. Weight decay is set 3.-5. 5. The batch-size is 1024. Epoch number is set to 250. Eight Tesla V100 GPUs were used.
frozen. Network weight parameters
trained using
For architecture evaluation on ImageNet, the stacked network starts with three convolution layers which reduce the imput image resolution from 224x224 to 28x28, using stride 2. After the three convolution layers. 12 normal cells and 2 reduction cells are stacked. Initial channel number is set to 48. The network is trained from scratch using SGD for 250 epochs, with batch size 1024, initial learning rate 0.5, weight decay 3e- 5, and momentum 0.9. For the first 5 epochs, learning rate warm-up is used. The learning rate is linearly decayed to 0. Label smoothing and auxiliary loss tower used.
M.4 P-DARTS BASED EXPERIMENTS
The search process has three stages. At the first stage, the search space and stacked network in P-DARTS are mostly the same as DARTS. The only difference is the number of cells in the stacked network in P-DARTS is set to 5. At the second stage, the number of cells in the stacked network is 11. At the third stage, the cell number is 17. At stage 1, 2, 3, the initial Dropout probability on skip- connect O, 0.4. and 0.7 for CIFAR- 10, 0.1, 0.2, and 0.3 for CIFAR- 100; the size of operation space is 8, 5, 3, respectively. The final searched cell is limited to have 2 skip-connect operations at maximum. At each stage, the network is trained using the Adam optimizer for 25 epochs. The batch size is set to 96. The learning rate is set to 6c-4. Weight decay is set to le-3, Momentum is set to 09.5,..999). In the first 10 epochs, architecture variables are frozen and only network weights are optimized.
For architecture evaluation on CIFAR-100 and CIFAR-10, 10, the stacked network consists of 20 cells. The initial channel number is set to 36. The network is trained from scratch using SGD. The epoch number is set to 600. The batch size is set to 128. The initial learning rate is set to 0.025. The learning rate is decayed to O using cosine scheduling. Weight decay is set to 3 for CIFAR-10
27

---

Published as a conference paper at ICLR 2023
and 5e-4 for CIFAR-100. Momentum is set to 0.9. Drop-path probability is set to 0.3. Cutout regularization length is set to 16. Auxiliary towers of weight 0.4 are used.
For architecture evaluation consists of 14 cells. The initial channel number is set to 48. The network is trained from scratch using SGD for 250 epochs. Batch size is set 1024. Initial learning rate is set to 0.5. The learning rate is linearly decayed after each epoch. In the first 5 epochs, learning rate warmup is used. The momentum is to 0.9. The weight decay set to 3e 5, Label smoothing and auxiliary loss tower are used during training. The network was trained on 8 Nvidia Tesla V100 GPUs.
ImageNet, the settings similar to those of DARTS. The network
M.5 PR-DARTS BASED EXPERIMENTS
The operations include: 3x3 and 5x5 separable convolutions, 3x3 and 5x5 dilated separable con- volutions, 3x3 average pooling and 3x3 max pooling, zero, and skip connection. The stacked network consists of k cells. The k/3- and 2k/3-th cells are reduction cells. In reduction cells, all operations have stride of two. The rest cells are normal cells. Operations in normal cells have stride of one. Cells of the same type (either reduction or normal) have the same architecture. The inputs of each cell are the outputs of two previous cells. Each cell contains four intermediate nodes and one output node. The output node is a concatenation of all intermediate nodes.
For architecture search on CIFAR- 100 and CIFAR-10 10, the stacked network consists of 8 cells. The initial channel number is set to 16. In PR- DARTS, 21, 22, and 2 are set to 0.01, 0.005, and 0.005 respectively. The network was trained for 200 epochs. The mini-batch size is set to 128. Architecture variables are trained using Adam. The learning rate is set to 3e 4. The weight decay is set to le-3. Network weights are trained using SGD. The initial learning rate is set to 0.025. The momentum is set to 0.9. The weight decay is set to 3e 4. The learning rate is decayed to 0 using cosine scheduling. For acceleration, per itcration, only two operations on each edge are randomly selected to update. The temperature t is set to 10 and is linearly reduced to 0.1: a = -0.1 and b 1.1. Pruning on each node is conducted by comparing the gate activation probabilities of all non-zero operations collected from all previous nodes and retaining top two operations.
For architecture evaluation on CIFAR1O and CIFAR 100, the stacked network consists of 18 normal cells and 2 reduction cells. The initial channel number is set to 36. The network is trained from scratch using SGD. The mini- mini-batch size is set to 128. The epoch number is set to 600. The initial learning rate is set to 0.025. The momentum is set to 0.9, The weight decay is set to 3e 4. The gradient norm clipping is set to 5. The drop-path probability is set to 0.2. The cutout length is set to 16. The learning rate decayed to O using cosine scheduling.
For architecture evaluation on ImageNet, the input images are resized to 224x224. The stacked network consists of 3 convolutional layers, 12 normal cells, and 2 reduction cells. The channel number is set to 48. The network is trained using SGD for 250 epochs. The batch size is set to 128. The learning rate is set 0.025. The momentum set to 0.9. The weight decay set to 3e 4. The gradient norm clipping is set to 5. The learning rate is decayed to 0 via cosine scheduling.
M.6 HYPERPARAMETER TUNING STRATEGY
To tune the hyperparameters à and we randomly sample 2.5K data from the 25K training set and sample 2.5K data from the 25K validation set. Then we use the 5K sampled data as a hyperparameter tuning set. 2 and are tuned in (0.01, O. 0.2, 0.5. 1,2). For each configuration of 2 and we use the remaining 22.5K training data and 22.5K validation data to perform architecture search and use their combination to perform architecture evaluation (retraining a larger stacked network from scratch). Then we measure the performance of the stacked network on the 5K sampled data. and y yielding the best performance on the 5K sampled data selected. For other hyperparameters, they mostly follow those in DARTS (Liu et al. 2019a). P-DARTS (Chen et al. 2019). PC-DARTS (Xu et al. 2020), and PR-DARTS ZZhou et al. 2020b).
28

---

Published as a conference paper at ICLR 2023
M.7 IMPLEMENTATION DETAILS
We use PyTorch to implement all models. The version of Torch is 1.4.0 (or above). We build LPPDT upon official python packages for different differentiable search approaches, such as "DARTS "P-DARTS"" and "PC-DARTS?".
N ADDITIONAL ABLATION STUDIES
We perform the following additional ablation study (denoted by NoAux): ignore the auxiliary task entirely, define quadruple relative similarity (QRS) loss the validation data, add this loss the validation loss of the main model, and use the added loss to optimize the architecture.
The results in Table 22 an important contributor to the effectiveness of our method.
Our method outperforms NoAux. This shows that the auxiliary task is
Method
Test error on CIFARIO0
Test error on CIFARIO
Dort22d
09.96
2.88+ 2.62
Ours+Darts2nd
17.21=10.10
PdAut+Paarts
19.62+00.06
2.60=E0.04
Ours+PParts
16.05.10.08
2.47:10.03
Table 22: Results of the NoAux ablation study.
O EXPERIMENTAL DETAILS OF RESULTS IN FIGURE 2 IN THE MAIN PAPER
First, we create collection C of quadruples which have correct relative similarity relationships. We randomly sample two data instances x and y from the same class and randomly sample two data instances z and w from different classes. Since and are from the same class, their similarity is considered being larger than that between z and w which are from different classes. Then we put (x,y) > (z, w) into C. This procedure repeats 10K times. In the end, C contains 10K quadruples. Then from C, we create another three collections of quadruples: C25, Cso, and C7s. with increasing levels of incorrect quadruple relative similarity relationships. To get C2s, randomly sample 25% quadruples from C; then for each sampled quadruple (x.y) > (z, w). we switch the order of these two pairs and get (z) > (x, which an incorrect quadruple. In C25. 25% quadruples have incorrect relative similarities. C5o and C75 are obtained in similar ways.
Given C25, we it train auxiliary model, by solving the following problem:
MInA (A E (A), (*) DA)
s.f. Eg (A), argmin La(A, E,, H4, Ha, 2 A (Lm)) max -0 -c((x,x;A, (.2,,,,,E..))
(22)
Then measure the classification error of this trained auxiliary model the validation set D (val) of the auxiliary. Similarly, we do this for Cso and C7s- The classification errors are shown as the blue eurve in Figure i the main paper.
Next, we discuss how to obtain the orange curve. Given C25, we use it to train a main model, by solving the following problem:
minM E(cy) max (0, 1,00--f0 M, F clz. w; A.M.F. F (M)))
(23)
S.F. (M), HM) argming,,,11 Lm (M, Em Hm. D
Then we measure the classification error of this trained main model on the validation set (val) of the main model. Similarly, we do this for Cso and C75- The classification errors are shown as the orange curve in Figure 2 in the main paper.
l hntps//ggitubb.oo/guakkkko 2 hhaps://gtthub.ccm http://hwwh...oo//cccceeenn/llaa 3 Pttp:///ghhhacmmmyyyiiii//////PP
29

---

Published as a conference paper at ICLR 2023
P EXPERIMENTAL DETAILS OF RESULTS IN FIGURE 2 IN THE MAIN PAPER
In the pseudo-labellig (PS) baseline, the main model and auxiliary model predict class probabilities (pseudo labels) the same images, then KL divergence is defined on these probabilities and the auxiliary model is trained by minimizing the KL divergence. The setup follows equation in (Zhang al. 2018).
For the pairwise similarity (PS) baseline, the formalism is: given two images, the main model and auxiliary model predict the probabilities regarding whether these images are similar, then cross-entropy loss is defined on these two probabilities; the auxiliary model is trained by minimizing this cross-entropy loss.
For the triple- piple-wise similarity (TS) baseline, given three data examplesx, y. z. the main model predicts whether is more close y or to z, the auxiliary model trained by minimizing a hinge loss that is consistent with the prediction of the main model on this triple. The formalism follows equation 6 (You et al. 2017).
Q EXPERIMENTAL DETAILS OF EVALUATING ROBUSTNESS AGAINST OVERFITTING
The four search spaces $1 - S4 are designed by (Zela et al. 2020
S1: In this search space, each edge has only two candidate operations. To identify these operations, operations that have the least importance in the original search space of DARTS are iteratively removed.
S2: For each edge, the candidate operations are 3 3 SepConv and SkipConnect.
$3: For each edge, the candidate operations are: x SepConv, SkipConnect, and Zero.
S4: For each edge, the candidate operations are: 3 3x3 SepConv and Noise. In the Noise op- eration, every value from the input feature map is replaced with random variables sampled from univariate Gaussian distribution.
R SIGNIFICANCE TEST RESULTS
To check whether the performance of our proposed methods are sigmificantly better than baselines, we perform statistical significance test using double-sided T-test. We use the function in the python package "scipy.stats, ttest Isamp" and report the average results over 10 different runs. Ta- ble 23 and shwshow the results.
From these two tables, we can see that the p-values are small between baselines methods and our methods, which demonstrate that the enrors of our methods are significantly lower than those of baselines.
MODEL PARAMETERS, SEARCH COSTS, AND FLOPS ON IMAGENET
Table 25 shows the number of model parameters, search costs, and FLOPSs on ImageNet. The parameter numbers, search costs, and FLOPPs of our methods are close to those in differentiable basclines.
T FULL LISTS OF HYPERPARAMETER SETTINGS IN NAS EXPERIMENTS
The hyperparameter settings for the auxiliary model such as the optimizer, momentum, weight de- cay, learning rate, number of layers were the same those of the original DARTS and PC-DARTS implementations. The hyperparameter settings for the main model were also set to the same values those of the auxiliary model. Tables 26 29 show the hyperparameter settings used in architecture search experiments. Tables 33 show the hyperparameter settings used in architecture evaluation experiments.
30

---

Published as a conference paper at ICLR 2023
Our method
Baseline
p-value
2nd
MTL-darts2nd
9.31e-5
TETLO-darts2ng
ES-darts2nd
2.66e-8
ATT-darts2nd
7.05e-9
TETLO--drrt2nd
Darts2nd
4.49e-11
TETLO-pdase -daarts
MTL-pdarts
3,92e-5
TETL.--pdart
ES-
6.30e-6
TET1.O-pd -at.---pparts
ATT-ppdarts
7.19e-6
TETLO- -0.00--pdrtss
Pdarts
4.82c-7
FETTLO-pcdart
MTL-pcdarts
5.38e-6
TETLO-pcdart.
ES-pedarts
7.21e-6
TETLO-pedarts
ATT-pcdarts
4.63e-6
TETLO- TETLO-pedarts
Pedarts
8.31e-7
TETLO
TETLO-prdarrt --prdarts
MTL--pddats
1.65e-7
TETLO -ETLO-prdarts
ES- ES-prdarts
3,07c-6
NETLO-prdarte
ATT-prdarts
4.77c-7
TETLO- O--rrdattssss
Prdarts
5.28e-6
Table 23: Significance test results on CIFAR-100
Our method
Baseline
p-value
MTL-darts2nd
5.02e-9
TETLO-darts2ne
ES-darts2nd
7.39e-10
TErttO-d
ATT-darts2nd
6.20e-10
TETItO-dart22nd
Darts2nd
3.51e-10
TETLO-pdarrs
MTL-pdaarss
4.09e-7
TETLO-pde -ddarts
ES ES-pdarts
6.83e-8
TETLO- --pdartsssssss
ATT-pdarts
8.04e-8
TETLO-pon --darts
Pdarts
6.37e-7
TETLO -ecdarts
MTL-pcdarts
7.26e-10
TETL.--pcdan -pelats
ES-pedarts
3.51e-10
TETI-O-pedarts
ATT-ppedrrs
5.05e-10
TETLO-pon -pddartss
Pedarts
6.82e-10
TETLO -at.----rdrtss
MTL-prdarts
6.93e-11
TETLO- TETLO-prdarts
ES- ES-prdarts
3,85e-12
TETLO-prdant
ATT-prdarts
7.22e-7
TETLO- O-rdarrtsssss
Prdarts
3.59e-5
Table Significance test results on CIFAR-10
31

---

Published as a conference paper at ICLR 2023
Table 25: Top-1 and top-5 classification errois on ImageNet test set, number model weights (millions), and search cost (GPU days), and FLOPss (M). TIFTLO-dart22nd--550-iciie architecture searched by applying TETL. (with ResNet-50 as main encoder) to DARTS-2nt on CIFAR-10. Similar meanings hold for other notations having such format. Results marked with
denotes
are taken from DARTS (Chu et al.) (2200a) and DrNAS (Chen et al., 2020b). The other notations
are the same as those Table in the main paper.
Top-1 Error (%)
Top-5 Error (%)
Param (M)
Cost
FLOPs
Method
(GPU days)
(M)
*Inception-v vl Szegedy et al.
2015
30.2
10.1
6.6
1448
"ShuffleNet x (v2) [Ma
2018
25.1
7.6
7.4
299
*NASNet-A (Zoph al. 2018)
26.0
5.3
1800
564
*AmoebaNet-C (Real el al. 2019)
24.3
7.6
6.4
3150
570
"SDARTTS-AD--CIFAR" (0 (Chen & Hsich 2020)
25.2
7.8
5.4
1.3
*PC-DARTS- (Xu ct al. 2020)
25.1
7.8
5.3
0.1
586
*ProxylesNNAS-Imageet (Car Caree 2019
8.3
465
*FarDARRTS-ImageNe (Chu et al.
2019
24.4
7.4
4.3
3.0
440
*DARTS* -CIFARICIOR (Liang
2019)
23.7
7.2
5.1
0.2
591
DDrts2nd-cifar10 (Liu et al.
2019a)
26.7
8.7
4.7
4.0
574
ATT-darts2nd-cffarIt
26.3
8.2
4.8
4.4
539
ES ES-dart22nd-cifar10
4.0
582
MTL--arts2nd-cifa110
26.4
8.3
4.7
4.2
546
TETLO-darts2nd--an (ours)
25,4
7.8
4.6
4.0
551
*Pdarts-cifar 10 (Chen al. 2019)
24.4
7.4
4.9
0.3
557
ATT-puarts-cifarIU
4.9
0.5
583
ES-pdarts-cifar10
24.4
7.3
5.0
0.3
526
MTL-pdarts-cifall
24.3
7.2
4.8
0,4
559
TETLO TETLO-pdarts cifarriiill0 10 (ours)
23.6
7.0
4.8
0.3
538
Pdarts-cifar 100 (Chen et al. 2019
24.7
0.3
577
ATT-pdarts-cifarrt
24.6
7.5
5.1
0,5
549
ES ES-pdarts- pdarts-ccfarrl00 100
24.6
7.5
5.3
0,3
552
MTL-pdarts-cifann 100
24.7
7.7
5.1
0.4
561
TETLO- PrrT----aats-cfffrrt (ours)
0.3
531
"Pedarts-imagee Micaarts-inageeet Xuu et al. 2020
24.2
7.3
5,3
3,8
597
ATT-pedarts-imageee
24.0
7.1
5.5
4.3
583
ES pedarts-imagenet
24.1
7.2
5.2
3.8
542
MTL MTL--edarts--magee
23.9
7.3
5.5
3.9
566
TETLO- SeTTOO-pedart--mmageee (ours)
23.3
6.2
5,3
4.0
574
Name
Value
Optimizer
SGD
Learning rate of architecture variables
3e-4
Epochs
30
Weight decay architecture variables
le-3
Batch size
32
Drop path probability
3e-1
Initial channels
16
Learning rate of network weights
2.5e-2
Number of layers
8
Momentum
9e-1
Weight decay for network weights
3e-4
Gradient clip
5
Lambda
0.1
Train portion
0.5
Unrolled
True
Cutout
False
Table 26: Hyperparameter settings of TETLO method when applied to DARTS, during archi- tecture search on CIFAR-10 and CIFAR-100.
32

---

Published as a conference paper at ICLR 2023
Name
Value
Optimizer
SGD
Learning rate of architecture variables
6c-4
Epochs
50
Weight decay of architecture variables
le-3
Batch size
96
Drop path probability
3e-1
Initial channels
16
Learning rate of network weights
le-1
Number of layers
8
Momentum
9e-
Weight decay for network weights
3e-4
Grad clip
5
Lambda
0.5
Train portion
0.5
Unrolled
True
Cutout
False
Table 27: Hyperparameter settings of our TETLO method when applied to PC-DARTS, during architecture search on CIFAR-10 and CIFAR-100.
Name
Value
Optimizer
SGD
Learning rate of architecture variables
3e-4
Epochs
30
Weight decay of architecture variables
le-3
Batch size
32
Drop path probability
3e-1
Initial channels
16
Learning rate network weights
2.5e-2
Number of layers
8
Momentum
9e-1
Weight decay for network weights
3e-4
Gradient clip
5
Lambda
0.1
Train portion
0.5
Unrolled
True
Cutout
False
Table 28: Hyperparameter settings of our TETLO method when applied to PDARTS, during archi- tecture search CIFAR- 10 and CIFAR-100.
33

---

Published as a conference paper at ICLR 2023
Name
Value
Optimizer
SGD
Learning rate of architecture variables
6e-4
Epochs
50
Weight decay of architecture variables
le-3
Batch size
96
Drop path probability
3e-1 l
Initial channels
16
Learning rate of network weights
le-1
Number of layers
8
Momentum
9e-1
Weight decay for network weights
3e-4
Grad clip
5
Lambda
0.5
Train portion
0.5
Unrolled
True
Cutout
False
Table 29: Hyperparameter settings our TETLO method when applied to PR-DARTS, during architecture search on CIFAR-10 and CIFAR-100.
Name
Value
Optimizer
SGD
Learning rate
2.5e-2
Epochs
600
Weight decay
3e-4
Batch size
96
Momentum
9c-1
Initial channels
36
Number of layers
20
Auxiliary
True
Auxiliary weight
0.4
Cutout
True
Cutout length
16
Drop path probability
0.2
Gradient clip
5
Table 30: Hyperparameter settings of our TETLO method when applied to DARTS, during archi- tecture evaluation CIFAR-10 10 and CIFAR-100.
Name
Value SGD
Optimizer
Learning rate
2.5e-2
Epochs
600
Weight decay
3e-4
Batch size
96
Momentum
9e-1
Initial channels
36
Number of layers
20
Auxiliary
True
Auxiliary weight
0.4
Cutout
True
Cutout length
16
Drop path probability
0.3
Gradient clip
5
Table 31: Hyperparameter settings of our TETLO method when applied to PC-DARTS, during architecture evaluation on CIFAR-10 and CIFAR-100.
34

---

Published as a conference paper at ICLR 2023
Name
Value
Optimizer
SGD
Learning rate
2.5e-2
Epochs
600
Weight decay
3e-4
Batch size
96
Momentum
9e-1
Initial channels
36
Number of layers
20
Auxiliary
True
Auxiliary weight
0.4
Cutout
True
Cutout length
16
Drop path probability
0.2
Gradient clip
5
Table 32: Hyperparameter settings of our TETLO method when applied to PDARTS, during archi- tecture evaluation on CIFAR-10 and CIFAR-100.
Name
Value
Optimizer
SGD
Learning rate
2.5e-2
Epochs
600
Weight decay
3e-4
Batch size
96
Momentum
9e-1
Initial channels
36
Number of layers
20
Auxiliary
True
Auxiliary weight
0.4
Cutout
True
Cutout length
16
Drop path probability
0.3
Gradient clip
5
Table 33: Hyperparameter settings of our TETLO method when applied to PR-DARTS, during architecture evaluation on CIFAR-10 and CIFAR-100.
35

---

Published as a conference paper at ICLR 2023
U VISUALIZATION OF SEARCHED ARCHITECTURES
Using the graphviz package, we plot the searched architecture cells in all the experiments. Each architecture consists of normal cell and reduction cell. Figure 7 show the cells searched on CIFAR-10 10 and CIFAR-I00 by our TETLO methods.
sep_conv_31t
skin_connect
conv 3x3
(kkk2)
sep cep_conv_oo _333_3x3 __3x3
(k)
sep_conv
[_k-l] }
connect
sep conv _x3
dil conv conv_3x3
<kip connect
____ppoo__x3
c_(k-2]
max paol__oo_3x
miax_pooo 3x3
2
max__ooo__x3
sep_conv 3x3
avg_pool_ 3x3
(K)
C_ 1)
avg_pool_3xx
3x3
Figure 7: Normal cell (top) and reduction cell (bottom) searched by TETLO-DARTS CIFAR- 100.
36

---

Published as a conference paper at ICLR 2023
sep_cont 3x3
skip_connect
sep_conv _x3 _
c_(k-2)
sep_comt _x3
3
C c_ (k)
dil_conv_550 _x_
2
c_(k-1
skip_connect
sep_conv 3x3 333
sep_conv_ _55
ovg pool 3x3
avg_pool px3 3x3
conv 3x3
dil, dil_conv_33g 3x33
avg pool 3x3
<_kk-
sep 5x5
skip, cunnect
Figure Normal cell (top) and reduction cell (down) searched by TETLO-DARTS on CIFAR-10.
oote 303
spp
2o0. 3x]
d4p, connect
(x-3)
3x3
E4-23
Figure 9: Normal cell (top) and reduction cell (bottom) searched by TETLO-PDARTS CIFAR- 100.
37

---

Published as a conference paper at ICLR 2023
11-33
upp
Jee 201
(k-2
pOL 3x3
dil conyoooyy_a 5x5
di conv 5x5
(I-1)
5x5
5x5
di conv, 5x5
Figure 10: Normal cell (top) and reduction cell (bottom) searched by TETLO-PDARTS on CIFAR- 10.
sime 3G
sep_comy, S65
Sep CONY, Se5
ppprree 203
c.14
Figure 11: Normal cell (top) and reduction cell (bottom) searched by TETLO-PCDARTS on CIFAR- 100.
cp mex
CX
Figure 12: Normal cell (left) and reduction cell (right) searched by on CIFAR- 10.
Lo3
1k:2]
50g cony. 5X5
ssp_coBv 313
000. 3x3
rep Se5
sep_ 5X3
[QL
cony 3x3
<4g_cotv Rx5
sep_com__55 553
stp_coav_303
Figure 13: Normal cell (left) and reduction cell (right) searched by TETLO-PRDARTS CIFAR- 100.
<Ml connect
vy.
Figure 14: Normal cell (left) and reduction cell (right) searched by TETLOOPRDARTSS CIFAR- 10.
38

---

Published as a conference paper at ICLR 2023
V FORMULATIONS OF BASELINES
For multi-task learning (MTL), the formulation is:
minMA Lm L,,,, (M. (M.A). H. (M). (val)
(M ((
S.f.
Em (M.A). (M) E M, MM = -agg=arggn......mm FM (A,
Lo,
(M, Em, Hm. Dm L syenn max(0, (cm (f,
(24)
For encoder sharing (ES), the formulation is:
minA Lm (M, E ( (M), HMM,HHH. H., (M), DD (M, H Da
(25)
s.s. E (M), H. (M). Hr, (M) = argming H. (M, E, Da Lon LM E,,,Hmmmll D(!r)
where E is the shared encoder.
W ASSUMPTION ON TRANSFER LEARNING
Since our proposed transfer learning method based on predicting quadruple relative similarities, the assumption is that the relative similarities between data examples are unchanged between do- mains. In addition, we assume the input data modality and the output label space are uncharged between domains.
We expect the main and auxiliary tasks to be similar, to ensure the knowledge can be successfully transferred from the main model to the auxiliary model. If the two tasks are largely dissimilar, the auxiliary may not able to help the main model to improve.
X ADDITIONAL RELATED WORKS
Bi-level optimization. Our work is based on tri-level optimization, which is related to bi-level op- timization (BLO). BLO (Dempe 2002) has been broadly applied for hyperparameter tuning (Feurer ctt al. 2015). neural architecture search (Liu et al. 2019a). meta learning (Finn et al., 2017). data reweighting Shu al. 2019 Ren al. 2020 Wang al. 2020). learning rate adjustment (Baydin) (et al.) 2017). label denoising Zheng et al. 22099). data generation (Such et al.] 2019). In these meth- ods, meta parameters (e.g, hyperparameterss, neural architectures, data weights, etc.) are optimized by minimining validation losses and model weights are learned by minimizing training losses.
Multi-task learning (MTL). MTL (Ruder 2017: Zhang & Yang 2021 aims to improve multiple tasks simultaneously by learning jointyy and transferring knowledge across tasks. Various MTL
approaches have been proposed, based on 1) hard parameter sharing (Caruana 1998 Long et al.,
2015 Doersch & Zisserman 2017; Kokkinos 2017, Sener & Koltun 2018 Leang et al. 2020), where multiple tasks share the same weight parameters, such as encoder weights; 2) soft parameter sharing DDong al. 2015 Dai al. 2016 Misra al. 2016 Yang & Hospedales 2016 Lu et al. 2017 Liu et al. 2019cc Maninis et al. (019). where parameters of different tasks are constrained
to be similar; 3) task similarity learning (Williams al.) 2007 Zhang & Yeung 2014 Bingel & Søgaard 2017 Standley et al., 2020) Zamir et al. 2020). which identifies similarity between tasks and encourages similar tasks share more commonalities: 4) loss weighting (Chen et al. 2018
Kendall et al., [2018] /Snner & Koltun 2018 Gong et al., 2019 Leang et al. 2020). which weighs each task's loss. Our work is also related to meta learning, which learns transfer meta knowledge to end tasks (Lee & Choi 2018 Yao et al.) 2020 Hospedales et al.] 2020 Andrychowicz et al.)
2016
Li& Malik 2016
Ravi & Larochelle, 2016 Finn al. 2017 al.
2017 Antonint al.
2018
Mishra et al. 2017
Qiao et al. 2018 Gidaris & Komodakis 2018 Lekkala & Itti 2020 Sun
et al. 2020).
Neural architecture search (NAS). NAS aims automatically identify highly-performing
chitectures of deep neural networks instead of manually designing them by humans. Various ap- proaches have been proposed for NAS, including differentiable search methods (Cai et al. 2019
39

---

Published as a conference paper at ICLR 2023
Liu et al. 2019a Xie et al.] 2019) and those based on reinforcement learning Zoph & Le 2017 Pham et al. 2018: Zoph et al,, 2018) and evolutionary algorithms (Liu et al. 2018b Real et al. 2019). In RL-based approaches, policy is learned to iteratively generate new architectures by maximizing reward which is the accuracy on the validation set. Evolutionary algorithm based ap- proaches represent architectures individuals in population. Individuals with high fitness scores (validation accuracy) have the privilege to generate offispring, which replaces individuals with low fitness scores. Differentiable approaches adopt a network pruning strategy. On top of an over- parameterized network, importance weights of building blocks are learned using gradient descent. After learning, blocks whose weights are close to zero are pruned. There have been many efforts devoted to improving differentiable NAS methods. In P-DARTS (Chen et al., 2019), the depth of searched architectures allowed to grow progressively during the training process. Search space approximation and regularization approaches are developed to reduce computational overheads and improve search stability. PC-DARTS (Xu et al. 2020) reduces the redundancy in exploring the search space by sampling small portion of a super network. Operation search is performed in subset of channels with the held-out part bypassed in a shortcut.
40


Reviews:
[{"review_id": "zoJs2aeeyN", "reviewer": "Reviewer_3zEW", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a method to improve the stability of differentiable NAS. It can improve stability and performance without introducing too much cost.\n\nThe major contributions of this work include:\n+ We propose a transferability-encouraging tri-level optimization framework to improve\nthe generalizability and stability of differentiable NAS methods.\n+ We propose a new knowledge transfer approach based on matching quadruple relative similarities.\n+ We perform various experiments which demonstrate the effectiveness of our method.\n", "summary_of_the_review": "I think this is an informative article. The solution is more reasonable, and a large number of experiments are used to support the conclusions in the text.", "strength_and_weaknesses": "## Strengths\n1. Although the content of the article is a bit large, it is generally organized and the article is easy to understand. The formulas are also deduced in detail.\n2. The experimental part is solid and has achieved good performance.\n3. The way to solve differentiable NAS by introducing an auxiliary task is innovative.\n\n## Weaknesses\nThere are two points I don't quite understand:\n1. Why use a method based on matching quadruple relative similarities for knowledge transfer? What other options are there to do this, and why is this way better than others? I don't think the insight behind this is sufficient.\n2. In this paper, the stability of the method is proved by the experiments in Table 8. The stability I understand should be the frequency of network structure degradation in a large number of experiments, not the number of skip-connections in the network. I don't know if there is a more reasonable way to prove this.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "o1DGV49jgS9", "reviewer": "Reviewer_8AHX", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors tackle the current limitation that most existing NAS methods suffer from, which are unsatisfactory generalizability and stability, such as generating a dominant number of skip connections or poor test performance. To address this, the authors propose a transferability-encouraging tri-level optimization framework for improving the main model and auxiliary models. They demonstrate the effectiveness of their methods. ", "summary_of_the_review": "I enjoyed reading the paper, but several improvements seem to be required.", "strength_and_weaknesses": "**Strength**\n- The paper is easy to read.\n- They tackle the practical limitations of neural architecture search.\n- Experiments are extensive.\n\n**Weaknesses**\n- My main concern for this paper is the quality of the validation sets, which play a key role in measuring “transferability” from the main model to the auxiliary models. The poor representativeness of validation sets (class-imbalanced, noise, etc) for the main model and auxiliary model may affect the optimization process (step 3) and this should be clearly analyzed. What if we cannot get the class-balanced and non-noisy validation set in a practical scenario? The proposed method with a poor validation set still outperforms the baseline models without any degeneration? If not, are there any possible methods to handle this issue?\n- For one of the baseline models, OFA-large (Table 4) seems to show the best performance, but I was not able to find any explanation for it. As far as I know, OFA can sample subnets from the supernet while keeping (or transferring) the supernet’s pretrained knowledge, which is similar to the proposed method. It would be good to compare the proposed method with OFA more closely. \n- (minor) It would be helpful if the overview illustration for the proposed method is provided. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Teyzo0cRFe", "reviewer": "Reviewer_5nMq", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The author adopts knowledge transfer to solve the inhabited stability and generalization issue of differential NAS. The method first trains the weights of the main model with alpha fixed, then transfer the knowledge from main to auxiliary by quadruple relative similarities, and finally optimizes the alpha of the main model by optimizing the transferability to the auxiliary.\nThe intuition behind this is that, once the architecture equips with a good architecture that can extract useful knowledge, the obtained knowledge is accurate and good enough to transfer to the auxiliary one, and the auxiliary reaches high quality.", "summary_of_the_review": "Overall, this is a good paper. The writing is clear and the evaluation is good enough. The results show its effacy. \n\n# Update\n\nSince my questions are well answered, I have no further comments and raise my score to 8.\n\n", "strength_and_weaknesses": "# Strength\n- S1.The paper solves the stability issues of DARTS from a  different aspect, transfer learning view, which is somehow novel and interesting. The motivation is reasonable for me.\n- S2.The paper is well-written and easy to follow. \n- S3.The evaluations are sufficient to prove the validity of the proposed methods.\n- S4.The results are promising on C10, C100, and ImageNet.\n\n# Weaknesses\n- Q1.What are the computation complexity and extra cost when applying QRS compared with second-            order or third-order? Intuitively, when increasing the order, the training cost may increase either. Ablations about this should be added.\n- Q2.In Page 5, the author claims that the auxiliary model ( encoder) is the same as the main model which is a DARTS. It is clear that the main model weights are trained in the first stage and the alpha is trained in the third stage. But it is unclear in the second stage. how to optimize the auxiliary model. Whether to fix the alpha and only train the weights or alternatively train the weights and alpha just like the original DARTS[1], with the head changed to predict the correlations between two pairs of examples. The author should clarify this procedure.\n- Q3.As shown in Page 5, the author ties the architectures and weights of main and auxiliary models.  In what stage they are tied? And my concern is, if the main and the auxilary are the same model both in weights and alpha, actually there are no knowledge trainsfer but only model copy. Why a copied model can take effect in the selection of the alpha?\n- Q4.Is it necessary that auxiliary model have to be the DARTS? Does the author try any other auxiliary mode, such as ResNet50? \n- Q5.Further, maybe the choice of the auxiliary model would be the bottelneck of the method. Maybe more ablations about how the size of the auxilary model affect the main model is needed. Such as, choose ResNet18, ResNet50 and ResNet101 or DARTS with 1,3,5,8 choice blocks as auxiliary model.\n- Q6. Just curious, what is the results when applying on DARTS 1st order? \n- Q7.The method may be too complex and hand-crafted\n- Q8.What is the generalization performance when search a model from C100 and evaluate it on C10 and search from C10 and evaluate on C100.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "6QfB3ekkWt", "reviewer": "Reviewer_8tqH", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to improve the generalizability and stability of differentiable NAS methods. Specifically, the authors first train a main model and then transfer knowledge from the main model to an auxiliary model. Finally, the authors optimize the architecture of the main model by maximizing its transferability to the auxiliary model. Experiments on several datasets show the effectiveness of the proposed model. ", "summary_of_the_review": "Although the proposed method is simple, the solution is somewhat trivial (e.g., quadruple relative similarity relation using cosine similarity strategy). The experiments need comparison with more latest methods.", "strength_and_weaknesses": "Strengths:\n\nThis paper develops a transferability-encouraging tri-level optimization framework in differentiable NAS and demonstrates that they can achieve higher performance on Imagenet and CIFAR compared to similar methods.\n\nThe authors introduce a new knowledge transfer approach based on matching quadruple relative similarities (QRS) to improve the auxiliary model and demonstrate its effects with ablations.\n\nThe paper is well written and easy to understand.\n\nWeaknesses:\n\n1. The method consists of training the main model, training the auxiliary model with the help of the main model (to determine similar data pair), and optimization. The main contribution seems to be the auxiliary model using the data pairs to learn the weights. In other words, the fixed main model provides the supervisory signal for the auxiliary model. The main methodology does not seem to be novel.\n\n2. Most of the baseline methods in Table 3&4 are at least 2 years ago. The work lacks comparison with more latest methods in experiments.\n\n3. Are the two data pairs (x,y) and (z,w) randomly sampled?\n\n4. In the case of cosine similarity c(x,y) > c(z,w), it appears that x,y,z and w examples are 4 class examples. In this case, how to optimize them?\n\n5. What is the difference between the data D_a and D_m?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster