Unsupervised Performance Predictor for Architecture Search
Authors: Xiangning Xie
Yanan Sun
Yuqiao Liu
Venue: Submitted to ICLR 2023 (2023)
Keywords: Neural Architecture Search
AutoML
Performance Predictor

Abstract: Performance predictors can directly predict the performance value of given neural architectures without training, thus broadly being studied to alleviate the prohibitive cost of Neural Architecture Search (NAS). However, existing performance predictors still require training a large number of architectures from scratch to get their performance labels as the training dataset, which is still computationally expensive. To solve this issue, we develop an unsupervised performance predictor called USPP, which can avoid costly dataset construction by using existing fully-trained architectures. Specifically, a progressive domain-invariant feature extraction method is proposed to assist in extracting domain-invariant features due to the great transferability challenge caused by the rich domain-specific features. Furthermore, a learnable representation (denoted as operation embedding) is designed to replace the fixed encoding of the operations to transfer more knowledge about operations to the target search space. In experiments, we train the predictor by the labeled architectures in NAS-Bench-101 and predict the architectures in the DARTS search space. Compared with other state-of-the-art NAS methods, the proposed USPP only costs $0.02$ GPU days but finds the architecture with $97.86\%$ on CIFAR-10 and $96.50\%$ top-1 accuracy on ImageNet.

Under review as a conference paper at ICLR 2023
PERFORMANCE PREDICTION VIA UNSUPERVISED Do- MAIN ADAPTATION FOR ARCHITECTURE SEARCH
Anonymous authors Paper under double-blind review
ABSTRACT
Performance predictors directly predict the performance value of given neu- ral architectures without training, thus broadly being studied to alleviate the pro- hibitive cost of Neural Architecture Search (NAS). However, existing performance predictors still require training a large number of architectures from scratch to get their performance labels the training dataset, which still computationally pensive. To solve this issue, we develop an performance predictor by applying the unsupervised domain adaptation technique called USPP, which can avoid costly dataset construction by using existing fully-trained architectures. Specifically, a progressive comain-innvrrint feature extraction method proposed to assist in ex- tracting commann-invarian features due to the great transferability challenge caused by the rich domain-speciff features. Furthermore, learnable representation (de- noted operation embedding) is designed to replace the fixed encoding of the operations to transfer more knowledge about operations to the target search space. In experiments, train the predictor by the labeled architectures in 101 and predict the architectures in the DARTS search space. Compared with other cate-of-theeart NAS methods, the proposed USPP only costs 0.02 GPU days but finds the architecture with 97. 97.86% on CIFAR-10 10 and 76.50% top-1 accuracy ImageNet.
INTRODUCTION
Neural Architecture Search (NAS) (Flsken et al.) 2019) aims to automatically design high- performance neural architectures and has been a popular research field of machine learning. In recent years, the architectures searched by NAS have outperformed manually-designed architectures in many fields (Howard et al., 2019 Real et al. 2019). However, NAS generally requires massive computation resources to estimate the performance of architectures obtained during the search pro- cess (Real et al. 2019 Zoph et al., 2018). In practice, this is unaffordable for most researchers interested. As result, how speed up the estimation of neural architectures has become hot topic among the NAS community.
Performance predictor (Wen al. 2020) is a popular accelerated method for NAS. It directly predict the performance of neural architectures without training, thus greatly accelerating the NAS process. large number of related works are carried out because of its superiority in reducing the costs of NAS. For example, E2EPP (Sun et al. 2019)) adopted random forest (Breiman) 2001) the regression model effectively find promising architectures. ReNAS (Xu et al. 2021 used a simple LeNet-5 network (LeCun et al. 1998) as the regression model, and creatively employed a ranking-based loss function to train the predictor, thus improving the prediction ability of the performance predictor. Although existing performance predictors gain huge success in improving the efficiency of NAS, sufficient architectures need to sampled from the target search space and be fully trained to obtain their performance value the label (Wen et al. 2020). The performance predictor is trained by these labeled architectures, and then is used to predict the performance of architectures. In order to ensure the prediction accuracy of the performance predictor, it is usually necessary to train at least hundreds of architectures as the dataset, which a huge cost.
In recent years, many benchmark datasets such JAS-Bench-11 (Ying et al. 2019), NAS-Bende 201 (Dong & Yang 2020). PAS-Bench-NLL [Klyuchnikov et al., 2020) are released for promoting the research NAS. There are a large number of architecture pairs (i.e., the architecture and its

---

features, thus assisting in the alignment of the source and Under review as a conference paper at ICLR 2023
performance) in these datasets. As result, we are motivated utilize the rich architecture knowl- edge in these datasets to predict the architectures in the target search space (i.e., the search space in which the architecture needs to be predicted.). In this way, we avoid training large number of architectures in the target search space, thereby alleviating the expensive cost of building the dataset for performance predictors. However, the search space designed in the benchmark datasets is very different from the real-world search spaces. The performance predictor trained on existing labeled architectures cannot be applied the target search space.
In this paper, we proposed an UnSupervised domain adaptation-based Performance Predictor (USPP) with the usage of the domain adaptation technique. Different from the traditional perfor- mance predictors that need the training data and the predicted data in the same search space, USPP can leverage the labeled architectures existing benchmark datasets (e-g., NAS-Bench-101 (Ying et al. 2019)) to build a powerful performance predictor for the target search space (e.g., the DARTS search space (Liu al. 2018b)). As a result, USPP can avoid expensive data collection for the target search space. Specifically, the contributions can be summarized as follows:
A progressive Comain-invariant feature extraction method proposed to reduce the transfer difficulty caused by the huge difference between source and target search spaces. The progressive method explicitly models the somain-specific features and gradually separates them from the target search spaces.
A learnable representation for the operations in architectures, i.e., operation embedding, is designed to transfer more knowledge about operations to the target search space. Compared to the widely used fixed encoding method, the operation embedding can effectively capture the inner meaning and structural role of each operation in the source search space and applied them in the target search space to reduce transfer difficulty.
USPP only costs 0.02 GPU days to search for the architectures the DARTS search space because there is no need to annotate the architectures in the target search space. Fur- thermore, the searched architecture by USPP achieves 97.86% classification accuracy on CIFAR-10 and 76.50% classification accuracy ImageNet and outperforms all the state- of-the-ar art methods compared.
RELATED WORK
2.1
NAS AND PERFORMANCE PREDICTORS
NAS can automatically design perforrefree neural architecture and consists of search space, search strategy, and performance estimation strategy (Elsken et al. 2019). Specifically, the search space defines the collections of the candidate architectures. The search strategy corresponds to the employed optimization algorithms for the search, which can be mainly classified into evolutionary algorithms (Bäck et al. 1997). reinforcement learning (Kaelbling et al.] 1996) and gradient de- scent (Liu [ 201 The performance estimation strategy defines how to evaluate the architec- tures to obtain their performance. During the search, the NAS algorithms use the search strategy to
search architecture in the predefined search space and obtain the performance value of the searched architectures by the performance estimation strategy.
No matter what search strategy is used, a lot of neural architectures need to be estimated. Because of the heavy cost of the traditional GPU-based estimation method, many accelerated methods are proposed such as early stopping policy (Sun et al. 2018). proxy dataset (Sapra & Pimentel 2020), weight-sharing method (Bender et al.) 2018). However. the first two methods may lead to poor generalization and low-fidelity approximation of performance value, and the weight-sharing method may be unreliable in predicting the relative ranking among architectures (Li et al. Yang et al.)., which disobeys the goal of finding the architecture with the highest ranking. Performance predictor free from the aforementioned shortcomings and has received great attention in recent years. However, existing predictors have the constraint that the source search space and the target search space must be the same. The proposed USPP breaks the limitation and creatively uses the existing labeled architectures in the source search space to predict the architectures in the target search space, thus removing the reliance on potentially costly labels in the target search space.
2

---

Under review as a conference paper at ICLR 2023
2.2 DOMAIN ADAPTATION
In many machine learning methods, there is a major assumption that the training data and testing data from the same distribution (Wilson & Cook 2020). When is not held, the network trained on the source domain will face a performance decline when testing on the target domain (Patel et al.) 2015). Transfer learning (Weiss et al. 2016) can solve the problem by learning knowledge from the source domain and applying it to the target domain. Domain Adaptation (DA) (Wilson & Cook, 2020 is a subcategory of transfer learning, and its goal to train a network that performs well on different but related target domain by the labeled data in the source domain. The scenario in this paper is a problem of Unsupervised Domain Adaptation (UDA), where the labeled data are only available in the source domain.
The mainstream method to address the UDA aligning source and target domains by learning the domain- invaaiiinn feature representation. Specifically, the features said to be domain-invarizn features extracted from data in both source and target domains follow the same distribution. If network performs well in the source domain using network can be generalized well to the target domain. Generally, the domain-invariant methods can be classified into two categories. The first category explicitly reduces the domain discrepancy to obtain the domain-invariant by some distribution discrepancy metrics such as Maximum Mean Discrepancy (MMD) (Gretton et al., 2006 2012), correlation alignment (Sun al. 2016), and con- trastive domain discrepancy (Kang et al., 2019). Motivated by the Generative Adversarial Network (GAN) (Goodfellow et al., 2014). another category learns versarial training (Ganin et al.) (016)) and this is also what our method belongs to. The methods generally train domain discriminator to distinguish the source domain from the target domain, and train a feature extractor to fool the discriminator to extract domain-invarian versarial domain adaptation achieves great success in many fields, and a lot of works are proposed. For example, symnets (Zhang et al. 2019) designed symmetric classifiers to play the role of do- main discriminator. GVB (Cui et al., 2020b constructed bridge layers on both the generator and discriminator to reduce the overall transfer difficulty. However, most existing adversarial methods are performed on the classification task of Computer Vision (CV) or Neural Language Processing (NLP), and cannot be directly exploited in performance predictor which is regression problem and deals with a completely different data type (i.e., graph data). In this paper, we apply the adversarial iamain-invariant feature extraction
feature representation, the
representation by ad-
representations. Ad-
methods in performance predictors and propose a progressive method to reduce the transferring difficulty. As far as we know, this is the first work to alleviate the high costs of performance predictors by the adversarial domain adaptation techniques.
APPROACH
3.1 FORMULATION
In the scenarios of our paper, the architecture dataset in the source domain is denoted as Ds = (xx)x)))
} with Ns labeled architectures, where x and ya denote the 7th architecture and its performance value, respectively. The dataset in the target domain is denoted as DD {xx"' J Nr with N7 unlabeled architectures. USPP aims to train a performance predictor on Ds and Dy. and achieve promising results in the performance prediction of the architectures in target domain.
The overall framework of the proposed USPP is shown in Fig I Specifically, to build the predictor, the neural architectures first should be represented into form that can be fed into the predictor. Gen- erally speaking, most neural architectures can be regarded as the Directed Acyclic Graph (DAG). The node in the DAG corresponds one specific operation (c.g., convolution 3 3x3), and the edge stands for the connection between operations. Consequently, each architecture can be represented by the adjacency matrix A and the operation list Trop- where A stands for the topology connec- tions between nodes and Lop represents the operations of nodes. Because the most commonly used encoding method for Lop (i.e., one-hot method) ignores the intrinsic relationship between different operations, we adopt learnable operation embedding & to map Lop to continuous space RR to avoid the drawback, and the operation embedding will be elaborated in Sec. 3.4 Then, we use a Graph Convolution Network (GCN) as the first component of USPP to map every architecture into the feature space Z. In specific, GCN is a powerful technique to learn useful features from graph data which is a kind of non-Fuclidean data, and has achieved success in processing neural architecture
3

---

Under review as a conference paper at ICLR 2023
Source Domain
Feature Extractor (G)
Regressor (R)
GCN
Freg
Target Domain
Lri
Operation Embedding
GOOd
B
L
Discriminator (D)
Figure 1: The overall framework USPP.
data. After learning the latent representation of the architectures by G, regressor Rt the second component of USPP, is exploited to predict the performance of the architectures.
To ensure the prediction accuracy in target domain, the prediction results in source domain needs to be guaranteed first. It can be achieved by minimizing the regression loss Freg on G and when training. The regression loss can be calculated
Ns Creg Ns
R())))(((x)))
where L is the L2 loss fuction in USPP.
3.2 ADVERSARIAL TRAINING
As mentioned above, we are motivated to use the existing labeled architectures in the benchmark datasets to build a performance predictor for the target search space. However, the gap between the architectures in different search spaces so large that the performance predictor trained with the labeled architectures in the source domain cannot be directly applied to predict the architectures in the target domain. Accordingly, we leverage the adversarial training (Wilson & Cook 2020) to learn representations, thus reducing the discrepancy between source and target domains. In this way, the performance predictor can generalize well to the target domain.
To perform adversarial training, discriminator D is designed as the third component of USPP, and it is a fully connected layer. Similar to the regressor R, is attached after the feature extractor G. Specifically. the discriminator is essentially domain classifier differentiate between the features from the source domain and the target domain, and only works during the training process of USPP. During the training, the discriminator D optimized correctly classify the domains, while the feature extractor is optimized to fool the discriminator so that the discriminator cannot distinguish the source domain from the target domain. Through the adversarial training, the feature extractor encouraged to extract the common features shared by domains (i.e., Nomain-invarian features).
To adversarially train the discriminator and the feature extractor, the domain classification loss Cets is designed, which can be formulated as:
Ns log (G(
Lets
NT Σ log - D (xf))
Ns
To make D distinguish between the source domain and target domain, train D to minimize the domain classification loss Lete- At the same time, we train to maximize Cetes thus making the feature distributions from source and target domains similar as possible to fool D.
3.3 PROGRESSIVE DOMAIN-INVARIANS FEATURE EXTRACTION
Although the adversarial training can align the source and target domains to a certain extent, difficult to reduce the divergence between source and target domains in practice due to
4

---

Under review as a conference paper at ICLR 2023
the existence of rich domain-speciffe features in respective domains. Specifically, domain- sppciifi features refer to the features unique to each domain and greatly hinder the alignment of the data distributions from source and target domains. To further reduce the negative influence of domain- specific features, we propose simple but effective progressive domain-invariant feature extraction method.
The method reduces the influence domain-specing features mainly by explicitly modeling them. In fact, there are some similar works mitigate domain-specifie features in this way. For example, DSN (Bousmalis et al. 2016) separately modeled the domain-specific features of each domain and the domain-invarian feature shared by source and target domains and reconstructed the input data by the extracted domain-specinfe and domain inriiinnivariaat features to add generalizability. However, the modeled domain-invarian features also participated the reconstruction, leading to them with lot of domain--peccffc properties. HDA (Cui et al. 2020a) proposed a heuristic framework, and leveraged the modeled cemman--ppecffc features as heuristics to gradually gain invaarian fea- tures. However, the multiple sub-networks in the heuristic network made it difficult to optimize. The proposed progressive feature extraction method can effectively model the domain- -specffe features and gradually separate them from the features for the architecture data processed in this paper.
To alleviate the domain- -ppaif- features more easily, we hypothesize that the domain -mmain-specific fea- features according to HDA. This
tures casier to be captured compared with the the same domain generally share common domain-specific charac-
because the architectures teristics. For example, the cell in the NAS-Bench-10) only has input while that in the DARTS search space has two inputs. Furthermore, the operations in NAS-Bench-100 101 are completely dif- ferent from those in the DARTS search space. Compared with the domain-invariant features, these domain-specific features are easier to be extracted. The proposed method is embodied in the design of the regressor R. The regressor is used to map the extracted feature from to the predicted label, and consists of fundament layer F, bridge layer B, and prediction layer P. Specifically, fundament layer F used to model the current feature representation r,, and bridge layer B utilized to ex- plicitly model the discrepancy from the current feature to the ideal domain-invariant feature, i.e., the domain- comain--pecific part b, in the current feature. By subtracting the results of bridge layer b, from the results of ri, the domain-invariant invariant features d; can be obtained. This is because b; is easier to be obtained than d, and can give guide to the constructing d, Finally, the obtained d, is fed to the prediction layer P to get the final prediction label ÿ. The overall process can be formulated as:
R(f.)) = P(F(f.) B(J))
where fi denotes the extracted feature from G. During training, we seek to reduce the influence of the modeled domain domain-specifi specific feature b, help the extraction of comainn-nvvariin representation. To achieve this goal, we design an extra loss Liri as following:
N.+N, L
Lori (Ns + NT)
Σ
b
(4)
where is the size of b; During the training of USPP, we aim to gradually minimize Cori, thus mitigating dpmain-specific representation. Therefore, at the beginning of training. the designed bridge layer assists in the progressive extraction of the of training, by is close zero, which will not have too much impact on training.
feature. In the later stage
In conclusion, USPP has three objectives in the training phase: 1. Minimizing freg on G and R to get high prediction accuracy in the source domain. 2. Minimizing Lats on 1) and maximizing Lats on G to perform adversarial training. 3. Minimizing Lors on R and G to assist in the extraction of the domain-invariant features. Let Ldom Lotsr the overall objectives during training as following:
min G.R
Greg Loom -Lbrii
max D Ldoo
(5)
where 7 and μ are the hyper-parameters.
3.4 OPERATION EMBEDDING
As is mentioned above, neural architectures can be represented by the adjacency matrix and the operation list Topps where A represents the connection relationship between different nodes, and
5

---

Under review as a conference paper at ICLR 2023
Lop represents the operation type of every node. Generally, the one-hot method is used to encode the operation list Top Mlen(Lop)xK and A are fed into GCN obtain the feature representation of a given architecture. However, there is large limitation that the process ignores the computational relationship between different operations. This because every operation thought be independent in the one-hot method, but it is obviously not in line with reality. For example. the role of 1 X 1 convolution may be more similar to that of 3 x convolution compared with that of the max pooling operation. According to this motivation, the trainable operation embedding is proposed instead of the one-hot method.
fixed matrix MMlem(Lop) (Liu et al. 2021). Then, the fixed matrix
Specifically, we operation embedding & to map each operation in Lop to a continuous space plen(Fop)xX K is the dimension of the mapped vector and can be set during the training. Then, the K-ddimensional vector obtained by & along with the adjacency matrix is mapped to the feature representation by GCN. During the training process of USPP, the operation embedding is regarded as a part of GCN, and the weight of the operation embedding is optimized with the weights of GCN. Through this, the operation embedding can seek the inner relationship between different opera- tions during the training. In specific, the operation embedding is similar to the word embedding in NLP, except that the mapped object words are replaced by the operation in the architecture. Word embedding can encode the semantic and syntactic information of every word, where semantic infor- mation related the meaning of words, and syntactic information corresponds to the structural roles of words (Li & Yang, 2018). Similarly, the inner meaning and structural information of each operation can be also learned from the source domain and applied the target domain through the operation embedding. As result, operation embedding helps USPP transfer more information about the operation from the source domain to the target domain, thus improving the prediction performance in the target domain.
3.5 THEORETICAL ANALYSIS
In this section, we analyze the expected target-domain error for the proposed method based on the theories of domain adaptation (Ben-David et al.] 2010). Let S and T be the source and target distributions for family of fundament layer F' in the hypothesis space H. ST(F) and (F( be the expected error of a hypothesis F E on target and source domains. bound the target error, we use the Hid-dist to measure the distance between distributions S and T, which is defined as the following:
dHAn(S,T) 2 sup PE (f) + h2(()) PEWT (f) * h2()))
(6)
We can get the target error of hypothesis F by the theory in (Ben-David et al. 2010):
(F)F) S es(F) dH T)
where À ST (F*) + ts (F*) is the combined error of ideal hypothesis F" (es + eT(F)) on both domains. By the definition of H&H--distance, dHAD (Ds, DT) 2 2sup Es (h, ht) - ST (1), Hence,
(F(F) S (E((F) suples (F,F" ET (F,F")+
Because the ideal discrepancy B' from the current feature to the ideal Nomain-invariant feature may be not fully represented by the practice discrepancy B, B < B*. Furthermore, B* F F* Hence, there exists positive correlation between B and F:
B
k(F E [0.
As mentioned in Subsec. 3.3 = + P. Furthermore, the goal of F and P both to better obtain the performance of the given architecture. As result, the ideal hypothesis F" = pt. We could obtain the relationship between F and P:
(1 -1- k) (F
(P-P")
(10)
Equation (00 is held on source and target distributions. Hence, Es (P,P*) = (1- k))s ( (F) and εT (P, Px) (1 k)er (F, F*). The P and F can both correctly predict the performance

---

Under review as a conference paper at ICLR 2023
Table I: Comparison with other methods in DARTS search space CIFAR-10. The second
shows the state-of-the-ar NAS method, and the third row shows the advanced performance predictor.
The fourth row shows the methods that do not require training architecture USPP does.
Architecture
Accuracy( %)
#Params.(M)
Cost (GPU day)
N (oph et al. 2018)
97.35
3,3
1,800
AmoebaNet-A (Real et
2019)
96.66-00.06
3.2
3,150
ENAS (Pham et al. 2018
97.11
4.6
0.5
DARTS (Liu et al. 20186)
97.24=0.09
3.4
SNAS (Xe et
2018)
97.02
2.9
1.5
PC-DARTS (Xu et al. 2019
97.43
3.6
0.1
P-DARTS (Chen al.
2019)
97.50
3.4
0.3
DARTS-PT (Wang et a 20222
lb)
97.54
3.9
0.29
RANK-NOSH Wang al.
2021a)
97.50
3.5
NAONet (Luo et al. 2018
97.02
28.6
200
BONAS (Shi al. 2020)
97.31
3.45
2.5
CTNAS Chen et al
2021
97.41=0.04
3.6
0.3
TNASP Lu el al.
(2221a)
97.43=0.04
3.640..1
0.3
NASWOT (Mellor al.
2021
95.73
4.96
TENAS (Chnn et al. 2020)
97.37+10.04
3.8
0.05
USPP(ours)
97.86
3.3
0.02
of architectures, the differences between them on the source domain are small. Hence, es(P) FF). Combined with Equation (10), we can obtain Equation (II):
eT(P) es(P) + sup lss (P, - (P,P)) $ (F) + (1 k) sup ES (F,F*) Fr (F,F*)+
ET
(11)
$ (F) + (1 k) (S,
can be seen from Equation 11 that the upper bound of the target-domain error for with the usage of B is lower than that for F' without the usage of B. Therefore, the proposed method can help to reduce the upper bound of the target-domain error, which also proves the effectiveness of our proposed method from theoretical point of view.
EXPERIMENTS
In this part, choose the as target domains to train USPP. We first report the results of neural architecture search in DARTS to verify the effectiveness of USPP. Then, we perform extensive ablation studies var- ious components. Please note that we perform additional experiments
as source domain and the DARTS search space
11-BBnch-101--NA..
Bench-201, NAS-Bench-201- -NAS-BBne- 101, NAS-Bench-201-2 -DARTSSSS NDS ResNet++NDS
ResNeXt Radosavovic et al. (2019). and NDS ResNeXt-NNDS ResNet) in Appendix B All ex- periments the paper performed on NVIDIA GTX 2080Ti GPU. The experimental settings
presented in Appendix A
4.1 NEURAL ARCHITECTURE SEARCH ON DARTS
Training process. The regression loss Creg. the domain classification loss Catss and the bridge loss Cori are shown in Fig. 4.1 a). As expected, Creg. Cole, and Cors reduce repaidly. Kendall's Tau (KTau) (Sen) 1968), which can describe the correlation between the predicted value and the ground- truth values, used measure the prediction performance of USPP. The trend of the KTauu value of USPP is shown Fig. 4.1b) b). It can be seen that the KTTuu of USPP goes up very quickly and converges.
CIFAR-10. The experimental results on CIFAR- 10 are shown in Table I can be seen that the proposed USPP only costs 0.02 GPU days, which is the least among all methods compared. Please

---

Under review as a conference paper at ICLR 2023
Table 2: Comparison with other methods in DARTS search space ImageNet. The second shows the state-of-the-art NAS method, while the third row shows the advanced performance pre- dictor. The fourth row shows the methods that do not require training architecture USPP does. The symbol " denotes that no results are available in original papers. represents the results with label smoothing, autoaugment and SE methods.
Architecture
Top-1(%)
Top-5(%)
#Params.(M)
Cost (GPU day)
NASNet-A Zoph et al. 2018
74.0
91.6
5.3
2,000
AmoebaNet-A (Real al. 2019)
74.5
92.0
3.2
3,150
PNAS (Liu et al. 20188
74.2
91.9
5.1
225
DARTS (Liu al. 2018b)
73.3
91.3
4.7
SNAS (Xie al 2018)
72.7
90.8
4.3
1.5
PC-DARTS Xu ei al. 2019
74.9
92.7
3.6
0.1
DropNAS (Hong et al. 2021
75.5
92.6
5.2
0.6
DARTS- PT (Wang et al. (2021b)
74.5
92.0
4.6
0.29
NAONet (Luo al. 2018)
74.3
91.8
11.35
200
Neural Predictor Wen et al.
2020)
74.7
RANK-NOSH (Wang et 2021a)
74.8
5.3
0.3
TENAS (Chen et al. (020)
75.5
92.5
5.4
0.17
USPP(ours)
75.9
92.9
5.0
0.02
USPP(ours)
76.5
93.0
5.3
0.02
note that we do not take the costs for building the NAS-Bench- 101 into consideration because NAS- Bench-101 101 is a public dataset and constructed for work. Besides, TENAS with the least cost among other methods, takes 2.5 times as long as USPP. This is because USPP takes advantage of the fully-trained architectures in MAS--Benc--1e 101, thus eliminating the cost of building new architec- tures in the DARTS search space. Furthermore, the searched architecture by USPP gets the highest test accuracy 97.86% compared with all the other state-of-the-ee methods. Specifically, racy of the architecture searched is still 0.32% higher than that of the best performance architecture searched by other methods. This shows that our method can effectively transfer knowledge from the source search space to the target search space.
aceu-
ImageNet. The comparison results on ImageNet are shown in Table g It can seen from the results that the search cost of USPP is the lowest, and the PC-DARTS method with the least cost of the other methods is five times of the USPP. Besides, the searched architecture by USPP trained with- out any tricks can obtain the high- est top-1 and top-5 accuracy, and the training result with positive technique largely surpasses existing methods in top-1 and top-5 accuracy. As result, USPP can transfer useful information from NAS-Bench- 101 and learn the inner correspondence between archi- tectures and performance in DARTS.
The losses of USPP.
(b) The KTau value of USPP.
Figure 2: The training process 101 -ARTS..
on NAS-Bench-
Table 3: The ablation study on the progressive domain-invariant feature extraction. Bridge Layer Kendall's Tau
Table 4: The ablation study on the operation embedding.
Encode Method of Lop
Kendall's Tau
0.2679
one-hot 0.3104
0.4113
operation embedding
0.4113
8

---

Under review as a conference paper at ICLR 2023
Table 5: The ablation study on the domain adaptation method.
Domain adaptation method
Kendall's Tau
MMD (Gretton al.
2006)
0.3162
DANN (Ganin & Lempitsky
2015
0.2424
GVB (Cu et al.
2020b
0,2699
HDA (Cu et al.
2020a
0,3173
USPP()urs)
0.4113
4.2 ABLATION STUDY
In this part, we conduct ablation experiments on the progressive domain-invariant feature extraction method, operation embedding, and domain adaptation methods. We randomly sample 100 architec- tures from JAS-Bench-331 [Siems et al. (2020) to evaluate the proposed USPP.
The progressive Nomain-invariant feature extraction. To validate the effectiveness of the pro- posed progressive method, we have performed ablation experiments on the core components of the feature extraction method, i.e., the bridge layer. It can be seen from Table 3 that the performance predictor with the bridge layer gets higher Kendall's Tau value and 0.1434 more than the result without the bridge. The result indicates that the progressive domain- invariant feature extraction plays a positive role in improving the performance of USPP.
progressive
Operation embedding. The results of the ablation study on the op- eration embedding are reported in Table 4 It can be seen from the result with operation embedding is more 0. 0.1009 than that without operation embedding, which illustrates that operation embedding improves the performance of the predictor. Furthermore, the length of operation embedding is set as 3 in the experiment because of the small number of the operations, and the 3D visual result is shown in Fig. 3 Surprisingly, convolution x 1 and output the most similar pair in all pairs of operations. Furthermore, just as we sus- pected, the distance between convolution operations is smaller than the distance between convolution and pooling operations.
1x1
33
14
mn 34
1104
575
0.5
Different domain adaptation methods. We adapt the distribution discrepancy metric-based method MMD (Long et al.] 22155) and the adversarial training-basse based methods DANN (Ganin & Lempit- sky 2015), GVB (Cui et al.] 2020b). and HDA (Cui et al.) (000a). The hyperparameters of these methods are optimized, and the best results of them are reported in Table 7 It is obvious that our method outperforms all the peer com-
Figure 3: The visual result of operation embedding.
petitors.
5 CONCLUSION
In this paper, we propose an performance predictor with the unsupervised domain adaptation tech- nique to reducing the high cost of annotating architectures. Specifically, we design a progressive domain-invariant feature extraction to reduce the difficulty alignment by explicitly modeling the domain-specifee and separating them of domain-invariant features. Moreover, we use operation em- bedding to map the operation list of every architecture to continuous space, and learn the deeper meaning and structural information of each operation during training. The USPP achieves the test accuracy of 97.86% and 76.5% on CIFAR-10 and ImageNet, respectively. Unfortunately. this pa- per does not consider sequence-based search spaces and will propose more general domain adaptation framework in the future.
REFERENCES
Thomas Bäck, David B Fogel, and Zbigniew Michalewicz. Handbook of evolutionary computation. Release, 97(1):B1, 1997.

---

Under review as a conference paper at ICLR 2023
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wort- man Vaughan. A theory of learning from different domains. Machine Learning, 79(1):151-175, 2010.
Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, and Quoc Le. Understand- ing and simplifying one-shot architecture search. In International Conference on Machine Learn- ing. pp. 550-559. PMLR, 2018.
Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan, and Dumitru Erhan. Domain separation networks. Advances in Nneural Information Processing Systems, 29, 2016.
Leo Breiman. Random forests. Machine Learning, 45(1):5-32, 2001.
Wuyang Chen, Xinyu Gong, and Zhangyang Wang. Neural architecture search on imagenet in four gpu hours: theoretically inspired perspective. In International Conference on Learning Representations, 2020.
Xin Chen, Lingxi Jun Wu, and Qi Tian. Progressive differentiable architecture search: Bridging the depth gap between search and evaluation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1294-1303, 2019.
Yaofo Chen, Yong Guo, Qi Chen, Minli Li, Wei Zeng. Yaowei Wang, and Mingkui Tan. Contrastive neural architecture search with neural architecture comparators. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 9502-9511, 2021.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 113-123, 2019.
Shuhao Cui, Xuan Jin, Shuhui Wang, Yuan He, and Qingming Huang. Heuristic domain adaptation. Advances in Neural Information Processing Systems, 33:7571-7583, 2020a.
Shuhao Cui, Shuhui Wang, Junbao Zhuo, Chi Su, Qingming Huang, and Qi Tian. Gradually van- ishing bridge for adversarial domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12455-12464, 2020b.
Xuanyi Dong and Yi Yang. MAS-Bench-2011 Extending the scope of reproducible neural architec- ture search. arXiv preprint arXiv:2001.003320 2020.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. The Journal of Machine Learning Research, 1997):1977-2017, 2019.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In International Conference on Machine Learning, pp. 1180- 1189. PMLR, 2015.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural net- works. The Journal of Machine Learning Research, 17()):096-2030) 2016.
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in Neural Informa- tion Processing Systems, 27, 2014.
Arthur Gretton, Karsten Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alex Smola. A kernel method for the two-sample-problen Advances in Neural Information Processing Systems, 19, 2006.
Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-773, 2012.
Weijun Hong, Guilin Li, Weinan Zhang, Ruiming Tang, Yunhe Wang, Zhenguo Li, and Yong Yu. DropNAS: grouped operation dropout for differentiable architecture search. In Proceedings of the Twenty-Ninh International Conference on International Joint Conferences on Artificial Intel- ligence, pp. 2021.
10

---

Under review as a conference paper at ICLR 2023
Andrew Howard, Mark Sandler, Grace Chu, Liang- Chang-Chich Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for MobileNetV3. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 1314-1324, 2019.
Jie Hu, Li Shen, and Gang Sun. Squeere-and-exciaaton networks. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 7132-7141, 2018.
Leslie Pack Kaelbling, Michael L Littman, and Andrew W Moore. Reinforcement learning: A survey. Journal of Artificial Intelligence Research, 4:237-285, 1996.
Guoliang Kang, Lu Jiang, Yi Yang, and Alexander Hauptmann. Contrastive adaptation network for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 4893-4902, 2019.
Nikita Klyuchnikov, Ilya Trofimov, Ekaterina Artemova, Mikhail Salnikov, Maxim Fedorov, and Evgeny Burnaev. MAS-Bench-NLP: Neural architecture search benchmark for natural language processing. arXiv preprint arXiv:2006.07716, 2020.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- based learning applied to document recognition. Proceedings of the IEEE, 86(11):227--233 1998.
Liam Li, Mikhail Khodak, Nina Balcan, and Ameet Talwalkar. Geometry-aware gradient algorithms for neural architecture search. In arXiv preprint arXiv:2004.07802 2020.
Yang Li and Tao Yang. Word embedding for understanding natural language: A survey, In Guide to big data applications, pp. 83-104. Springer, 2018.
Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. Proceed- ings of the European Conference Computer Vision (ECCV), pp. 19-34, 2018a.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. arXiv preprint arXiv:1806. 000::0006009055 2018b.
Yuqiao Liu, Ychui Tang, and Yanan Sun. Homogeneous architecture augmentation for neural predic- tor. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12249- 12258, 2021.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In International Conference Machine Learning, pp. 97-105. PMLR, 2015.
Shun Lu, Jixiang Li, Jianchao Tan, Sen Yang, and Ji Liu. TNASP: A transformer- based NAS predictor with self-evolution framework. Advances Neural Information Processing Systems, 34, 2021a.
Zhichao Lu, Gautam Sreekumar, Erik Goodman, Wolfgang Banzhaf, Kalyanmoy Deb, and Vishnu Naresh Boddeti. Neural architecture transfer. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(9):2971-2989, 2021b.
Renqian Luo, Fei Tian, Tao Qin, Enhong Chen, and Tie- Kan Liu. Neural architecture optimization. Advances in Neural Information Processing Systems, 31, 2018.
Joe Mellor, Jack Turner, Amos Storkey, and Elliot Crowley. Neural architecture search without training. In International Conference on Machine Learning, pp. 7588-7598. PMLR, 2021.
Vishal M Patel, Raghuraman Gopalan, Ruonan Li, and Rama Chellappa. Visual domain adaptation: A survey of recent advances. IEEE Signal Processing Magazine, 32(3):53-69, 69, 2015.
Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. Efficient neural architecture search via parameters sharing. In International Conference on Machine Learning, pp. 4095- 4104. PMLR, 2018.
11

---

Under review as a conference paper at ICLR 2023
Iijaa Radosavovic, Justin Johnson, Saining Xie, Wan- Yen Lo, and Piotr Dollár. On network de- sign spaces for visual recognition. In Proceedings of the IEEE/CVE international conference on computer vision, pp. 1882- 1890. 2019.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Le. Regularized evolution for image classifier architecture search. In Proceedings of the AAAI Conference Artificial Intelligence, pp. 4780-4789, July 2019.
Dolly Sapra and Andy D Pimentel. Constrained evolutionary piecemeal training design volutional neural networks. In International Conference on Industrial, Engineering and Other Applications of Applied Intelligent Systems, pp. 709-721. Springer, 2020.
Pranab Kumar Sen. Estimates the regression coefficient hased on kendall's tau. Journal of the American Statistical Association, 63(324) 1968.
Han Shi, Renjie Pi, Hang Xu, Zhenguo Li, James Kwok, and Tong Zhang. Bridging the gap between sample-based and one-shot neural architecture search with bonas. Advances in Neural Information Processing Systems, 33: 18:118881111 2020.
Julien Siems, Lucas Zimmer, Arber Zela, Jovita Lukasik, Margret Keuper, and Frank Hutter. Nas- bench-301 and the case for surrogate benchmarks for neural architecture search. 2020. [Online]. Available: 2XXi:20088.0977)
Baochen Sun, Jiashi Feng, and Kate Saenko. Return of frustratingly easy domain adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016.
Yanan Sun, Bing Xue, Mengjie Zhang, and Gary G Yen. A particle swarm optimization-bo based flex- ible convolutional autoencoder for image classification. IEEE Transactions on Neural Networks and Learning Systems, 2005:)22222200000 2018.
Yanan Sun, Handing Wang, Bing Xue, Yaochu Jin, Gary Yen, and Mengjie Zhang. Surrogate- assisted evolutionary deep learning using end-to-end random forest-based performance predic- tor. IEEE Transactions on Evolutionary Computation, 24(2):350-33) 364, 2019.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2818-8888800 1818-2826, 2016.
Ruochen Wang, Xiangning Chen, Minhao Cheng, Xiaocheng Tang, and Cho-Jui Hsieh. Rank-nosh: Efficient predictor-based architecture search via non-uniform successive halving, In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10377-10386, 2021a.
Ruochen Wang, Minhao Cheng, Xiangning Chen, Xiaocheng Tang, and Cho-Jui Hsieh. Rethinking architecture selection in differentiable nas. In International Conference on Leurning Representa- tion, 2021b.
Karl Weiss, Taghi M Khoshgoftaar, and DingDing Wang. A survey of transfer learning. Journal of Big data, 3(1):1-40, 2016.
Wei Wen, Hanxiao Liu, Yiran Chen, Hai Li, Gabriel Bender, and Pieter-Jan Kindermans. Neural predictor for neural architecture search. In European Conference on Computer Vision, pp. 660- 676. Springer, 2020.
Garrett Wilson and Diane J Cook. A survey of unsupervised deep domain adaptation. ACM Trans- actions on Intelligent Systems and Technology (TIST), 11(5):1-46, 2020.
Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. SNAS: stochastic neural architecture search. In International Conference on Learning Representations, 2018.
Yixing Xu, Yunhe Wang. Kai Han, Yehui Tang, Shangling Jui, Chunjing Xu, and Chang Xu. ReNAS: Relativistic evaluation of neural architecture search. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4411 14200, 2021.
12

---

Under review as a conference paper at ICLR 2023
Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Guo-Jun Qi Qi Tian, and Hongkai Xiong. PC- DARTS: Partial channel connections for memory-effiiint architecture search. arXiv preprint arXiv:1907.055737, 2019.
Antoine Yang, Pedro Esperança, and Fabio M Carlucci. NAS evaluation is frustratingly hard. arXiv preprint 1rXiv:1912. 2019.
Chris Ying. Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter. NAS- Bench-101: Towards reproducible neural architecture search. In International Conference on Machine Learning. pp. 7105-7114. PMLR, 2019.
Yabin Zhang, Hui Tang, Kui Jia, and Mingkui Tan. Domain-symmetric networks for adversarial domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5031-5040, 2019.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8697-8710, 2018
A EXPERIMENTAL SETTING
In this section, we report the expermental settings of the experiments in Section
Data processing. In order to keep the DARTS search space and NAS-Bench-10 101 search space uni- form, we convert the architecture of the DARTS search space with operations on edges to one with operations on nodes following the convention (Liu et al.) 202T). Then, since the types of operations in the two search spaces are very different, we group all operation types into five categories: input, output, convolution 1 X 1, convolution 3 x 3, convolution 5 x 5, and pooling.
Setting of USPP. The Gradient Reversal Layer (GRL) utilized to simplify the adversarial training- Specifically, GRL can flip the gradients from D to G so that we can train G and R to minimize the sum of Lreg. Ldom and Lirs. We train the USPP with the batch size of 1, 000 for 100 epoch. We employ the SGD with the weight decay of 0.0005 and the learning rate is initialed with 0.1. We set in the loss which is verified to perform well in the experiments.
= and
Search setting. Followed the convention of performance predictor (Wen et al. 2020), randomly sample 100, 000 architectures from the DARTS search space, and predict them by the trained USPP. Then, select the architecture with the highest predicted performance value as the searched archi- tecture.
Training setting of the searched architecture. For the training setting CIFAR-10, we follow the convention of (Lu et al.] 2021b). As for the training setting on ImageNet, we train the searched architecture under setting, we train architecture for 350 epoch with the batch size of 512. Besides, we use the SGD optimizer with a momentum of 0.9, an initial learning rate of 0.2, and the weight decay of 4 x 10-5 In the second setting, the cosine learning rate strategy is used during the training on the basis of the first setting. addition, we use some enhancement techniques to improve the classification aceu- racy on ImageNet. Specifically, label smoothing (Szegedy et al. 2016) and autoaugment (Cubuk al. (2019) which are effective regularization methods are utilized in the training. Furthermore, the Bineezze-and-Exxitaaie (SE) module (Hu et al. 2018) is attached after each cell.
types of training settings compare with other methods fairly. In the first
B SUPPLEMENTARY EXPERIMENTS
In this part, we perform additional experiments cell-based search space and block-based search space, and compare our method with other domain adaptation methods. The detail the cell-based search space and the block- based search space used are shown in Fig. 6 and Fig. respectively. Specifically, MAS-Bench- 101, NAS- BuS--Bech-200 201, and DARTS belong to the cell-based search space, while NDS ResNet and NDS ResNeXt belong to the block-based search space.
13

---

Under review as a conference paper at ICLR 2023
Table 6: The description of three cell-based search spaces.
Search space
Size
Operation Position
#Node
#Edge
#Operation type
NAS MAS-Bench-101
423k
Node
9
NAS-Bench-201
15.6K
Edge
6
DARTS
10
Edge
6
8
7
Table 7: The description of two block-based search spaces.
Search space
Size
Depth
Width
Ratio
Group number
NDS ResNet
1260k
12
NDS ResNeXt
I139IK
B.1 CELL-BASED SEARCH SPACE
Table 8: The KTau value of USPP on NAS- Bench-201 WNAS-Bench1100 101-=NAS-Beemm
Table 9: The KTTau value of USPP NAS-
Bench- 101 MASS-Bench-201--NE 101).
201).
Methods
KTauu
Methods
KTTuu
MMD Gretton et al.
2006
0.406
MMD (Gretton al. 2006
0.306
DANN
Gannn & Lempitsky
2015
0.500
DANN
(Ganin & Lempitsky
2015)
0.325
GVB (Cu et al.
2020b
0.496
GVB CCu al.
20206
0.302
HDA Cui et al.
2020a
0.516
HDA (Cui et al.
2020a
0.307
USPP (ours)
0.655
USPP (ours)
0.349
Transferring from NAS-Bench-10e to NAS-Bench-201. The comparisons of the KTau value on
between USPP and other domain adaptation methods are shown in Table. It can be seen that the USPP gets the highest KTTau value among all peer competitors. This illustrates that the proposed method can effectively transfer knowledge from NAS-Bench-10. 101 to
Transferring from NAS-Bench-201 to NAS-Bench-101. Table g shows the comparison results of USPP NAS-Bench-101. It can be seen that USPP achieves the highest KTau value among all peer competitors, which verifies the effectiveness of USPP to transfer from NAS-Bench-201 to NAS-Bench-101.
Transferring from NAS-Bench-201 to DARTS. USPP evaluated in MAS-Bench-301. The com- parison results of transferring from NAS-Bench-201 DARTS search space are shown in Table. I0 It can be observed that although USPP surpasses all competitors, the K value of USPP is low. We infer that this is because the domain discrepancy between this learned from NAS-Bench-201 is not enough accurately predict the architecture DARTS.
too small compared DARTS (15.6k vs 10 and and DARTS is too large. The information
Table 10: The KTau value of USPP on DARTS search space WAAS-Bench-201--DAAR
Methods
KTauu
MMD (Gretton et al.
2006)
0.160
DANN (Ganın Lempitsky
2015
0.185
GVB (Cu et al.
2020b
0.255
HDA (Cui al. 2020a
0.153
USPP (ours)
0.287
14

---

Under review as a conference paper at ICLR 2023
Table 13: The regression and the domain classification accuracy.
Loss and accuracy
Value
Regression error 0.056
Domain classification accuracy
0.325
B.2 BLOCK-BASED SEARCH SPACE
Table 11: The KTauu value of USPP on NDS
Table 12: The KTTau value USPP on NDS ResNeXt (NDS ResNet->NDS ResNeXt).
ResNet (NDS ResNeXI-NNDS ResNet).
Methods
KTauu
Methods
KTau
MMD (Gretton et al. 2006)
0.373
MMD Gretton et al. 2006
0,319
DANN (Ganin Lempitsky
2015
0,334
DANN (Ganin & Lempitsky
2015
0,513
GVB (Cu et al. 20205
0,427
GVB (Cui et al. 2020D
0,485
HDA Cuu et al. 2020a
0.239
HDA (Cu et al.
2020a
0.391
USPP (ours)
0.442
USPP (ours)
0.616
Transferring from NDS ResNet to NDS ResNeXt. We apply the 25k public annotated architec- tures in the NDS ResNeXt search space and the unlabeled architectures in the NDS ResNet search space train USPP. Please note that we do not use the proposed operation embedding method, and replace the GCN (i.e.. feature extractor) with the MLP. The comparison results of the KTauu value shown in Table, IT The results show that the proposed USPP surpasses all competitors. This verifies the effectiveness of our approach in NDS ResNet which belongs to the block-base search space.
Transferring from NDS ResNeXt to NDS ResNet. Similar to the last experiment (i.e., NDS ResNeXt-tNDS ResNet), train USPP on 25k labeled architectures in the NDS ResNet search space and test USPP the public annotated architectures in the ResNeXt search space. The com- parison results are shown in Table [2| USPP achieves the highest KTau value. This illustrates that the proposed method can effectively learn the correlation between architecture and performance and transfer it to the NDS ResNeXt search space.
C DETAILS OF THE TRAINING PROCESS
Quantity of the domain-adaptation features. We have performed experiments on NAS-Benchi 201 DARTS and NAS-Bench-104 101 + NAS-Bench-2011 and computed the mean KL-Divergence and the mean cosine similarity. Specifically, the KL-Divergence is 0.0086, and the mean cosine similarity is 0.5018. These results show that the domain invariant features learn are similar in different cases (i.e., NAS-Bench-201 - DARTS, NAS-Bench- 101 + NAS-Bench-201) and further prove the quality of domain invariant features.
The regression error and the domain classification accuracy. The results are shown in Table 13 The regression error low, and illustrates that the regressor learns the correspondence between the architecture and performance in the source domain. Furthermore, the domain classification accuracy can verify that the discriminator cannot distinguish between domains.
15


Reviews:
[{"review_id": "my9gYxyWW2_", "reviewer": "Reviewer_YaG5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an unsupervised performance predictor called USPP to reduce the training/search cost of NAS. To bridge the source and target search spaces, the authors develop a progressive domain-invariant feature extraction method to obtain domain-invariant features of architectures. Moreover, this paper provides sufficient ablation studies for each module/element of the proposed method. Nevertheless, both the novelty of the learning operation embedding and the application scenario of the proposed method are very limited.", "summary_of_the_review": "The application scenario of the proposed method seems very limited.", "strength_and_weaknesses": "Strengths:\n1. This paper develops a new NAS method that trains a NAS model and a regressor model in an adversarial scheme.\n2. The proposed progressive domain-invariant feature extraction method is interesting.\n\n\nWeaknesses:\n\n1. There is a typo in the abstract. The accuracy on ImageNet should be 76.50% instead of 96.50%.\n\n2. As highlighted in the paper, one of the contributions is the learnable operation embedding. However, it is not novel because almost all Reinforcement Learning (RL) based NAS methods exploit a learnable embedding for each operation, such as ENAS. Please further clarify it if there are some other differences from what is used in ENAS.\n\n3. The motivation for reducing the divergence between source and target search space seems questionable. For example, in Figure 1, the architectures in source and target domains may share exactly the same or similar architecture. In other words, the discriminator would definitely fail in this case since it cannot distinguish between two identical/similar architectures. Instead, what should be bridged/distinguished between two spaces is the accuracy y instead of the architecture x.\n\n4. The application scenario of the proposed method seems very limited. According to the paper, this method can be only used on the search space of DARTS. Given another search space, e.g., MobileNet based space used in OFA, is the proposed method still applicable in this case?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "CZxGLSnlRp", "reviewer": "Reviewer_4ZdJ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a surrogate model that predicts the accuracy of an architecture.  The surrogate model has access to the architectures and their corresponding accuracies from the source search space. It is tasked with predicting the accuracies of architectures that are sampled from the target search space. It uses adversarial training for unsupervised domain adaptation. A Graph convolution network is used to extract the latent features from the architectures and serves as a generator. The discriminator in turn tries to differentiate the domain of the architecture.  A fundament layer F models the entire feature space.  A bridge layer B captures the domain specific features. The difference between the $F(x_{i})$ and $B(x_{i})$ gives the domain invariant features. The domain invariant features are fed to a predictor layer which in turn predicts the accuracy. \n\nThe overall loss function is a combination of the domain loss from the discriminator, the bridge layer loss and the regression loss from the predict layer. The input to the surrogate model is an adjacency matrix. It learns an embedding for the operations of the architecture. ", "summary_of_the_review": "Domain adaptation to learn a surrogate model for NAS is very useful for practical purposes. But the paper is mainly adapted from \"Gradually Vanishing Bridge for Adversarial Domain Adaptation\". Using operations for embeddings has also been done in the past. So the novelty is limited.", "strength_and_weaknesses": "**Strengths**:\n1. The paper is able to learn a surrogate model on the target search space without any training data\n\n**Questions**:\n1. Could you evaluate the quality of the domain invariant features that are learnt? For example, use the source domain as nasbench-201 and target as Darts, source domain as nasbench-101 and target as nasbench-201 and compute the distance (KL-Divergence) or similarity (cosine similarity) between the learnt domain invariant features using source domain 1 and source domain 2.\n2. Could you also include a table for the regression error and the domain classification  accuracy?\n3. Is USPP DANN + bridge layer. If so, in the ablation study, when the bridge layer is removed, why is the kendall tau different from DANN? For table 5, are you using operation embeddings for all the baselines?\n4. For table 5, rather than training the architectures for darts search space, why not use NDS-Darts [1] benchmark? Evaluate it on 100 architectures sampled from that search space.  Similarly, for table 2, if you could actually use NDS-Darts to find the best architecture and report the numbers for other algorithms also from the same benchmark, it would be a fair comparison.\n5. Please include darts-pt [2] also in table 1 and 2.\n\n[1] On Network Design Spaces for Visual Recognition, Radosavovic et al.\n[2] Rethinking Architecture Selection in Differentiable NAS, Wang et al.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4192QAOgJZM", "reviewer": "Reviewer_8UES", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed an unsupervised (UDA) performance predictor called USPP, which aims to mitigate the architecture performance prediction gap between source and target domain. ", "summary_of_the_review": "The overall quality of this paper is good. The proposed method has promising results on the setting of transferring NAS101 to DARTS search space. I would expect the author to conduct more transfer settings to validate the effectiveness of the proposed method. I would like to recommend this paper to score 6 temporarily.", "strength_and_weaknesses": "Pros:\n1. The paper is well written and easy to follow.\n2. The proposed method is technically sound. \n3. The idea of using UDA and adversarial training to address the domain gap issue in NAS is smart.\n\nCons:\n1. The name of the paper is a little bit confusing at the first glance. I thought it is a method related to unsupervised learning, but turns out to be related to Unsupervised Domain Adaptation.\n2. Missing recent baselines in the result table[1][2].\n3. The author uses GPU day as a metric to evaluate searching speed. My question is that are the compared methods using the same GPU? If not, then they are not comparable.\n4. The author only report one domain transfer result (NAS101 --> DARTS). I would expect to see more domain transfer case, like NAS101 --> NAS102,NAS102 -->DARTS etc.\n\n[1] Mellor, J., Turner, J., Storkey, A., & Crowley, E. J. (2021, July). Neural architecture search without training. In International Conference on Machine Learning (pp. 7588-7598). PMLR.\n[2] hen, W., Gong, X., & Wang, Z. (2021). Neural architecture search on imagenet in four gpu hours: A theoretically inspired perspective. arXiv preprint arXiv:2102.11535.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject