Bridge the Gap Between Architecture Spaces via A Cross-Domain Predictor
Authors: Yuqiao Liu
Yehui Tang
Zeqiong Lv
Yunhe Wang
Yanan Sun
Venue: NeurIPS 2022 Accept (2022)
Keywords: neural architecture search
neural predictor
domain adaptation

Abstract: Neural Architecture Search (NAS) can automatically design promising neural architectures without artificial experience. Though it achieves great success, prohibitively high search cost is required to find a high-performance architecture, which blocks its practical implementation. Neural predictor can directly evaluate the performance of neural networks based on their architectures and thereby save much budget. However, existing neural predictors require substantial annotated architectures trained from scratch, which still consume many computational resources. To solve this issue, we propose a Cross-Domain Predictor (CDP), which is trained based on the existing NAS benchmark datasets (e.g., NAS-Bench-101), but can be used to find high-performance architectures in large-scale search spaces. Particularly, we propose a progressive subspace adaptation strategy to address the domain discrepancy between the source architecture space and the target space. Considering the large difference between two architecture spaces, an assistant space is developed to smooth the transfer process. Compared with existing NAS methods, the proposed CDP is much more efficient. For example, CDP only requires the search cost of 0.1 GPU Days to find architectures with 76.9% top-1 accuracy on ImageNet and 97.51% on CIFAR-10. 

Bridge the Gap Between Architecture Spaces via A Cross-Domain Predictor
Yuqiao Liu", Yehui Tang Zeqiong Lv Yunhe Wang", Yanan Sun
CCollege of Computer Science, Sichuan University
School of Artificial Intelligence, Peking University Huawei Noah's Ark Lab
1ygguitardgaiil.cm com, yhtangāpku. edu. en, lzq_june@foxmail com, yunhe. wanghhuawednu com, ysundacu. edu. cn
Abstract
Neural Architecture Search (NAS) can automatically design promising neural architectures without artificial experience. Though achieves great success, pro- hibitively high search cost required to find high-performance architecture, which blocks its practical implementation. Neural predictor can directly evaluate the performance of neural networks based their architectures and thereby save much budget. However, existing neural predictors require substantial annotated architectures trained from scratch, which still consume many computational sources, To solve this issue, we propose a Cross-Domain Predictor (CDP), which is trained based on the existing NAS benchmark datasets (e.g.. NAS-Bench 101), but can be used to find high-pete nigh-performance architectures in large-scale search spaces. Particularly, we propose progressive subspace adaptation strategy address the domain discrepancy between the source architecture space and the target space. Considering the large difference between two architecture spaces, an assistant space is developed to smooth the transfer process. Compared with existing NAS methods, the proposed CDP is much more efficient. For example, CDP only requires the search cost of 0.1 GPU Days to find architectures with 76.9% top-1 accuracy on ImageNet and 97.51 on CIFAR-10. The source code will be available
Introduction
Neural Architecture Search (NAS) [6] can automatically design promising architectures of deep neural networks for real-world applications, such as image classification, language modeling, and medical image segmentation [50 46 62) In recent years, there have been many architectures searched by NAS methods, which can surpass the performance of those manually designed [43] 23 Generally, performing NAS methods often requires thousands of GPU Days. Unfortunately. such a large amount of computing resources is not necessarily available to most researchers and enterprises interested. As result, NAS researchers are paying increasing attention to improve the efficiency of NAS through developing effective acceleration methods, complete the search within an acceptable cost 34
Generally speaking. most of the existing search acceleration methods developed based some heuristic strategies, e.g., reducing the size of search space [65] [5], searching upon a smaller proxy dataset 43 33), and sharing the parameters of different architectures (weight sharing)
44 42
Equal contribution.
Corresponding author:
(Mnnar/ (MindSpore)
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

However, the shortcomings of the above methods are obvious. Reducing the size of search space also limits the diversity of architectures [52], and the architectures optimized on the proxy dataset are not guaranteed to perform best on the target dataset [4] Weight sharing first trains supergraph that is manually designed in advance and then searches subgraphs as the potential architectures in the supergraph. The weights in the same components between different architectures are shared from the trained supergraph without further training, thus the efficiency of performing of NAS is greatly improved. However, the weight sharing also suffers from performance instability and dramatic performance collapse B 6., which severely limits its application.
Neural predictor is promising acceleration method, which directly estimates the performance of neural architectures without any training of the neural network during NAS [55]. For example. Sun et al. [49] proposed an end-to-end performance predictor that employed random forest [2] which reduced the consumed time of NAS significantly. Tang el al. (54) used a graph convolution network (GCN) as the neural predictor and fully exploited the unlabeled architectures to improve the prediction accuracy. Though these predictor-base methods tend to be more efficient than other methods, sufficient architectures which play the annotated examples to train the predictor, are still required to sample from the target search space. These sampled architectures are usually required to train fully to obtain their actual performance 555 49]. A large search space requires mass of samples, which becomes a bottleneck for further improving predictor predictor-based methods. Consequently, neural predictor still requires massive computing costs largely due to annotating the sampled architectures.
Fortunately, we note that many benchmark datasets have been released for NAS research, such as Be--ch-nch11 [60] NAS 20- 13] NAS-Bench-nlp [29]. NAS-Bench-ASR [3 etc. These datasets contain a great number of architectures with known real performance, and constructing them has consumed massive computational resources. If these data can come in handy, there is no need to annotate other architectures for predictors and the search cost is greatly reduced.
However, the search spaces of these datasets are small and have huge difference from the large search spaces adopted by popular NAS algorithms such as DARTS (33) and ProXylessNAS [ Although the architectures are all made up of operations, the settings are quite different, such as the types and the maximum number of operations. As a result, these well-trained benchmark architectures cannot be directly used to build predictors.
In this paper, develop Cross-Domale Predictor (dubbed as CDP) by exploiting the existing NAS benchmarks, and then the predictor is implemented to find high-performance neural architectures in large search spaces.
The contributions of CDP are summarized as follows: 1) We divide the architecture space for subspace adaptation, and propose a progressive subspace partition to accurately exploit the local information. 2) We design an assistant space. to achieve smoother transfer from the source space to the target one because of the significant differences between searching domains. 3) We experimentally show that CDP can search promising architectures with only 0.1 GPU days on ImageNet and CIFAR- 10, achieving 76.9% and 97. 77.11 % classification accuracy, respectively.
2 Related Works
2.1 NAS and Neural Predictor
Target Architecture
Existing Methods: Annotale sufficient archibectures
NAS
0,83
Space
Space
In general. there are three important parts in NAS [66. namely architecture space" search strategy, and performance evalua- tion. During the search process, each NAS method first specifies the architecture space that includes all the potential architectures to be searched. Secondly, a particular search strategy utilized to perform the search. In practice, reinforcement learning [26), evolu- tionary computation [ and maddient-based methods are the mainstream search strate-
0.76
Select
Train
0.82
Search Strategy
Neural Predictor
Train
Method: Use existing
Predicted value 0.60
massive laboled archutectures
Figure I: Neural predictor-assisted NAS. Existing neu- ral predictors need to annotate sufficient architectures from the target space, which incurs a high computa- tional cost. Our method uses existing NAS benchmark datasets where massive available labeled architectures can be used to train the neural predictor. It is also named as search space in some literature, in this paper, treat them as the same.

---

gies widely used. Finally, the performance evaluation assesses the architectures searched to guide the search strategy.
In recent years, how to accelerate the performance evaluation is one of the most valuable research directions in NAS 34 As a promising acceleration method, neural predictors have received great attention from researchers since 2017 [II. The red and black parts in Fig, | illustrate a typical neural predictor-assisted NAS for a glance. Recently, with the design of new predictors [49 54 55 the
prediction performance of neural predictors has been continuously improved. However, existing predictors request the source and the target architecture spaces to be the same. If the predictor makes predictions for specific NAS method, it must collect sufficient labeled architectures from the search space of the corresponding NAS method in advance. The CDP algorithm we proposed in this paper gets rid of the constraint that the source and the target architecture spaces must be the same. Particularly, CDP directly utilizes the existing benchmark datasets as the training architectures the source space, yet to predict the architectures in other target spaces. In this way, the neural predictor will not need to sample labeled data from the target space of the partieular NAS method, thus greatly reducing the cost of annotating the training architectures.
2.2 Transfer Learning and Domain Adaptation
Transfer learning is to apply knowledge learned in previous domains to other domains, thus reducing the need and effort recollect labeled data 40. Specifically, domain adaptation a subcategory of transfer learning, where the source and target tasks are the same and labeled data is available in the source domain, which is the scenario of CDP. According to [64,, transfer learning can be classified into two different categories based on the consistency between the source and the target domains. If the feature spaces are different in both domains, the scenario is termed heterogeneous transfer learning. Otherwise, it is homogeneous transfer learning. The proposed CDP algorithm in this paper follows heterogeneous transfer learning because the architecture spaces are different. But observing that much more current literature addresses the scenario of homogeneous transfer learning [10], we proposed a uniform encoding strategy that unifies the architecture spaces, so that can be treated as homogencous transfer learning.
The feature-based transfer is one of the most prevalent methods. It aims at reducing the distribution difference between the source domain and the target domain with the help of some instances. A common metric for measuring the difference is Maximum Mean Diserepancy (MMD) [18] which can quantify the difference by calculating the average distance of the instances sampled from both domains (64). Inspired by MMD, many variants and improved approaches are proposed such as MK-MMD 19). weighted MMD [59], and Local Maximum Mean Discrepancy (LMMD) [63 Among them, LMMD can take advantage of local affinity for more delicate adaptation and can achieve extraordinary results. Specifically, LMMD first classifies the domain into several subspaces and then aligns the subspaces of the same category in pairs. Secondly, LMMD uses the pseudo-labels output by the neural network to classify the target data throughout the training process. However, it is well known that the neural network cannot achieve satisfactory results in the initial training stage, that is, the pseudo-label obtained through it may be seriously inconsistent with the real label in the early training process. In this paper, we propose progressive approach to alleviate the disadvantages of misclassification.
Approach
In this section, we will document how the proposed cross-domain predictor bridges the gap between different architecture spaces. The overall process of the proposed CDP algorithm is shown in Fig. 2
3.1 Formulation
Supposing the architectures in the search space X are X = X2, ..., }, and their respective
performances in space ) are Y = (y1, U2,,.,,, Un ). the predictor P X - V, which is trained to predict the performance of architecture, can be formulized by Equation (1)
min C(P(W, X,. X,),,n) W
(I)
11 n=1
3

---

GCN Predictor
FC layers
Progressive Subdomain Partition
9
Early epoch
Later spoch
Assistant
Uniform
Space
Encoding
PSP
Elements in Adjicency Matrix
Operation Features
Architecture Performance Value
The CDP flowchart for evaluating neural architectures. The blue and red dashed circles represent the source architecture space and target architecture spaces, respectively. The size of assistant space between those of source and target spaces, and this space is used in early training for a smoother transfer. Architectures from these spaces are required to go through the uniform encoding to be represented by adjacency matrix M and operation features Og. Note that the true label of source architecture data and the pseudo-label of target architecture data, i.e., y and ø need to go through the proposed Progressive Subspace Partition (PSP) strategy first, and then are used to calculate LMMD. The detailed figure of PSP is displayed on the right. In the beginning, this strategy divides all architectures into the same space. As the neural network training progresses, the number of subspaces will continue to increase until it finally reaches the specified maximum number. The partition of subspace is based on true and pseudo labels.
Figure 2:
where W denotes the trainable weight of predictor P, and L is the loss function. After the training, the predictor with trained weight W will evaluate other architectures, say x; where i is not in {1, ..., n), via Equation (
U: P(W,x.).
(2)
Mathematically, neural architectures can be presented as directed graphs, and a common encoding scheme of architecture is to feature vectors of operations On and adjacency matrix M, which are further collectively employed to represent the type of each operation and the connection between operations. GCN [28] is prevalent technique tackling -on-Euclidean data such as graphs and has achieved great success in representing neural architectures (55). Therefore, it is used as the first component in P extract features of neural architecture G X Z. where the feature space.
In this paper, we follow the convention and choose the GCN version implemented in [55] as the neural predictor, and the feature vectors are updated by Equation 33
1 1 RcLU(MO,,;)) / RROLU(M"
O4 2
(3)
where M is the transposed matrix of M and ReLU is an activation function. Wi+ and W are two trainable weights of the 1-th layer in GCN. Specifically, the last two terms of Equation (3) allow the graph information to flow forward and backward, respectively, which makes it more suitable for dealing with directed graphs like neural architectures. In addition, the second component of P is hypothesis h Z + y, consisting of several fully connected layers, and is tailed to the GCN for obtaining the predicted performance values. As for loss function L, we also follow the convention suggested in the reference (55] using mean square error.
3.2 Progressive Subspace Adaptation
Although neural predictor can largely save search cost for NAS, the limitation of itself is also serious: it cannot be well trained due to the lack of available labeled architectures of the corresponding NAS algorithm. Observing that many architecture datasets have been released, these data can be potentially used to train the neural predictors. However, the distribution of these massive labeled architectures is significantly different from that of the architectures from the popular search space that often large. In principle. domain adaptation is useful for reducing the distribution gap between these architecture spaces. Let D& and Dr be the source and target distributions over Z. We will specifically introduce how to bridge the gap between D& and Dr in the proposed method as below.
Research has shown that deep neural networks have the ability to learn the transferable features well [1 39]. In addition, many methods [3| 63 add metrics of discrepancy to the objective function

---

to jointly optimize the neural network. Based on Equation T the term of adaptation regularizer d(.,)) is added, and the objective function becomes:
1 ()) Od(Ds, Dr),
min
(4)
where x,, and yn denote architectures with labels in source search space Xs, and 0 > is penalty parameter.
Maximum Mean Discrepancy (MMD) 18 is a classic metric and has been widely used in domain adaptation. Let H4 denote the reproducing kernel Hilbert space (RKHS) endowed with a characteristic kernel k. and Us, Ut are samples of sizes ns n, drawn from Ds, Dr respectively. The regularizer term d(::-) .) can be implemented by MMD), and the unbiased estimator of MMD is defined by latent representation zz, zz
(Ds,, Dr) -
Σ ((Z)) E
Σ Thg
(t)))) E
(5)
where denotes feature map function mapping original space Hs and it is associated with k(z", x²) = (ő(z"), o(z))) where (+-)) means the inner product operation.
Considering that the differences between search spaces quite large, the gap cannot be handled well if only MMD used. In addition, MMD is global domain adaptation, which often ignores the local relationship. In order to take full use of the local information between architectures, present progressive subspace adaptation strategy. First, the source space and the target space are divided into several subspaces whose number will increase during the training process. Secondly, MMD loss is used between corresponding subspaces for adaptation. Specifically, these two spaces will be divided according to the performance Y {y1 1/2 -... Un) of architectures. Between these two spaces, the architectures with good performance may share some characteristics, such as long shorteut connections and preferred operations.
We directly use the true labels y of architectures in the sources space and predict the pseudo-labels ÿ of architectures in the target space to divide the subspaces. The performance value y; is a scalar, which needs to classified first to become a classification label y; that is a vector. Secondly, the weight w which presents the z; belonging relationship the e-th subspace can be calculated via Equation (6):
W
(6)
Ex VG'
where of denotes the e-th element label vector y,. With the weight wf and Equation ( the unbiased estimator of the discrepancy between subspaces becomes Equation 7
(Ds. DT) A
Σ Σ (Z))
Σ E/Eū
C zeeis
This discrepancy is termed Local Maximum Mean Discrepancy (LMMD) 63 that takes the correla- tion from the same category of subspaces into consideration.
With the help of the kernel k. Equation 7 can be further modified to obtain the following formula for easy calculation of LMMD:
de Dr) 1
w kz
c=1
i=1
(8)
w w 10
kz, 2))]
1=1
However, there is a severe issue that the pseudo-label y may be incorrect, especially when the neural network is not fully trained. This will further affect the accuracy of du (Ds, Dr) in Equation 8
5

---

In order to alleviate the misclassification due to inaccurate pseudo-labels, we propose a progressive subspace partition strategy. All the architectures are partitioned into one space in the early stage of neural network training so that there will be no classification errors caused by incorrect pseudo labels. As the training progresses, the neural network can predict more accurately. At this time, we slowly increase the number of subspaces. Finally, the number of subspaces will end at the specified maximum K.
Progressive Subspace Partition: Supposing the neural network is specified be trained for a total of E epochs, and the maximum number of subspaces is K, when the training progresses arrive at the e-th epoch, the number subspace categories C, can be scheduled by Equation (
Ce Schee(; K),
and Vi < Schelii; K) < Scchej; E, K). Based on experience and experiments, in this paper we let C. = K Icos( e) KJ, where 1.] denotes the round-down operation. Ce calculated by this schedule meets the characteristies discussed above, i.e., when is relatively small, C. equal to I, while is close to E, C. will be K. Furthermore, we use the cosine function to increase the number of epochs with a small value C.. This strategy ensures that the neural network has been sufficiently trained before the Ce becomes larger. Therefore, the occurrence of classification errors will be greatly reduced.
After determining the number of subspace categories C. during the e-th training epoch, we can use fractile to generate new label y for the original label y which is scalar. To be more specific, the fractiles {fo, f1, f2...., sc.) } of scalar labels need to be found at first, where fc. is the largest one in the labels and fo is the lower bound. Then, we can find the corresponding subspace category e for y: if fa-1 < yi I fe. After that, we set the c-th element of y; to one and other elements as zeros. Finally, the classification vector y, is used by Equation (6) for the calculation.
Here, we will further analyze the role of the proposed progressive subspace partition strategy. When the number of subspaces one, the numerator yf in Equation (6) is always one, and the denominator E,,ey Vf is equal to the number of source (or target) data. Hence, w." is equal to 1/n" and w/eh is equal to 1/n'.. As a result, the unbiased estimator of LMMD (Equation (7) becomes that of MMD (Equation (5)) which is global adaptation. While the number of subspaces increases to the specified maximum number K, the measurement of the discrepancy we used is actually LMMD which focuses on the relationship between the same categories of subspaces to help the subspace adaptation.
3.3 Assistant Architecture Space
Considering that the source architecture space and target architecture space are quite different, is difficult to directly transfer the source archi- tecture space to the target space. To address this issue, we design an assistant architecture space to make the transfer smoother.
Source Spaces
Assistant Space
Target Space
NAS Hen--Hncb-Ja 101 NAS Bench 201
Tiny DARTS
DARTS
Small
Middie
Large
Figure 3: An example of assistant space. Generally, the source space, where all architectures can be trained from scratch obtain the per- formance of each architecture, is often relatively small. However, the target architecture spaces are often large in practice. An example is given based on two widely used architecture spaces, namely NAS-Bench-10
[60] and DARTS K3. In terms of the number of possible architectures, NAS-Bench- 101 423K and DARTS about 1018, terms of the maximum number of operations in cell, BASS-ennch-10 is 5 while DARTS is 8. Thus, the size of the assistant space should be between the size the source space and the target space. Besides, other features should be similar to these two architecture spaces. In this paper, we specify the assistant space more similar to the target space.
Specifically, an assistant space for NAS-Bench 101, NAS- MAS--ench-20 2011 following the above principles, dubbed as 'tiny DARTS' space. Fig. B is an illustration of this assistant space, where we use dark black to represent the large architecture space and gray to represent the smaller one. The participation the assistant space makes the adaptation be a continuous process and alleviates the inadaptability caused by large differences. The only difference between tiny DARTS and DARTS is that the number of intermediate nodes has changed from to This directly reduces the number of operations from 8 to 6 which is equal to the situation in TAS-Bench-201. Except
13 and DARTS are designed

---

Theorem shows that the expected error on the target domain is bound by two terms, the empirical validation error in source domain és, and MMD da (Dss Dr) which is minimized as one term of the objective function. This means that when observing the predictor performs well on the source for this change, any other settings, including the types of operations, etc., have not been changed. Therefore, the features in DARTS have also been preserved. Specifically, there are two input nodes in tiny DARTS, and all intermediate nodes are connected to the output node.
In the early training epochs of CDP, the assistant space (e.g., tiny DARTS) is used the target domain, i.e., the input at the bottom in Fig. 2 In this stage, CDP focuses on bridging the smaller gap between the source space and the assistant space. After several training epochs, the gap between the source space and the target space can be also reduced as the assistant space and target space are very similar. Therefore, it is much easier reduce the gap between the source and target spaces at this time. As result, the space transfer process becomes smoother with the assistant space, which improves the prediction performance.
After the determination the target domain, an encoding strategy is required for different search spaces. We propose uniform strategy with four main steps, where The detailed descriptions shown in Appendix A
3.4 Theoretical Analysis
In this section, we give a theoretical analysis of the expected error of target architecture search space. We first introduce Lemma [ which is a basic theorem in domain adaptation, and then further deduce the target error bound of CDP.
Let (h) and es(h) be the expected error on target and source domain and H. be hypothesis space, for h E Hi:
Lemma 1.
[3]
(T(h) S Ee((h) (D.
(10)
where À = er(h") + es(")) the combined error of ideal hypothesis h* arg (er (h)) on both domains and (D,, DT) is the upper bound of the A-distance (27).
Let Üs Usttrann: Us,valid be the training and validation datasets randomly sampled from the distribution Dy respectively, and Usarain n Udg = After the predictor is trained on the training dataset Ug.trains the validation dataset Ug is used to validate it. The validation hypothesis 7t' is subset of H. and only determined by Usstrain The following theorem provides upper bound of expected torget-domain error in terms of the empirical error on Ussyatid and MMD dy (Dg, Dr)-
Theorem 2.
Let d' be the VC-dimension of H', m be the size Us, m' be the size of unlabeled samples Us and Üp. With probability of 1 8, for h E H':
cr (h) S &S,vaaia (h) 2d (Ds. Dr) + 2(d' log m-log
3m
(11)
2(d' log log 6) d' log(2m') log( +
4V
m
m
Proof sketch: The proof is mainly divided into two steps. We first use the empirical error on source validation dataset (h) to represent the upper bound of expected error on source domain (h):
(h) S vatiaaa +
2(d' log m m-
2(d' log maaggm-oogg log
(12)
3m
The detailed proof can be found in Equations 19
24p in Appendix Secondly, the upper bound of
the distance du (Ds. Dr) is constrained by MMD du (Ds, Dr), i.e.:
dx DT) 2 2dh (5s, DT) 4V
d' log(2m²)+1ogff
(13)
mz
26 in Appendix BranndxxBsshww... Bhsoow the proof in detail. Finally, by applying these inequalities to Lemma] the theorem can be obtained.
and Equations (55
validation dataset, the expected error on the target domain will also be small.
7

---

Table Comparison with state-of-the-art and neural predictor-assisted NAS methods on ImageNet. We report the results with the lowest cost if there multiple architectures searched in the literature of the compared methods. The symbol denotes that no results are available in original papers.
Model
Cost (GPU Days)
Top-1 (%)
Top-5 (%)
FLOPs (M)
# Params (M)
NASNet-A A
65
2000
74.0
91.6
564
5.3
AmocbaNet-A
43
3150
74.5
92.0
555
5.1
DARTS 33
4
73.3
91.3
574
4.7
MobileNerV3
23
#3000
75.2
92.2
219
5.4
EfficientNet B0
53
==3000
76.3
93.2
390
5.3
FairDARTS-B
0.4
75.1
92.5
541
4.8
PC-DARTS [58
0.1
74.9
92.2
586
5.3
MoGA-A
12
75.9
92.8
304
5.1
DropNAS 22
0.6
75.5
92.6
572
5.2
DOTS (20
0.2
75.7
92.6
581
5.2
NAO 17
200
74.3
91.8
584
11.35
Neural Predictor 55
119 Archs
74.7"
SemiNAS R6
4
76.5
93.2
599
6.3
WcakNAS 56
2.5
76.5
93.2
591
5.5
CDP (Ours)
0.1
76.9
93.0
548
5.4
The search cost is 119 architectures trained ImageNet, measured in GPU Days. An approximate value estimated from figure its original paper.
4
Experiments
In this section, we will empirically investigate the effectiveness of the proposed method, including the experiments conducted on ImageNet [12 and CIFAR-10 (30), and the ablation studies. Our experiments are conducted with Py'orrch and MindSpore 25
4.1 Experimental Settings
Architecture spaces for CDP:
and NAS- JAS-Bench-201 are chosen as the source spaces and DARTS the target space. The training labels NAS-Bench-10) and HAS-Bench-200 are normalized separately. This can alleviate the problems caused by different performance scales.
Search strategy: Following the convention in neural predictor [55], large of architectures in the target search space are randomly sampled and their performance is predicted. addition, according to the previous work DARTS search space (3), we limit the max number of skip connections to 2. After that, we choose the architecture with the best-predicted performance as the searched architecture.
4.2 Search Results on ImageNet and CIFAR-10
The experimental results of ImageNet are shown in Table I where the competitors all the state-of-the-art NAS methods (shown in the second row) and neural predictor- assisted NAS methods (shown in the third row). As can be seen from Table [ the pro- posed CDP method only consumed 0.1 GPU Days to complete the search, and it is of the least costly search methods. Specifically, compared with the chosen neural predictor- assisted NAS methods, the search cost of CDP is x000 less than NAO, 40x less than SemiNAS, and 25x less than Weak- NAS. The main reason for the extremely low cost of CDP lies in that CDP can make full use of a large amount of existing data from the benchmark datasets. In addition, although CDP consumes similar budget to that of PC-DARTS which is the state-of-the-ee NAS method using weight sharing, CDP achieves the best top-1 accuracy. The searched architectures displayed in Appendix C
Table 2: Comparison with state-of-the-arr NAS meth- ods on CIFAR-10. GPU Days is used to measure search cost. 'Ace.' denotes the accuracy and 'Params' denotes the parameters.
Model
Cost
Acc. (%)
Params (M)
NASNet-A
2000
97.35
3.3
ENAS
0.5
97.11
FairDARTS
0.4
97.46
2.8
P-DARTS
0.3
97.50
3.4
PC-DARTS
0.1
97.43
3.6
PVLL-NAS
0.2
97.30
3.3
AmoebaNet-A
3150
96.66 1 0.06
DARTS
4
97.24 1 0.09
3.3
BRP-NAS
6
97.29 T 0.07
97.51
CDP (Ours)
0.1
3.3
97.37 1 0.08

---

Table 5: Ablation study on do- main adaptation approaches.
Table 3: The impact of differ- ent progressive strategy (PS).
Table 4: Ablation study assistant space (AS).
Approach
KTTau
PS
KTau
AS
Size
KTau
DANN 1171
0,4686
None 0.3650
0.5176
CORAL 48
0.4306
Linear 0.4949
Small
0.5051
MMD 35
0.4549
Cosine
0.5176
Medium
0.5306
LMMD
0.4969
LMMD + PSP
0,5306
The comparison results of CIFAR-10 are shown in Table In this experiment, in addition to the methods compared in Table I, we also compare with three other state-of-the-an methods. They are ENAS 42), P-DARTS (5), PVLL-NAS [31], and predictor-assisted NAS method BRP-NAS (14) The parameters of all comparison methods are similar. In terms of the search cost, CDP requires only 0.1 GPU Days, which is one of the lowest cost methods in the table. In addition, the architecture searched by CDP also outperforms all the NAS methods compared.
4.3 Ablation Study
To the best of our knowledge, CDP is the first work of neural predictors designed from the cross- domain view, by directly exploring the architectures of existing NAS benchmark datasets. Therefore, it is not practical to conduct any experiments on NAS-Bench- 101 and NAS-Bench- To deal with it, we create a new dataset for neural predictor termed shallow DARTS by randomly sampling 100 shallow architectures from DARTS search space. order to save training cost, CIFAR- 10 is chosen as the dataset, and these architectures have only 8 layers which are shallower than the architectures reported in Table 2 Moreover, we train these shallow architectures for 50 epochs to further save cost.
by tradition.
After CDP has been trained on performance of 100 architectures in the shallow DARTS dataset. Following the conventions, we also pay more attention to the relative ranking between the architectures. Therefore, we use Kendall's Tau (KTau) [45] which is widely used to measure the quality of neural predictors [49] (54) in this ablation study. KTau concerns the relative ranking between the architectures, and its values vary in [-1, 1]. The larger the value, the more the predicted ranking matches the real ranking.
101 and NAS-Bench-201, it is used predict the
Table 3 displays the results with and without the proposed progressive subspace partition strategy. Specifically, we specify that the numbers of subspaces are all 3 in the end. 'None' denotes no progressive strategy is used. In addition, progressive subspace partition with the linear increasing rate in Equation (9) is also tested to show the effectiveness of the cosine function. Generally, the performance of the predictor is much better when working with the proposed progressive subspace partition strategy. Furthermore, the cosine function also works better than the linear increasing rate. In summary, the ablation study proves the effectiveness of the proposed progressive subspace partition.
The results the ablation study on the assistant space are reported in Table 4 The KTTuu value of the first line is obtained with the help of the progressive subspace partition. On this basis, two assistant spaces are added separately. The first one is similar to DARTS and the number of internal nodes is set as 2, which makes its size be smaller than both the source and target architecture spaces. The second is the tiny DARTS whose size is between the source and target architecture spaces. As can be seen from Table 4 the small assistant space fails to improve the performance of the predictor. But with the help of tiny DARTS (the medium space), the prediction performance can be further improved on the basis of the progressive subspace partition.
Finally, we conduct an ablation study on domain adaptation approaches. In addition to the MMD and LMMD introduced above, adversiiadvesariilll neural network (DANN) [17] and CORrelation ALignment (CORAL) 48 are also compared. We fine-tune the hyper-parameters of these approaches (if feasible), and report the best KTau values they achieved in Table 51 It is obvious that our approach is far better than peer competitors.
To sum up, the ablation studies prove that the proposed progressive subspace partition and assistant space can significantly improve the prediction performance of CDP, Other ablation studies can be found in Appendix E

---

5 Conclusion
This paper aims to develop a cross-domain predictor, named CDP, which does not require training neural architectures in the particular search space. This goal has been achieved by the two proposed components. Specifically, the progressive subspace partition has been proposed to largely alleviate the issue of inaccurate pseudo-label. In addition, the assistant space has been developed address the inadaptability caused by large differences between source and target domain. The training architectures of CDP are collected from the existing NAS benchmark datasets, it does not have to spend a lot of computation cost on training neural architectures as the existing neural predictors do. CDP can find high-performance architectures in the large search space (such as DARTS) with 0.1 GPU Days and achieves the classification accuracy of 76.9% and 97.51 % on ImageNet and CIFAR-10, 10, respectively.
Acknowledgments
This work was sponsored in part by Zhejiang Lab (NO.2022GGOAB02) in part by CAAl-Huawei MindSpore Open Fund, and in part by the Science and Technology Innovation Talent Project of Sichuan Province under Grant 2021JDRCO001. We gratefully acknowledge the support of MindSpore, CANN (Compute Architecture for Neural Networks) and Ascend AI Processor used for this research.
References
[1] Thomas Bäck. David B Fogel, and Zbigniew Michalewicz. Handbook of evolutionary computation.
Release, 97(I):BI, 1997.
[2] Irwan Bello, William Fedus, Xianzhi Du, Ekin D Cubuk, Aravind Srinivas, Tsung- Lin, Jonathon Shlens, and Barret Zoph. Revisiting resnets: Improved training and scaling strategies. arXiv preprint
wrXiv:200300555 2021.
[3] Shai Devid, John Blitzer, Koby Crammer, Fernando Pereira, et al. Analysis of representations for
domain adaptation. Advances in neural information processing systems, 19:137, 2007.
[4] Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search on target task and hardware. In 7th International Conference on Learning Reprexentations, ICLR 2019, New Orleans, LA. USA, May 6-9. 2019. OpenReview.net, 2019.
[5] Xin Chen, Lingxi Xie, Jun Wu, and Qi Tian. Progressive differentiable architecture search: Bridging the depth gap between search and evaluation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1294-1303 1333, 2019.
[6] Xiangxiang Chu, Xiaoxing Wang, Bo Zhang, Shun Lu, Xiaolin Wei, and Junchi Yan. DARTS-; robustly stepping out of performance collapse without indicators. In 9th International Conference on Learning Representations, ICL, 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. [7] Xiangxiang Chu, Bo Zhang, and Ruijun Xu. Moga: Searching beyond mobilenetv3. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4042-4046. IEEE, 2020.
[8] Xiangxiang Chu, Tianbao Zhou, Bo Zhang, and Jixiang Li. Fair darts: Eliminating unfair advantages in differentiable architecture search. In European conference on computer vision, pages 465-480. Springer, 2020.
[9] Ekin D Cubuk, Barret Zoph. Dandelion Mane, Vijay Vasudevan, and Quoc Le. Autoaugment: Learning augmentation strategics from data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 113-123, 2019.
[10] Oscar Day and Taghi M Khoshgoftaar. A survey on heterogeneous transfer learning. Journal of Big Data,
4(1):1442, 2017.
[11] Boyang Deng, Junjie Yan, and Dahua Lin. Peephole: Predicting network performance before training.
2017. [12] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and puttern recognition, pages 248-255. Ieee, 2009.
arXiv preprint 1:Xivv/712.03332
[13] Xuanyi Dong and Yi Yang. Nas-hench-201: Extending the scope of reprodueible neural architecture search.
In International Conference on Learning Representations (ICLR), 2020.
[14] Lukasz Dudziak, Thomas Chau, Mohamed Abdelfattah, Royson Lee, Hyeji Kim, and Nicholas Lane. Brp-nas: Prediction-based nas using gens. Advances in Neural Information Processing Systems, 33: 10:10- 10490, 2020.
[15] Thomas Elsken, Jan Hendrik Meizen, and Frank Hutter. Efficient multi-objective neural architecture search
via lamarckian evolution. In International Conference on Learning Representations, 2018.
[16] Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. The Journal of Machine Learning Research, 20 1091779997-0011 2019.
10

---

[17]
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette.
Mario Marchand, and Victor Lempitsky. Nomain-adversarial training of neural networks. The journal of machine learning research, 170))2209-2033, 2016.
[18] Arthur Gretton, Karsten M Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel iwo-sample test. The Journal of Machine Learning Research, 13(1):223-73, 2012.
[19] Arthur Gretton. Dino Sejdinovic, Heiko Strathmann. Sivaraman Balakrishnan, Massimiliano Pontil, Kenji Fukumizu, and Bharath K Sriperumbudur. Optimal kernel choice for large-scale two-sample tests. In Advances in neural information processing systems, pages 1205-1213. Citesecr, 2012.
[20] Yu-Chao Gu, Li-Juan Wang. Yun Liu, Yi Yang, Yu-Huan Wu, Shao-Ping Lu, and Ming-Ming Cheng. Dots: Decoupling operation and topology in differentiable architecture search. In Proceedings of the IEEE/CVF
Conference on Computer Vision and Pattern Recognition, pages 12311-12320, 2021.
[21] Tin Kam Ho. Random decision forests. In Proceedings of 3rd international conference document analysis and recognition, volume pages 278-282. IEFF, 1995.
[22] Weijun Hong. Guilin Li. Weinan Zhang, Ruiming Tang, Yunhe Wang. Zhenguo Li, and Yong Yu. Dropnas: Grouped operation dropout for differentiable architecture search. In I/CAI. pages 2326-2332, 2020. Mark Sandler, Grace Chu, Liang-Chicl Chen, Bo Chen, Mingxing Tan, Weijun Wang. Yukun Zhu, Ruoming Pang. Vijay Vasudevan, et al. Searching for mobilenetv3. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1314-1324, 2019.
[23] Andrew
[24] Jie Hu, Li Shen, and Gang Sun. Squeeze-and and-excitation networks. In Proceedings of the IEEE conference
on computer vision and pattern recognition, pages 7132- 7132-7141, 2018.
[25] Huawei. Mindspore. https: //www//wwww..o mi minnopore... ctt.eeerrreeeeeeeeeeeeee..e. 2020.
[26] Leslic Pack Kaelbling. Michael L Littman, and Andrew Moore. Reinforcement learning: A survey.
Journal of artificial intelligence research, 4:237-285, 1996.
[27] Daniel Kifer, Shai Ben-David, and Johannes Gehrke. Detecting change in data streams. In VLDB, volume 4,
pages 180-191. 180ono,,, Canada, 2004.
[28] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks.
arXiv preprint ar 160::/6009029000 2016.
[29] Nikita Klyuchnikov. Ilya Trofimov. Ekaterina Artemova, Mikhail Salnikov, Maxim Fedorov, and Evgeny
Burnaev. Nas-bench-nlp: neural architecture search benchmark for natural language processing. arXiv
preprint arXiv:2006.07 07) 16, 2020.
[30] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[31] Yanxi Li, Minjing Dong. Yunhe Wang, and Chang Xu. Neural architecture search in proxy validation
loss landscape. International Conference on Machine Learning, pages 5853-5862. PMLR, 2020.
[32] Yanxi Li, Zhaohui Yang, Yunhe Wang, and Chang Xu, Adapting neural architectures between domains.
Advances in Neural Information Processing Systems, 33, 2020.
[33] Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: differentiable architecture search. 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019.
OpenReview.net 2019.
[34] Yuqiao Liu, Yanan Sun, Bing Xue, Mengjie Zhang, Gary G Yen, and Kay Chen Tan. A survey on evolutionary neural architecture search. IELE Transactions on Neural Networks and Learning Systems, 2021.
[35] Mingsheng Long, Yue Cao. Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In International conference on machine learning, pages 97-105, PMLR, 2015.
[36] Rengian Luo, Xu Tan, Rui Wang, Tao Qin, Enhong Chen, and Tic- Yan Liu. Semi-supervised neural architecture search. Advances Neural Information Processing Systems, 33, 2020.
[37] Renqian Luo, Fei Tian, Tao Qin, Enhong Chen, and Tic- Yan Liu. Neural architecture optimization. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 0827-7838,22 2018
[38] Abhinay Mehrotra, Alberto Gil C. P. Ramos, Sourav Bhattacharya, Lukasz Dudziak, Ravichander Vipperla, Thomas C. P. Chau. Mohamed S. Abdelfattah, Samin Ishtiaq. and Nicholas Donald Lane. Nas-bench asr: Reproducible neural architecture search for speech recognition. In 9th International Conference on Learning Representations, ICIR 2021, Virtual Event, Austria, May 3-7, 2021. Open 2021.
[39] Maxime Oquab, Leon Bottou, Ivan Laptev, and Josef Sivic. Leaming and transferring mid-level image representations using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and puttern recognition, pages 1717-1724, 2014.
[40] Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on knowledge and data engineering. 22 12400:1345-1339 2009.
[41] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary De Vito, Zeming Lin, Alban Desmaison, Luca Autiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
[42] Hicu Pham, Melody Guan, Barret Zoph. Quoc Lc, and Jeff Dean. Efficient neural architecture search via parameters sharing. In International Conference on Machine Leurning, pages 4005-4104. PMLR 2018.
[43] Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Le. Regularized evolution for image classifier architecture search. In Proceedings of the aaai conference on artificial intelligence, volume 33, pages 4780-4789, 2019.
[44] Shreyas Saxena and Jakob Verbeek. Convolutional neural fabrics. Advances neural information processing systems, 19:4053-4061, 2016
11

---

[45]
Pranab Kumar Sen. Estimates of the regression coefficient based on kendall's tau. Journal of the American statistical association. 63(324):1379-11
1968.
[46]
David So, Quoe Lc, and Chen Liang. The evolved transformer. In International Conference on Machine Learning, pages 5877-5886, PMLR, 2019.
[47] Bharath K Sriperumbudur. Kenji Fukumizu, Arthur Gretton, Gert RG Lanckriet. and Bernhard Schölkopf. Kernel choice and classifiability for rkhs embeddings of probability distributions. In NIPS, volume 22, pages 1750-1758, 2009.
[48] Baochen Sun, Jiashi Feng, and Kate Saenko. Return of frustratingly casy domain adaptation. In Proceedings of the AAA/ Conference on Artificial Intelligence, volume 30, 2016.
[49] Yanan Sun, Handing Wang. Bing Xue, Yaochu Jin, Gary G Yen, and Mengjie Zhang. Surrogate-asso assistte-ssssedd evo- lutionary deep learning using an end to see--t--nd random forest-based performance predictor. IEEE Transactions Evolutionary Computation, 24(2):350- 364, 2020.
[50]
Yanan Sun. Bing Xue, Mengjie Zhang, and Gary G Yen. Evolving deep convolutional neural networks for image classification. IEEE Transactions on Evolutionary Compulation. 2019.
[51] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architccture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818-2826, 2016.
[52] Mingxing Tan, Bo Chen, Ruoming Pang. Vijay Vasudevan. Mark Sandler, Andrew Howard, and Quoe
Le. Mnasnet: Platform-ave aware neural architecture search for mobile. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2820-2828. 2019.
[53] Mingxing Tan and Quoe V. Lc. Efficientnet: Rethinking model scaling for convolutional neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors. Proceedings of the 36th International Con- ference Machine Learning. ICML 2019, 9-15 June 2019, Long Beach, California. USA, volume 97 of Proceedings of Machine Learning Research, pages 6105-6114, PMLR, 2019.
[54] Yehui Tang. Yunhe Wang. Yixing Xu, Hanting Chen, Boxin Shi, Chao Xu, Chunjing Xu, Qi Tian, and Chang Xu, A semi-supervised assessor of neural architectures. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18 1810-1819, 2020.
[55] Wei Wen, Hanxiao Liu, Yiran Chen, Hai Li, Gabriel Bender, and Pieter-Jan Kindermans. Neural predictor for ncural architecture scarch. In European Conference on Computer Vision, pages 660-676. Springer, 2020.
[56] Junru Wu, Xiyang Dai, Dongdong Chen, Yinpeng Chen, Mengehen Liu, Ye Yu, Zhangyang Wang, Zicheng Liu, Mei Chen, and Lu Yuan. Stronger nas with weaker predictors. Advances in Neural Information Processing Systems. 34:28904 28918, 2021.
[57] Vixing Xu, Yunhe Wang, Kai Han, Ychui Tang, Shangling Jui, Chunjing Xu, and Chang Xu. Renas: Relativistic evaluation neural architecture search. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 4411-4420, 2021.
[58] Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Guo-Jun Qi, Qi Tian, and Hongkai Xiong. PC-DARTS:
partial channel connections for memory-efficient architecture search. In 8th International Conference on Learning Representations. ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. [59] Hongliang Yan, Yukang Ding, Peihua Li, Qilong Wang. Yong Xu, and Wangmeng Zuo. Mind the class weight bias: Weighted maximum mean discrepancy for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 2272-2281. 2017.
[60] Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter. Nas-bench-
101: Towards reproducible neural architecture search. In International Conference on Machine Learning, pages 7105-71 14, 2019.
[61] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? Advances in Newral Information Processing Systems, 27:3320-3328. 2014.
[62] Wei Zhang, Lin Zhao, Qing Li, Shijie Zhao, Qinglin Dong, Xi Jiang, Tuo Zhang, and Tianming Liu. Identify hierarchical structures from task-based fimri data via hybrid spatiotemporal neural architecture search net. In International Conference on Medical Image Computing and Computer- Assistte Intervention.
pages 745-753. Springer, 2019.
[63] Yongchun Zhu, Fuzhen Zhuang, Jindong Wang, Guolin Ke, Jingwu Chen, Jiang Bian, Hui Xiong, and Qing He. Deep subdomain adaptation network for image classification. IEEE transactions on neural networks and learning systems, 32(4):1713-177)
2020.
[64] Fuzhen Zhuang. Zhiyuan Qi. Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing
He. A comprehensive survey on transfer learning. Proceedings of the IEEE, 109(1):43-78 76. 2020.
[65] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Lc. Learning transfcrable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8697-8710, 10, 2018.
12


Reviews:
[{"review_id": "yVXPlAt7x-z", "reviewer": "Reviewer_jvHq", "summary": "Recent advances in Neural Architecture Search (NAS) investigate the neural predictor, which directly estimates the performance of neural architectures, to alleviate the costs of performance evaluation. However, the neural predictor still requires a set of annotated architectures for training. To alleviate this issue, this paper proposes a cross-domain adaptation strategy (CDP; Cross-Domain Predictor), which mitigates such training costs for annotation by utilizing existing labeled datasets.", "questions": "- How does this paper measure GPU hours? While the seminal work of DARTS [Liu et al., 2019] stated that “All of our experiments were performed using NVIDIA GTX 1080Ti GPUs.”, I cannot find such statements in this paper.\n- Using Kendall’s Tau (KTau; Sen, 1968), in Section 4.3, the paper demonstrates the improvements from the proposed components for CDP (i.e., cosine scheduling, medium-sized assistant space, and LMMD) in terms of the quality of neural predictors. However, there are no comparisons with existing methods for training neural predictors, which utilize annotated datasets for the target space (e.g., Wen et al., 2020). It will be nice to check whether the proposed method produces high-quality neural predictors comparable to the previous approaches without such annotations.\n---\n- [Liu et al., 2019] DARTS: Differentiable Architecture Search. 2019.\n- [Sen, 1968] Estimates of the Regression Coefficient Based on Kendall's Tau. 1968.\n- [Wen et al., 2020] Neural Predictor for Neural Architecture Search. 2020.", "limitations": "This paper did not include both limitations and societal impacts of the work.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Although this paper borrows existing concepts from domain adaptation, it does not hurt the significance of the work; I believe that converting key ideas across sub-fields (in this case, from domain adaptation into neural architecture search) is still valuable for academia.\n\nOverall, the paper is well-written; it clearly presents the existing problems for training neural predictors and how the proposed method alleviates that. Applying domain adaptation methodologies to utilize existing labeled architecture space for another space seems reasonable. Moreover, NAS-specific improvements make the paper have its own contribution. I especially like the ablation study in Section 4.3, which measures the quality of the neural predictor for various settings of the proposed cross-domain adaptation strategies.\n\n*Caveat: For the other experimental results in the paper (including Sections 4.1 and 4.2), I would defer to the other reviewers; since I am not familiar with NAS experiments.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "m2Dldj9zDU", "reviewer": "Reviewer_tD8p", "summary": "This work targets on leveraging leverage existing NAS Bench to predict out-of-domain architecture’s performance. The author propose to learn a predictor on network's performance, via closing the domain feature gap between source and target architecture space.", "questions": "1. Eq. 9, the scheduler for the Subspace Partition, sounds empirical. The authors may consider down-play its contribution, or compare Eq. 9 with other possible schedulers to show its importance.\n\n2. I can understand the importance of the Assistant Architecture Space. However, is there any possible principles in designing it? Otherwise it will hurt the generality of the proposed method.\n\n3. d_H in Eq. 10 is not defined or discussed, or did I miss anything? I suppose it is a kind of distance measure between two domains?\n\n4. I think Thm 2 is too general to be helpful to the main purpose of this (NAS) work. As far as I understand, the most insight from Thm. 2 is that larger m and m’ would give us a lower error bound. But this conclusion is very general and is not leveraged in this work (thus sounds less important).", "limitations": "I think in general the target problem and the method of this work is novel. However, some designs in the method (Eq. 9, and the assistant space) may be empirical and may hurt its practical use.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. The targeted question quite makes sense: to use prepared NAS benchmarks to learn the performance predictor.\n\n2. The proposed method also sounds novel: to train the predictor by minimizing the distribution distance between the network's feature space and the label distribution.\n\n3. The authors provide good ablation studies.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "H2ZEHAq3cG", "reviewer": "Reviewer_KWRx", "summary": "The paper set out to solve the domain discrepancy issue between source architecture space and the target space in Neural Architecture Predictors. The paper proposed a progressive subspace adaptation strategy to train a Cross-Domain Predictor.", "questions": "(1) I'm curious how would the performance be, if we only include NAS-Bench-101 or NAS-Bench-201 as source domain?\n(2) In Table 1/2, would that be unfair that other methods do not have any transferred knowledge, while CDP has prior knowledge from NAS-Bench with the training time of NAS-Bench not included.", "limitations": "(a) Some of the up-to-date methods are not compared as stated in the Weakness part.\n(b) Some additional ablation is needed as stated in Questions.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n(1) The proposed CDP method is simple and intuitive, this idea of transfering knowledge from NASBenches and leveraging LMMD loss from transfer learning/domain adaption sounds novel to me.\n(2) The performance of transfer learning in NAS is impressive to me, though the search cost comparison is a bit misleading as stated in the questions. \n(3) The paper is overall well-written and well-presented.\n\nWeakness: \n(1) The 0.5306 KTau in Table 5 is not impressive, the author should consider reporting the TopK% (or Top-50/100 depending on search space size) Kendall-Tau correlation of predictor, since NAS only output the BEST predicted architecture, that's where different predictor-based NAS methods would make a real difference.\n(2) The comparison methods (NAO, Neural Predictor, SemiNAS) in Table 1 seem to be outdated, a comparison with more recent Predictor-based NAS methods [1][2] should also be included: \n[1] BRP-NAS: Prediction-based NAS using GCNs, NeurIPS 2020\n[2] Stronger NAS with Weaker Predictors. NeurIPS 2021\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "AcuV3g4OE_J", "reviewer": "Reviewer_Ywk3", "summary": "This paper develops a cross-domain predictor, called CDP, which does not require training neural architectures in the particular search space. It leverages the existing NAS benchmark datasets and adopts domain adaptation to adapt from the source space of NAS benchmark to the target space. Therefore, it does not require to train architectures in the target space and largely reduces the cost.", "questions": "1. How does CDP perform if the two domains are for different tasks? For example, from image classification to NLP task?\n2. How is the kernel function and mapping function designed?\n\n", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The motivation is clear to leverage the existing NAS benchmarks for training-free NAS.\n2. The proposed method is sound with proof.\n3. Improved experimental performances against other works on ImageNet and CIFAR-10.\n\nWeaknesses:\n1. The method relies on existing NAS benchmarks. Therefore, it is hard to apply to some domain/tasks that do not have collection of architecture-performance pairs.\n2. The design of assistant space could be impractical for other domains/complex spaces. The NAS-Bench space and DARTS space are still simple.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept