Network Architecture Search for Domain Adaptation
Authors: Yichen Li
Xingchao Peng
Venue:  (2021)

Abstract: Deep networks have been used to learn transferable representations for domain adaptation. Existing deep domain adaptation methods systematically employ popular hand-crafted networks designed specifically for image-classification tasks, leading to sub-optimal domain adaptation performance. In this paper, we present Neural Architecture Search for Domain Adaptation (NASDA), a principle framework that leverages differentiable neural architecture search to derive the optimal network architecture for domain adaptation task. NASDA is designed with two novel training strategies: neural architecture search with multi-kernel Maximum Mean Discrepancy to derive the optimal architecture, and adversarial training between a feature generator and a batch of classifiers to consolidate the feature generator. We demonstrate experimentally that NASDA leads to state-of-the-art performance on several domain adaptation benchmarks.

Under review as a conference paper at ICLR 2021
NETWORK ARCHITECTURE SEARCH FOR DOMAIN ADAPTATION
Anonymous authors Paper under double-bling review
ABSTRACT
Deep networks have been used to learn transferable representations for domain adaptation. Existing deep domain adaptation methods systematically employ pop- ular hand-crafted networks designed specifically for image-classificttio tasks, leading to sub-optimal domain adaptation performance. this paper, we present Neural Architecture Search for Domain Adaptation (NASDA), principle frame- work that leverages differentiable neural architecture search to derive the optimal network architecture for domain adaptation task. NASDA is designed with two novel training strategies: neural architecture search with multi-kernel Maximum Mean Discrepancy derive the optimal architecture, and adversarial training between a feature generator and a batch of classifiers to consolidate the feature generator. We demonstrate experimentally that NASDA leads to state-of-the-art performance on several domain adaptation benchmarks.
INTRODUCTION
Supervised machine learning models (+) aim to minimize the empirical test error (x() y)) by optimizing & on training data (x) and ground truth labels (y), assuming that the training and testing data are sampled i.i.d from the same distribution. While in practical, the training and testing data typically collected from related domains under different distributions, a phenomenon known as Lutionero-Candele et al., 2009). To avoid the cost of annotating
domain shift (or domain discrepancy) each new test data, Unsupervised Domain Adaptation (UDA) tackles domain shift by transferring the knowledge learned from a rich-labeled source domain ( (P(x", (P""".")) to the unlabeled target domain (Q(x')). Recently unsupervised domain adaptation research has achieved significant progress with techniques like discrepancy alignment (Long et al., 2017: Tzeng et al., 2014; Ghifary et al., 2014; Peng & Saenko, 2018: Long et al., 2015; Sun & Saenko, 2016), adversarial alignment (Xu et al., 2019a; Liu & Tuzel, 2016; Tzeng et al., 2017: Liu et al., 2018a; Ganin & Lempitsky, 2015; Saito et al., 2018; Long et al., 2018), and reconstruction-but
alignment (Yi et al., 2017; Zhu et al., 2017; Hoffman et al., 2018; Kim et al., 2017). While such models typically learn feature mapping from one domain (+(x*)) ) to another (T(x') )) or derive a joint representation across domains (0(x*) @ &(x*)) )), the developed models have limited capacities in deriving an optimal neural architecture specific for domain transfer.
To advance network designs, neural architecture search (NAS) automates the net architecture engi- neering process by reinforcement supervision (Zoph & Le, 2017) or through neuro-evlolution (Real al., 2019a). Conventional NAS models aim to derive neural architecture a along with the net- work parameters w. by solving a bilevel optimization problem (Anandalingam & Friesz, 1992): Pa.w arg min, Lval (w" (a), a)) s.t. w²(a)
Segmin,,Crrattt (w, at). where Ltrain and L in- dicate the training and validation loss, respectively. While recent works demonstrate competitive performance on tasks such as image classification (Zoph et al., 2018: 8; Liu et al., 201 0018;;;; Real et al., 2019b) and object detection (Zoph & Le, 2017), designs of existing NAS algorithms typically assume that the training and testing domain are sampled from the same distribution, neglecting the scenario where two data domains multiple feature distributions are of interest.
To efficiently devise a neural architecture across different data domains, we propose a novel learning task called Neural Architecture Search for Domain Adaptation (NASDA). The ultimate goal of NASDA is to minimize the validation loss of the target domain (C We postulate that a solution to NASDA should not only minimize validation loss of the source domain (Lrm)). but should also

---

Under review as a conference paper at ICLR 2021
search space
peedictions
predictions
masiinize disoeeency
mininiee discrrpncc
fix
update C
update
fix
(a)
(b)
(d)
Figure 1: An overview of NASDA: (a) Continuous relaxation of the research space by placing a mixture of the candidate operations each edge. (b) Inducing the final architecture by joint opti- mization of the neural architecture parameters a and network weights w, supervised by minimizing the validation loss the source domain and reducing the domain discrepancy. (c)(d) Adversarial training of the derive feature generator G and classifiers C.
reduce the domain gap between the source and target. To this end, propose new NAS learning schema:
D argming,Cuateee (a), a) + disc(# (x"0""""xxx"))""" (x"))
(I)
s.t. w (a)) = argmin Cirain( (w,,
where d- 4 (ar,o"()) (o)) and disc(""(x") 4 (x*)) denotes the domain discrepancy between the source and target. Note that in unsupervised domain adaptation, Crrain and C cannot be computed directly due to the lack of label in the target domain.
Inspired by the past works in NAS and unsupervised domain adaptation, we propose in this paper an instantiated NASDA model, which comprises of two training phases, as shown in Figure 1. The first is the neural architecture searching phase, aiming to derive an optimal neural architecture (a**), following Tecture Search (DARTS) (Liu et al., 2019a), we relax the search space to be continuous so that a can be optimized with respect to C%a and disc (x((")), ((x')) by embedding the hidden representations of the task-specific layers to a reproducing kernel Hilbert space where the mean embeddings can be explicitly matched by minimizing disc ① (x*), ((x))). We use nernell Maximum Mean Discrepancy (MK-MMD) (Gretton et al., 2007) to evaluate the domain discrepancy.
the learning schema of Equation 1,2. Inspired by Differentiable
)) by gradient descent. Specifically, we enhance the feature transferability
The second training phase aims to learn a good feature generator with task-specific loss, based on the derived deep neural network
the derived a" from the first phase. To establish this goal, (fa-) ) as the feature generator (G) and devise an adversarial training process between G and a batch of classifiers C. The high-level intuition is to first diversify C' in the training process, and train G to generate features such that the diversified C' can have similar outputs. The training process is similar Maximum Classifier Discrepancy framework (MCD) (Saito al., 2018) except that we extend the dual-classifier in MCD to ensembling of multiple classifiers. Experiments standard UDA benchmarks demonstrate the effectiveness of our derived NASDA model in achieving significant improvements over state-of-the-an methods.
Our contributions of this paper are highlighted as follows:
We formulate a novel dual-objective task of Neural Architecture Search for Domain Adap- tation (NASDA), which optimize neural architecture for unsupervised domain adaptation, concerning both source performance objective and transfer learning objective.
We propose an instantiated NASDA model that comprises two training stages, aiming to derive optimal architecture parameters a" and feature extractor G. respectively. We are the first to show the effectiveness of MK-MMD in NAS process specified for domain adaptation.
Extensive experiments multiple doos--domain recognition tasks demonstrate that NASDA achieves significant improvements over traditional unsupervised domain adaptation models well as state-of-the-art NAS-based methods.
2

---

Under review as a conference paper at ICLR 2021
2 RELATED WORK
Deep convolutional neural network has been dominating image recognition task. In recent years, many handcrafted architectures have been proposed, including VGG (Simonyan & Zisserman, 2014), ResNet (He et al., 2016), Inception (Szegedy et al., 2015), etc., all of which verifies the importance of human expertise in network design. Our work bridges domain adaptation and the emerging field of neural architecture search (NAS), a process of automating architecture engineering technique.
Neural Architecture Search Neural Architecture Search has become the mainstream approach to discover efficient and powerful network structures (Zoph & Lc, 2017; Zoph et al., 2018). The automatically searched architectures have achieved highly competitive performance in tasks such as image classification (Liu et al., 201 (0;)), object detection (Zoph et al., 2018), and semantic segmentation (Chen et al., 2018). Reinforce learning based NAS methods (Zoph & Le, 2017; Tan et al., 2019; Tan & Le, 2019) are usually computational intensive, thus hampering its usage with limited computational budget. To accelerate the search procedure, many techniques has been proposed and they mainly follow four directions: (1) estimating the actual performance with lower fidelities. Such lower fidelities include shorter training times (Zoph et al., 2018; Zela et al., 2018), training on a subset of the data (Klein al., 2017), on lower-resolution images. (2) estimating the performance based on the learning curve extrapolation. Domhan et al. (2015) propose to extrapolate initial learning curves and terminate those predicted to perform poorly. (3) initializing the novel architectures based on other well-trained architectures. Wei et al. (2016) introduce network morphisms to modify an architecture without changing the network objects, resulting in methods that only require a few GPU days (Elsken et al., 2017; Cai et al., 201 8a; Jin et al., 2019; Cai et al., 2018b). (4) one-shot architecture search. One-shot NAS treats all architectures as different subgraphs of a supergraph and shares weights between architectures that have edges of this supergraph in common (Saxena & Verbeek, 2016; Liu al., 20;9b; Bender, 2018). DARTS (Liu et al., 2019a) places a mixture of candidate operations on each edge of the one-shot model and optimizes the weights of the candidate operations with a continuous relaxation of the search space. Inspired by DARTS (Liu et al., 2019a), our model employs differentiable architecture search derive the optimal feature extractor for unsupervised domain adaptation.
Domain Adaptation Unsupervised domain adaptation (UDA) aims to transfer the knowledge learned from one or more labeled source domains to an unlabeled target domain. Various methods have been proposed, including discrepancy-bon
UDA approaches (Long et al., 2017: Tzeng et al., 2014; Ghifary et al., 2014; Peng & Saenko, 2018), adversary-based approaches (Liu & Tuzel, 2016; Tzeng et al., 2017; Liu al., 2018a), and reconstruction-base approaches (Yi et al., 2017: Zhu al., 2017: Hoffman et al., 2018; Kim et al., 2017). These models are typically designed to tackle single source to single target adaptation. Compared with single source adaptation, multi-source domain adaptation (MSDA) assumes that training data are collected from multiple sources. Originating from the theoretical analysis in (Ben- et al., 2010; Mansour et al., 2009; Crammer et al., 2008), MSDA has been applied to many practical applications (Xu et al., 2018: Duan et al., 2012; Peng et al., 2019). Specifically, Ben-David et al. (2010) introduce an H.H --divergeneee between the weighted combination of source domains and target domain. These models are developed using the existing hand-crafted network architecture. This property limits the capacity and versatility of domain adaptation as the backbones extract the features are fixed. In contrast, tackle the UDA from different perspective, not yet considered in the UDA literature. We propose a novel dual-objective model of NASDA, which optimize neural architecture for unsupervised domain adaptation. We are the first to show the effectiveness of MK-MMD in NAS process which is designed specifically for domain adaptation.
NEURAL ARCHITECTURE SEARCH FOR DOMAIN ADAPTATION
In unsupervised domain adaptation, we are given source domain D, { (xf, yf) of n. labeled examples and a target domain D,- = {x} } of nu unlabeled examples. The source domain and target domain are sampled from joint distributions P(x*, y²) and Q(x', y²), respectively. The goal of this paper is to leverage NAS to derive a deep network G x + y. which is optimal for reducing the shifts in data distributions across domains, such that the target risk (G) Exx, yn)a [G (x') ) * y"] is minimized. We will start by introducing some preliminary background in Section 3.1. We then describe how to incorporate the MK-MMD into the neural architecture searching framework in
3

---

Under review as a conference paper at ICLR 2021
Section 3.2. Finally, we introduce the adversarial training between our derived deep network and batch of classifiers in Section 3.3. An overview of our model can be seen in Algorithm 1.
3.1 PRELIMINARY: DARTS
In this work, we leverage DARTS (Liu et al., 2019a) our baseline framework. Our goal is search for a robust cell and apply it to a network that is optimal to achieve domain alignment between D, and D,.. Following Zoph et al. (2018), we search for a computation cell as the building block of the final architecture. The final convolutional network for domain adaptation can be stacked from the learned cell. A cell defined as a directed acyclic graph (DAG) of L nodes, j=1 where each node (6) a latent representation and each directed edge c(f-)) is associated with some operation olh-5) that transforms . DARTS (Liu et al., 2019a) assumes that cells contain two input nodes and a single output node. To make the search space continuous, DARTS relaxes the categorical choice of a particular operation to a softmax over all possible operations and is thus formulated as:
exp(a (1.))
Σ
OEO Eo'eo expla (1J)) o(z)
(3)
where O denotes the set of candidate operations and i j so that skip-connect can be applied. An intermediate node can be represented as æj = Σies (xh)) The task of architecture search then reduces to learning a set of continuous variables a = (a(i))) At the end of search, discrete architecture can be obtained by replacing each mixed operation g(i.)) with the most likely operation,
i.e. (1.a) argmax,co Og (t.f) and a² (..
3.2 SEARCHING NEURAL ARCHITECTURE
Denote by Ferein and Cval the training loss and validation loss, respectively. Conventional neural architecture search models aim to derive L by solving a bilevel optimization problem (Anan- dalingam & Friesz, 1992): &a, While recent work (Zoph et al., 2018; Liu et al., 2018c) have show promising performance on tasks such as image classification and object detection, the existing models assume that the training data and testing data are sampled from the same distributions. Our goal is to jointly learn the architecture and the weights within all the mixed operations (e.g. weights of the convolution filters) so that the derived model "w=".a can transfer knowledge from De to D, with some simple domain adapation guidence. Initialized by Equation 1, we leverage multi-kernel Maximum Mean Discrepancy (Gretton et al., 2007) to evaluate disc (x*))<"<x"" (x")..
arg min, Lea (w" (a),, a) s.t. (a) = wrrmin (w., a).
MK-MMD Denote by H4 be the Reproducing Kernel Hilbert Space (RKHS) endowed with characteristic kernel k. The mean embedding of distribution in Hy is a unique element (P) such that Ex-Pf (x) (f (x), M4g (P)) for all E H4. The MK-MMD d, (P, between probability distributions P and Q is defined as the RKHS distance between the mean embeddings of P and Q. The squared formulation of MK-MMD is defined
(P.QQ)
A Ep Pa (x*) Eq D (x)]]|
(4)
In this paper, we consider the case of combining Gaussian kernels with injective functions fa- where k(ε, exp(-Il.aaz)) fo (2)) Inspired by Long et al. (2015), the characteristic kernel associated with the feature map 4, k (x", x²) ( (x*), Φ (xt)), is defined as the convex combination of n positive semidefinite kernels {kiu),
n
K a
l
Baku
Σ 1 > Vu
(5)
where the constraints on {Ba} are imposed to guarantee that the k is characteristic. In practice we use finite samples from distributions to estimate MMD distance. Given X = {xi, .., x
P
and X = [x] x ~ Q, one estimator of dP(PP. P.P.G Q) is
(P.PQ)
x
k
k(x x(x))
(6)
2
2 343
4

---

Under review as a conference paper at ICLR 2021
Algorithm 1 Neural Architecture Search for Domain Adaptation
Phase I: Searching Neural Architecture
1: Create a mixed operation obاd)) parametrized by a!i.)) for each edge (i,
2: while not converged do
3: Update architecture by da fa (w 6 S firain (w, + Tee ( (0(x),,
a 4: Update weights wo by descending Tw firein (w, a)
5: end while
6: Derive the final architecture based the learned a".
Phase I:: Adversarial Training for Domain Adaptation
Stack feature generator G based on a", initialize classifiers C
2: while not converged do
3: Step one: Train G and C with "* --(x)= )x Σ yy log
4: Step two: Fix G, train C with loss: (x y (xav((x)((e 13)
5: Step three: Fix C, train G with loss: Lanv (x)v(x'))
6: end while
The merit of multi-kernel MMD lies in its differentiability such that it can be easily incorporated into the deep network. However, the computation of the de Q) incurs complexity of (m²), which is undesirable in the differentiable architecture search framework. In this paper, we use the unbiased estimation of MK-MMD (Gretton et al., 2012) which can be computed with linear complexity.
NAS for Domain Adaptation Denote by Lirain and a the training loss and validation loss on the source domain, respectively. Both losses are affected by the architecture in the network. The goal for NASDA is to find a that minimizes the validation loss aa (w" a" ) on the target domain, where the weights w² associated with the architecture are obtained by minimizing the training loss w² = argmin,, to Livain (w. a²). Due to the lack of labels in the target domain, it is prohibitive to compute L directly, hampering the assumption of previous gradient-ban based NAS algorithms (Liu et al., 2019a; Chen et al., 2019). Instead, we derive a" by minimizing the validation loss L (w", a ) on the source domain plus the domain discrepancy, disc($ (x(x") & (xx)). as shown in Equation!
well as by the weights w
Inspired by the gradient- hyperparameter optimization (Franceschi et al., 2018; Pedregosa, 2016; Maclaurin et al., 2015), set the architecture parameters
special type of hyperparameter. This implies bilevel optimization problem (Anandalingam & Friesz, 1992) with a as the upper-level variable and the lower-level variable. In practice, utilize the MK-MMD evaluate the domain discrepancy. The optimization can be summarized as follows:
a, argmin, (Ciac(w" (a), + Ad (f(x"), ((x)))
(7)
s.t. " (a) argmin, Cirain (w, a)
(8)
where À is the trade-off hyperparameter between the source validation loss and the MK-MMD loss.
Approximate Architecture Search Equation 7,8 imply that directly optimizing the architecture gradient is prohibitive due to the expensive inner optimization. Inspired by DARTS (Liu et al., 2019a), we approximate w" wa(a) (a:) by adapting w using only a single training step, without solving the optimization in Equation 8 by training until convergence. This idea has been adopted and proven to be effective meta-learning for model transfer (Finn et al., 2017), gradient-based hyperparameter tuning (I (Luketina et al., 2016) and unrolled generative adversarial networks. We therefore propose simple approximation scheme as follows:
da (hiu(w"(a)),) + dd (x(x') ((x'))
2 23 Lual w Ci (w, a), a) + 2
8a ( (x(x") (xx'))
Da
Dw
where E Sw Lirain (w, a) denotes weight for one-step forward model and is the learning rate for a step of inner optimization. Note Equation 9 reduces to V V. Cwat w, at) if w is already local optimum for the inner optimization and thus V Ltraaii (w, a) 0.
0
5

---

Under review as a conference paper at ICLR 2021
dil Lonv__oovt
skip_ connect
10-71
sep cony 3x3
skip connect
dil comre 3x3
[k:]
avg_pool _x3
max pool 3x3
CONY
sep_ rep__con__3xx 3x3
dlip_comm comnect
max poolpooooopooo 3x33
sep_comt 3x.
dil cil_coov__3 3x3
max pooo_p0333
[0k-4]
sep_ rep_ronv__3x _xx3
avg. avg_pool 3x3
(a) Normal cells (left) and Reduce cells (right) for CIF-II
tuny
dil canv
Skip DOUIIC
Method
Params
Search Cost
coovo 3x3
skip.
(M)
(GPU days)
dill Loiv 303
H--1
3x3
NASNet
1,800
AmoehaNet
3,2
3,150
PNAS
3.2
225
sep_conv 5
DARTS
3.3
1.5
avg avg__poo pool 3x3
SNAS
2.8
1.5
[ may pool 3x3
pool 3x3
[k
PDARTS
0.3
pool skip connert Skip_ comnect
NASDA
2.7
0.3
[1
(c) Network architecture statis- ties comparison
(b) Normal cells (upper) and Reduce cells (lower) for MNIST-USPPS
Figure 2: (a) Neural architecture for STL-CIIARIOO task. (b) Neural architecture for MNIST-UUSPS
results. (c) Comparison between our NASDA model and state-of-the-ar -frtt--rttttttttt NAS models.
The second term of Equation 9 can be computed directly with some forward and backward passes. For the first term, applying chain rule to the approximate architecture gradient yields
θ (w' a))
82
E( Irdw Lirain (w. a) Ow Lival (wo ,a))
(10)
dor
0 Lerain (w, (w. The expression above contains an expensive matrix-vector product where w' = w-C Sw in its second term. We leverage the central difference approximation to reduce the computation complexity, Specifically, let 17 be small scalar and w = w + aw C (w', , a). Then:
á
8² Fxd& Cirain (w, a) Ow Liall (wd, ,a)
2
8 Do Cirain (w+ a)
a Lirain (w .a) da
(II)
21
Evaluating the central difference only requires two forward passes for the weights and two backward passes for a, reducing the complexity from quadratic to linear.
3.3 ADVERSARIAL TRAINING FOR DOMAIN ADAPTATION
By neural architecture searching from Section 3.2, we have derived the optimal cell structure (a") for domain adaptation. We then stack the cells to derive our feature generator G. In this section, we describe how do we consolidate G by an adversarial training of G and the classifiers C. Assume C includes N independent classifiers (C(A) = J. and denote pa px(y[x) (x) the K-way propabilistic outputs of COO, where K is the category number.
The high-level intuition is to consolidate the feature generator G such that it can make the diversified C generate similar outputs. To this end, our training process include three steps: (1) train G and C on D. to obtain task- features, (2) fix G and train C' to make J Ji=] have diversified output,
(3) fix C' and train G to minimize the output discrepancy between C. Related techniques have been used in Saito et al. (2018); Kumar et al. (2018).
First, we train both G and C' to classify the source samples correctly with cross-entropy loss. This step is crucial as it enables G and C to extract the task-specific features. The training objective is min (x*,, y* and the loss function is defined as follows: G,C
K 1 mk log log((""(x')
L'(x', = -E -E(x'''>>> y"
(12)

---

Under review as a conference paper at ICLR 2021
To establish this goal, we fix G and train C to increase the discrepancy of C"s output. To avoid mode collapse (e.g. CC) outputs all zeros and C(2) output all ones), we add L fx"x, y²) ) as regularizer in the training process. The high-level intuition is that we do not expect C' to forget the information learned in the first step in the training process. The training objective min L'(x' y*) Laav (xa). where the adversarial loss is defined as:
In the second step, are aiming diversify C.
N-1
N
Ladv (x') = DD
Σ
((x:(ylx))Psdd) - J-
Σ
(13)
In the last step, we trying consolidate the feature generator G by training G to extract generalizable representations such that the discrepancy of C's output is minimized. To achieve this goal, we fix the diversified classifiers C? and train G with the adversarial loss (defined in Equation 13). The training objective is min Ladv(x;).
G
4 EXPERIMENTS
We compare the proposed NASDA model with many state- of-the-art UDA baselines on multiple benchmarks. In the main paper, only report major results; more details are provided in the supplementary material. All of our experiments are implemented in the PyTorch platform.
Method
STL CIFARIO
DANN
56.9
MCD
69.2
DWT
71.2
SE
74.2
In the architecture search phase, we use A=1 for all the searching experiments. We leverage the Rel.U-Conv-BI order for convolutional operations, and each separable convolution is always applied twice. Our search space o includes the following operations: 3 x 3 and 5 x 5 separable convolutions,
G2A
72.8
VADA
73.5
DIRTT
75.3
NASNet-Phasee
67.3
AmoebaNets PeeebaNettPaasee
67.0
DARTS+Phas II
68.8
3 and ble convolutions, 3 3 max pooling, identity, and zero. Our convolutional cell consists of N = nodes. Cells located at the and _ of the total depth of the network are reduction cells. The architecture encoding therefore is
dilated separa-
PDARTS+Phase
66.0
NASDA
76.8
Table 1: Accuracy (%) STL.
(Anormall Otreduce). ), where anormal is shared by all the normal cells and areduce is shared by all the reduction cells.
4.1 SETUP
Digits We investigate three digits datasets: MNIST, USPS, and Street View House Numbers (SVHN). We adopt the evaluation protocol of CyCADA (Hoffman et al., 2018) with three transfer tasks: USPS to MNIST (U + M), MNIST to USPS (M - U). and SVHN to MNIST (S + M). We train our model using the training sets: MNIST (60,000), USPS (7,291), standard SVHN train (73,257).
STL- Both CIFARIO 10 (Krizhevsky et al., 2009) and STL (Coates et al., 2011) are both 10-class image datasets. These two datasets contain nine overlapping classes. We remove the 'frog' class in CIFAR10 10 and the 'monkey' class in STL datasets as they have equivalent in the other dataset, resulting in 9-class problem. The STL images were down-scaled to 32x32 resolution to match that of CIFAR10.
SYN SIGNS-GGSRS We evaluated the adaptation from synthetic traffic sign dataset called SYN SIGNS (Moiseev et al., 2013) to real-world sign dataset called GTSRB (Stallkamp al., 2011). These datasets contain 43 classes.
We compare NASDA model with state-of-the-art DA methods: Deep Adaptation Network (DAN) (Long et al., 2015), Domain Adversarial Neural Network (DANN) (Ganin & Lempitsky, 2015), Domain Separation Network (DSN) (Bousmalis et al., 2016). Coupled Generative Adver- sarial Networks (CoGAN) (Liu & Tuzel, 2016), Maximum Classifier Discrepancy (MCD) (Saito al., 2018), Generate to Adapt (G2A) (Sankaranarayanan et al., 2018), Stochastic Neighborhood Embedding (d-SNE) (Xu et al., 2019b), Associative Domain Adaptation (ASSOC) (Haeusser et al., 2017).
7

---

Under review as a conference paper at ICLR 2021
Method
M -
U- M
$ + M
Avg(digits)
Method
SYN SIGNS - GTSRB
DAN
81.1
71.1
76.1
DAN
91.1
DANN
85.1
73.0
71.1
76.4
DANN
88.7
91.3*
82.7
DSN
CoGAN
91.2"
89.1
CORAL
86,9
MCD
94.2
94.1
96.2
94.8
MCD
94.4
G2A
95.0
90.8
92.4
92.7
ASSOC
82.8
SBADA-GAN
95.3
97.6
76.1
89.7
SRDA-RAN
93.6
d-SNE
99.0
98.7
96.5
98.1
DADRL
94.6
NASDA
98.0
98.7
98.6
98.4
NASDA
96.7
Table 2: Accuracy (%) on Digits and Traffic Signs for unsupervised domain adaptation.
4.2 EMPIRICAL RESULTS
Neural Architecture Search Results We show the neural architecture search results in Figure 2. We can observe that our model contains more "avg_pool"" and This will make our model more generic as the average pooling method smooths out the image and hence the model is not congested with sharp features and domain-specific features. We also show that NASDA model contains less parameters and takes less time converge compared with state-of -f-t-off- NAS architectures. Another interesting finding is that our NASDA contains more sequential connections in both Normal and Reduce cells when trained MNIST-USSS.
layer than other NAS model.
Unsupervised Domain Adaptation Results The UDA results for Digits and SYN SIGNS GTTSE reported in Table 2, with results of baselines directly reported from the original papers if the protocol is the same (numbers with indicates training on partial data). The NASDA model achieves 98.4% average accuracy for Digits dataset, outperforming other baselines. For SYN SIGNS~GTSRB task, our model gets comparable results with state-of-the-art baselines. The results demonstrate the effectiveness of our NASDA model on small images.
The UDA results on the CTL-RI recognition task are reported in Table 1. Our model achieves performance of 76.8%, outperforming all the baselines. To compare our search neural architecture with previous NAS models, we replace the neural architecture in G with other NAS models. Other training settings in the second phase are identical our model. As such, we derive NASNet+ II (Shah al., 2018). DARTS+Phase II (Liu al.,
II (Zoph et al., 2018), AmoebaNet+PPhene 2019a), and PDARTS+Phase II (Chen et al., 2019) models. The results in Table demonstrate that model outperform other NAS based model by large margin, which shows the effectiveness of our model in unsupervised domain adaptation. Specifically, we set PDARTS+Phase I/ as an ablation study demonstrate the effectiveness of our task-specific design in learning domain- aware features.
Analysis To dive deeper into the training process of our NASDA model, we plot the T-SNE embedding of the weights of C in USPS-MNIIS in Figure 3. This achieved by recording the weights of all the classifiers at each epoch. The black dot indicates epoch zero, which the common starting point. The color from light to dark corresponds to the epoch number from small to large. The T-SNE plots clearly show that the classifiers are diverged from each other, demonstrating the effectiveness of the second step of our NASDA training described Section 3.2.
Figure T-SNE for classifiers
5 CONCLUSION
In this paper, we first formulate novel dual-objective task of Neural Architecture Search for Domain Adaptation (NASDA) invigorate the design of transfer-aware network architectures. Towards tackling the NASDA task, we have proposed novel learning framework that leverages MK-MMD to guide the neural architecture search process. Instead of aligning the features from existing handcrafted backbones, our model directly searches for the optimal neural architecture specific for domain adaptation. Furthermore, we have introduced the ways to consolidate the feature generator, which is stacked from the searched architecture, in order to boost the UDA performance. Extensive empirical evaluations on UDA benchmarks have demonstrated the efficacy of the proposed model against several state-of-the-an domain adaptation algorithms.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
G Anandalingam and Terry L. Friesz. Hierarchical optimization: An introduction. Annals of Operations Research, 34(1):1-11, 1992.
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine learning, 79(1-2):151-175, 2010.
Gabriel Bender. Understanding and simplifying one-shot architecture search. International Confer- on Machine Learning, 2018.
Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan, and Dumitru Erhan. Domain separation networks. In Advances in neural information processing systems, pp. 343-351, 2016.
Han Cai, Tianyao Chen, Weinan Zhang, Yong Yu, and Jun Wang. Efficient architecture search by network transformation. In Thirly-Second AAAI conference on artificial intelligence, 2018a.
Han Cai, Jiacheng Yang, Weinan Zhang, Song Han, and Yong Yu. Path-level network transformation for efficient architecture search. International Conference on Machine Learning, 201 4018b.
Jiang-Chiehl Chen, Maxwell Collins, Yukun Zhu, George Papandreou, Barret Zoph, Florian Schroff, Hartwig Adam, and Jon Shlens. Searching for efficient multi-scale architectures for dense image prediction. In Advances in neural information processing systems, pp. 8699-8710, 2018.
Xin Chen, Lingxi Xie, Jun Wu, and Qi Tian. Progressive differentiable architecture search: Bridging the depth gap between search and evaluation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1294-1300 1304- 2019.
Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 215-121211 223, 2011.
Koby Crammer, Michael Kearns, and Jennifer Wortman. Learning from multiple sources. Journal of Machine Learning Research, I(Aug):157--1774 2008
Tobias Domhan, Jost Tobias Springenberg. and Frank Hutter. Speeding up automatic hyperparameter optimization of deep neural networks by extrapolation of learning curves. In Towenty-Foutth International Joint Conference on Artificial Intelligence, 2015.
Lixin Duan, Dong Xu, and Shih-Fu Chang. Exploiting web images for event recognition in con- sumer videos: A multiple source domain adaptation approach. In Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on, pp. 1338-1345. IEEE, 2012.
Thomas Elsken, Jan-Hendria Metzen, and Frank Hutter. Simple and efficient architecture search for convolutional neural networks. NIPS Workshop on Meta-Learning. 2017.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta- leannng for fast adaptation of deep networks. In Doina Precup and Yee Whye Tch (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 Proceedings of Machine Learning Research, pp. 1126-1135, International Convention Centre, Sydney, Australia, 06-11 Aug 2017. PMLR. URL http: btet::/proceesee http://pooceeeeneneeeeee .mlr. www.m.rrreeeseeee wrw.srresssveee 70 7 finm aa html.
Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, and Massimilano Pontil. Bilevel programming for hyperparameter optimization and meta-leamiig. ICML., 2018.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In Francis Bach and David Blei (eds.). Proceedings of the 32nd International Conference on Ma- chine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 1180-1189, Lille, France, 07-09 Jul 2015. PMLR. URL http: http::::///cceediieee ings. ml www.m...rrs///...... ganin15. html.
Muhammad Ghifary, W Bastiaan Kleijn, and Mengjie Zhang. Domain adaptive neural networks for object recognition. In Pacific Rim international conference on artificial intelligence, pp. 898-904. Springer, 2014.

---

Under review as a conference paper at ICLR 2021
Arthur Gretton, Karsten M Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alex Smola. A kernel method for the two-sample-probee problem. In Advances in neural information processing systems, pp. 511-5115500 2007.
Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. kernel two-sample test. Journal of Machine Learning Research, 13(Mar):723-773, 2012.
Haeusser, Frerix, A. Mordvintsev, and D. Cremers. Associative domain adaptation. 2017 IEEE International Conference on Computer Vision (ICCV), pp.
2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Judy Hoffman, Eric Tzeng, Taesung Park, Jun- Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell. CyCADA: Cycle-consistent adversarial domain adaptation. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1989-1998, Stockholmsmässan, press/ Stockholm Sweden, 10-15 Jul 2018. PMLR. URL httt:://proceet ngs. ml r..esss..rsss v80/hof htt/hoofmaal11 html.
Haifeng Jin, Qingquan Song, and Xia Hu. Auto-keras: An efficient neural architecture search system. In Proceedings of the 25th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pp. 1946-1956, 2019.
Tacksoo Kim, Moonsu Cha. Hyunsoo Kim, Jung Kwon Lee, and Jiwon Kim. Learning discover cross-domain relations with generative adversarial networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1857-1865, International Convention Centre, Sydney, Australia, 06-11 Aug 2017. PMLR. URL http:/ http::::/ppcceee.ng//// mlr www.mllrpp..... mtt.ppsss.rrs////////.. v70/king 17 html.
Aaron Klein, Stefan Falkner, Simon Bartels, Philipp Hennig, and Frank Hutter. Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets. In Aarti Singh and Jerry Zhu (eds.), Proceedings of the 20th International Conference Artificial Intelligence and Statistics, volume 54 Proceedings of Machine Learning Research, pp. 528-536, Fort Lauderdale, FL, USA, 20-22 Apr 2017. PMLR. URL http:://proceeddngg/ ml press/ 254/kleinl1 7 html.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Abhishek Kumar, Prasanna Sattigeri, Kahini Wadhawan, Leonid Karlinsky, Rogerio Feris, Bill Freeman, and Gregory Wornell. Co-regularized alignment for unsupervised domain adaptation. In Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 9345- 9356. Curran Associates, Inc., 2018. URL http: http://aapprssse nips ccwpapercpppp////pp 1e6-co-rrgularizee www.:larrizee-aaiimeeeteeee for pdf.
Alexander H. Liu, Yen-Cheng Liu, Yu-Ying Yeh, and Yu-Chiang Frank Wang. A unified feature disentangler for multi-domain image translation and manipulation. CoRR, abs/1809.0136/. 2018a. URL http:://///. //tp///aavvv///////////ee///aaaa org/ abs/ (809/ 01361.
Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shiens, Wei Hua, Li-Jia Li, Li Fei-Fei. Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 19-34, 2018b.
Hanxiao Liu, Karen Simonyan, Oriol Vinyals, Chrisantha Fernando, and Koray Kavukcuoglu. Hierar- chical representations for efficient architecture search. ICLR, 2018c.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. Interna- fional Conference on Learning Representations, 2019a.
10

---

Under review as a conference paper at ICLR 2021
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. Interna- tional Conference on Learning Representations, 2019b.
Ming-Yu Yu Liu and Oncel Tuzel. Coupled generative adversarial networks. In Advances in neural information processing systems, pp. 469-477777 2016.
Mingsheng Long, Yue Cao, Jianmin Wang. and Michael Jordan. Learning transferable features with deep adaptation networks. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 97-105, Lille, France, 07-09 Jul 2015. PMLR. URL http: http://ppooceednngs m mlr.press/ v37 long1 html.
Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I. Jordan. Deep transfer learning with joint adaptation networks. In Proceedings of the 34th International Conference Machine Learning, ICML. 2017, Sydney, NSW, Australia, 6-11 August 2017, pp. 2208-2217, 2017. URL http:/ http::/pproeee http:///oooeddddee ngs mlr. press / w000/oog1111 html.
Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael Jordan. Conditional adversarial domain adaptation. In Advances in Neural Information Processing Systems, pp. 1640-1650, 2018.
Jelena Luketina, Mathias Berglund, Klaus Greff, and Tapani Raiko. Scalable gradient- based tuning of continuous regularization hyperparameters, In International conference machine learning.
pp. 2952-29529 2016.
Dougal Maclaurin, David Duvenaud, and Ryan Adams. Gradient-based hyperparameter optimization through reversible learning. In International Conference on Machine Learning, pp. 21 21-2212 2015.
Yishay Mansour, Mehryar Mohri, Afshin Rostamizadeh, and A R. Domain adaptation with multiple sources. In D. Koller, D. Schuurmans, Y. Bengio, and L. Bottou (eds.), Advances in Neural Information Processing Systems 21, pp. 1041-1048. Curran Associates, Inc., 2009.
Boris Moiseev, Artem Konev, Alexander Chigorin, and Anton Konushin. Evaluation of traffic sign recognition methods trained on synthetically generated data. Jacques Blanc-Talon, Andrzej Kasinski, Wilfried Philips, Dan Popescu, and Paul Scheunders (eds.), Advanced Concepts for Intelligent Vision Systems, pp. 576-583, Cham, 2013. Springer International Publishing. ISBN 978-3-319-02895-8.
Fabian Pedregosa. Hyperparameter optimization with approximate gradient. 2016.
Xingchao Peng and Kate Saenko. Synthetic to real adaptation with generative correlation alignment networks. In 2018 IEEE Winter Conference on Applications of Computer Vision, WACV 2018, Lake Tahoe, NV, USA, March 12-15, 2018, pp. 1982-1991, 2018. doi: 10.1109/WACV.2018.0000 8.0019.. URL httpps://o// org/ / 0 1109, /WACV/CC..CC. 201 8 00219.
Xingehao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1406-1415, 2019.
Joaquin Quionero-Candela, Masashi Sugiyama, Anton Schwaighofer, and Neil D. Lawrence. Dataset Shift Machine Learning. The MIT Press, 2009. ISBN 0262170051, 9780262170055.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc Le. Aging evolution for image classifier architecture search. In AAAI 2019, 2019a.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc Le. Regularized evolution for image classifier architecture search. In Proceedings of the aaai conference artificial intelligence, volume 33, pp. 4780-4789, 2019b.
Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier discrepancy for unsupervised domain adaptation. The IEEÉ Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.

---

Under review as a conference paper at ICLR 2021
Swami Sankaranarayanan, Yogesh Balaji, Carlos D Castillo, and Rama Chellappa. Generate to adapt: Aligning domains using generative adversarial networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 8503-8512. 2018.
Shreyas Saxena and Jakob Verbeek. Convolutional neural fabrics. In Advances in Neural Information Processing Systems, pp. 4053-4061, 2016.
Syed Asif Raza Shah, Wenji Wu, Qiming Lu, Liang Zhang, Sajith Sasidharan, Phil DeMar, Chin Guok, John Macauley, Eric Pouyoul, Jin Kim, et al. Amoehanet: An sdn-enabled network service for big data science. Journal of Network and Computer Applications, 119:70-82, 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR, abs/1409. 1556, 2014.
Johannes Stallkamp, Mare Schlipsing, Jan Salmen, and Christian Igel. The German Traffic Sign Recognition Benchmark: A multi-class classification competition. IEEE International Joint Conference on Neural Networks, pp. 1453 1460, 2011.
Baochen Sun and Kate Saenko. Deep CORAL: correlation aligmment for deep domain adaptation. CoRR, abs/1607.00 00..1197999 19, 2016. URL http: /ttttpp//axxii... org/ abs 1607.01719.
Christian Szegedy, Wei Liu, Yangqing Jía, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-9, 2015.
Mingxing Tan and Quoe Le. EfficientNet: Rethinking model scaling for convolutional neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Research, pp. 6105-6114 14, Long Beach, California, USA, 09-15 Jun 2019. PMLR. URL http: 9t.1la.hmmmmm...m
Machine Learning, volume 97 of Proceedings of Machine Learning
//proceedings mlr press / /v97
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoe V Le. Mnasnet: Platform-aware neural architecture search for mobile. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2820-2828, 2019.
Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep domain confusion: Maximizing for domain invariance. arXiv preprint arXiv: 1412.3474 2014.
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Computer Vision and Pattern Recognition (CVPR), volume 1, pp. 4, 2017.
Tao Wei, Changhu Wang. Yong Rui, and Chang Wen Chen. Network morphism. In International Conference on Machine Learning, pp. 564-572, 2016.
Minghao Xu, Jian Zhang, Binghing Ni, Teng Li, Chengjie Wang. Qi Tian, and Wenjun Zhang. Adversarial domain adaptation with domain mixup. arXiv preprint arXiv: /9/2. 192000805 2019a.
Ruijia Xu, Ziliang Chen, Wangmeng Zuo. Junjie Yan, and Liang Lin. Deep cocktail network: Multi-source unsupervised domain adaptation with category shift. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3964-3973, 2018.
Xiang Xu, Xiong Zhou, Ragav Venkatesan, Gurumurthy Swaminathan, and Orchid Majumder. d- sne: Domain adaptation using stochastic neighborhood embedding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2497-2506, 2019b.
Zili Yi, Hao (Richard) Zhang, Ping Tan, and Minglun Gong. Dualgan: Unsupervised dual learning for image-to-image translation. In ICCV, pp. 2868-2876, 2017.
Arber Zela, Aaron Klein, Stefan Falkner, and Frank Hutter. Towards automated deep learning: Efficient joint neural architecture and hyperparameter search. ICML. 2018 Workshop on AutoMI. (AutoMI. 2018), 2018
Jun Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-imas translation using cycle-consistent adversarial networks. In Computer Vision (ICCV), 2017 IEEE International Conference on, 2017.
12

---

Under review as a conference paper at ICLR 2021
Barret Zoph and Quoe V Le. Neural architecture search with reinforcement learnig. ICLR, 2017.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoe V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8697-8710, 2018.
13


Reviews:
[{"review_id": "vGmbMiXfyk3", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThis work devises a two step process for searching optimal models for unsupervised domain adaptation. The first step involves a modification of DARTS where a discrepancy term between features for source and target domain is added to the negative-reward. The obtained feature transformer is then re-trained with an adversarial objective in order to ensure that it performs well across multiple classifiers.\n\nThe work addresses an interesting problem of automatically finding suitable architectures that transfer to unlabelled datasets. The novelty is involved in the formulation of a protocol that combines a modified DARTS objective with a post-processing step to get a suitable feature generator. The experiment section is well detailed. \n\nI have following concerns with respect to the current state of this work\n- It would be useful to get insights on the number of classifiers used in Phase-2 for current experiments along with the influence of this hyperparameter on the overall performance. \n- Is it possible to extend PDARTS with the MK-MMD regularisation term? Do authors have any insights in this direction?\n- Establishing the independent effectiveness of Phase-2 and regularisation term will add value to this work. These would also serve as relevant baselines. \n- If possible, could the authors detail the experimental setup wrt different NAS algorithms as it has been shown that often use of simple data augmentation techniques can lead to significant changes in performance. \n- Similarly, what part of the network was used as the feature generator? Does the performance vary with the number of layers used for G? Since the introduction of MK-MDD term led to models with reduced parameters as per the table in Figure 2 (c), it would be interesting to see the effect of the size of task-specific and feature-specific components on the performance. Or does the MK-MMD term incentivise the use of parameter free operators in the network as recently noted in Chen et al 2020 (https://arxiv.org/pdf/2002.05283.pdf)?\n- Minor remark: I think the paper can benefit from one through proofread for grammatical corrections and notational consistencies. For instance, in the line above Section 4, x_{t} should be changed to x^{t} and \\mathcal{L} in the line following eq(10) should include the superscript of “s”. Similarly, if the convention of underlined numbers in tables refers to second-best models that it should be made consistent across all tables. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "rgIQuNLVKkN", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summarize what the paper claims to contribute.**\n\nThis work introduces a two-step procedure for unsupervised domain adaptation; (1) neural architecture search for domain adaptation (NASDA) based on DARTS for neural architecture search and MK-MMD for differentiable unsupervised domain adaptation (2) adversarial training with a batch of classifiers. Based on induced architecture from the first component, it demonstrates unsupervised domain adaptation task supported by the second component.\n \n**List strong and weak points of the paper.**\n\n*strengths*\n\nIt is the first paper to adopt neural architecture search on unsupervised domain adaptation. The proposed combination, which is DARTS and MK-MMD, discovers a relevant neural architecture for unsupervised domain adaptation as shown in the experiment section.\n\n*weaknesses*\n\nThe novelty of both components is limited.\nThe novelty of the first component is not explained well in the paper. The first component seems to be a simple combination of DARTS with the additional loss function, MK-MMD. I can’t find any new modification on DARTS with MK-MMD in the paper. All descriptions on <section 3.2 searching neural architecture> are borrowed from the previous works, DARTS and MK-MMD.\nThe novelty of the second component is not related to neural architecture search for domain adaptation but multiple discriminators for generative adversarial networks. In this aspect, the ablation study on the second component is not concluded.\n\n**Clearly state your recommendation (accept or reject) with one or two key reasons for this choice**\n\nI give “Ok but not good enough - rejection (4)” to this paper. Although the proposed method deals with new and fancy stuff, the novelty of both proposed components is limited.\n\nI believe that the author should give more evidence to support the claims. If the author wants to insist that the first component, adopting MK-MMD on DARTS, is a good option, authors should prove that NASDA is effective for searching transferrable architecture by (1) giving a theoretical proof or (2) showing good performance on various practical benchmarks for domain adaptation. For example on argument (1), NASDA gives a tighter lower bound compared to a naive DA counterpart. For example on argument (2), NASDA shows good performance on several benchmarks such as office-31, ImageCLEF, and VisDA or multi-source domain benchmark.\n\nIf the author wants to prove the effectiveness of the second component, the author should compare the proposed method to (1) single discriminator and (2) the other multiple discriminator schemes. But even this comparison is not related to neural architecture search. \n\n**Provide additional feedback with the aim to improve the paper.**\n\n*Inaccurate descriptions on the paper*\n\nThe description of [conventional NAS ~ training and validation loss, respectively] in the second paragraph of the introduction section is that of DARTS (Liu, et. al., 2019), not generic NAS. For example, the work (Zoph & Le, 2017) generates a neural architecture to maximize the expected accuracy of the generated architecture on the validation set.", "extra_scores": {}, "extra_text": {}}, {"review_id": "qSANRlqHhjy", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In the work, the authors aim at improving the transferability of domain adaptation models from the perspective of neural architecture search. It consists of two phases, in particular, the first phase searches a neural architecture for domain adaptation based on a famous differentiable NAS method named DARTs, and the second phase develops an adversarial training method for domain adaptation by extending MCD to a multiple classifiers version. The empirical study evaluates the performance on some UDA tasks and shows the effectiveness of the introduced method.\n \nPros: \n1.\tTackling the problem of domain adaptation from the perspective of neural architecture search is interesting, and the authors successfully combine them together.\n\n2.\tThey not only minimize the validation loss of the source domain but also reduce the domain gap between the source and target since the labels in the target domain are not available. Though somewhat incremental,  this design works in some NAS benchmarks. On the contrary, the extension of MCD to multiple classifiers is somewhat straightforward.\n\n3.\tThis paper is well-written and easy to follow, and the empirical result on one small-scale benchmark named STL → CIFAR10 is impressive.\n \nConcerns: \n1.\tAs for the first phase, jointly minimizing the validation loss of the source domain and the MK-MMD based domain gap between the source and target, however, can NOT guarantee a small target error according to the analysis of [1], in which a simple counterexample is given in Figure 1 of [1]. My question is, by searching a differentiable neural architecture with the objective function defined in Equation 7, how can we guarantee it is suitable in the target domain and ensure a high target accuracy. The authors may want to search for a not so bad architecture first and then align the features across domains, however, the main purpose of this paper is the former but not the latter. Therefore, ablation studies are necessary to verify the effectiveness of the MK-MMD loss, such as comparing the architecture of the proposed method with that of DARTs. Note that, in my opinion, this comparison should be done without the second phase and different from the protocol of Table 1 to avoid the interference of the second phase.\n\n2. The technique novelty of the first phase of combining DARTS with MK-MMD is somewhat incremental and limited. It seems that the first phase is just an instance of famous DARTS (proposed in the field of NAS) with the additional popular MK-MMD loss function which also has already been proposed in the field of domain adaptation. For me, neither new ideas in the field of DA nor new techniques in the field of NAS are seen.\n\n3.\tAs for the second phase, the extension of MCD to multiple classifiers is straightforward and somewhat incremental. Meanwhile, I am still curious about the performance if we increase the classifier number from 2 to 4 or some larger. If we simply ensemble several classifiers without adversarial training, we have reasons to believe that we can achieve higher accuracy than the model with two classifiers since the effectiveness of ensemble learning has been verified in literature.\n\n4.\tAs shown in Table 2, the results of other NAS models with the same phase II are extremely low, however, there is no much difference between the architecture of NASDA with that of other NAS models. Especially, the architecture of NASNet [2] is also designed to be transferable across datasets (from Cifar to Imagenet), I am quite curious why this kind of architectures except NASDA are so weak. Would you mind giving some convincing analysis or explanations? It would be useful to shed more light on the differences with other NAS models, it is still unclear why these differences make such a big difference in the results. Meanwhile, the author has better show some evidence to justify that NAS can find more domain-invariant features than the classical architecture including VGG, ResNet, DenseNet.\n\n5.\tAnother key concern about the paper is the lack of rigorous experimentation to study the usefulness of the proposed method. State-of-the-art performances are more impressive and convincing when the benchmark adopts realistic settings. The typical domain adaptation benchmarks, such as Office-31, ImageCLEF-DA, VisDA, Office-Home, and DomainNet, as well as the newly proposed ImageNet-Sketch, have various category number and dataset scale. It would be better to conduct some experiments on these standard domain adaptation datasets and compare the proposed method with numerous baselines of these benchmarks.\n\n\n[1] Han Zhao⋆, Remi Tachet des Combes†, Kun Zhang⋆, Geoffrey J. Gordon⋆,† Carnegie Mellon University⋆, Microsoft Research Montrea. On Learning Invariant Representation for Domain Adaptation. In ICML 2019.\n\n[2] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8697– 8710, 2018\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "D0bF1PERwHm", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces an approach to search for the best network architecture for a domain adaptation task. This is achieved by following a differentiable architecture search strategy in which an additional loss function is included to account for the domain shift. Specifically, the loss function aims to minimize the discrepancy between feature representations from the two domains.\n\nStrengths:\n- To the best of my knowledge, this constitutes the first attempt at performing neural architecture search for domain adaptation\n- The results showing that accounting for the DA task during NAS outperforms simply taking a NAS model and applying a DA strategy to it are encouraging.\n\n\nWeaknesses:\n\nOriginality:\n- While I like the general concept of designing a NAS method specifically for domain adaptation, the proposed method lacks originality. In essence, it complements a standard NAS approach with standard DA strategies. This is fine, and the results are encouraging, but this seems to be a small contribution for an ICLR paper.\n\nMethodology:\n- Something bothers me in the formulation of Eqs. (1)-(2). While the discrepancy term of course depends on the architecture, it also depends on the weights w. It is therefore not very intuitive to my why the discrepancy term does not appear in the inner minimization problem of Eq. (2).\n- The other point that I find disturbing in the methodology is the fact that, during the second training stage, a different DA strategy, relying on adversarial training instead of MK-MMD, is used. Why not train the final model with MK-MMD? Alternatively, why not rely on adversarial training during the search phase? This unfortunately makes the overall method less principled.\n- As mentioned by the authors, one cannot optimized the target validation loss, because the target data is unsupervised. However, it would be interesting to evaluate what happens using pseudo-labeling, which has proven effective in DA, to approximate the target validation loss.\n\nExperiments:\n- In the comparison with the state-of-the-art DA methods in Tables 1 and 2, what architectures do these methods rely on? Are the comparisons fair, in the sense that all methods use networks with similar capacity?\n- The datasets used here are a bit outdated, and I recommend the authors to rely on newer benchmarks, e.g., VisDA or office-home.\n- The set of baselines differs across the different experiments. For example, d-SNE, which got the second best results on the digits datasets, is not reported for traffic signs and STL->CIFAR; DIRT-T that got the second-best results in Table 1 is not reported in Table 2.\n\nClarity:\n- The paper is reasonably easy to understand, but could benefit from thorough proofreading. \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work studies an intriguing problem of searching optimal architectures for unsupervised domain adaptation. It is based on a two-stage approach: (1) transferable architecture search via DARTS + MK-MMD; (2) transferable feature learning via Backbone + MCD.

The reviews for this paper are very insightful, constructive and of high quality. While all reviewers acknowledge the contribution of a new research problem, they unanimously have suggestions for further improving the paper:
- The novelty and soundness in the technical method are not fantastic. Authors should consider more elaborated loss designs for the approach, and unify the two stages with the same optimization objective.
- The empirical evaluation is by far not extensive and insightful. More stronger NAS approaches and larger-scale datasets should be included to give more evidence to support the claims that NAS-DA is better.
- A featured analysis of how non-iid architecture (as searched out by this work) differs from iid architecture would make the paper much more interesting.

Authors did not participate in the rebuttal and discussion phase.

AC scanned through the paper and believes that this paper studies a promising research direction, but the work cannot be accepted before addressing the reviewers' comments. The weaknesses are quite obvious and will have a high probability of being asked by the reviewers of the next conference. So the authors need to make sure that they substantially revise their work before submitting to yet another top venue.
Decision: Reject