Year: 2020

Abstract:
Deep networks have been used to learn transferable representations for domain adaptation. Existing deep domain adaptation methods systematically employ popular hand-crafted networks designed specifically for image-classification tasks, leading to sub-optimal domain adaptation performance. In this paper, we present Neural Architecture Search for Domain Adaptation (NASDA), a principle framework that leverages differentiable neural architecture search to derive the optimal network architecture for domain adaptation task. NASDA is designed with two novel training strategies: neural architecture search with multi-kernel Maximum Mean Discrepancy to derive the optimal architecture, and adversarial training between a feature generator and a batch of classifiers to consolidate the feature generator. We demonstrate experimentally that NASDA leads to state-of-the-art performance on several domain adaptation benchmarks.

heading: 1 INTRODUCTION
text: Supervised machine learning models (Φ) aim to minimize the empirical test error ( (Φ(x),y)) by optimizing Φ on training data (x) and ground truth labels (y), assuming that the training and testing data are sampled i.i.d from the same distribution. While in practical, the training and testing data are typically collected from related domains under different distributions, a phenomenon known as domain shift (or domain discrepancy) (Quionero-Candela et al., 2009). To avoid the cost of annotating each new test data, Unsupervised Domain Adaptation (UDA) tackles domain shift by transferring the knowledge learned from a rich-labeled source domain (P (xs,ys)) to the unlabeled target domain (Q(xt)). Recently unsupervised domain adaptation research has achieved significant progress with techniques like discrepancy alignment (Long et al., 2017; Tzeng et al., 2014; Ghifary et al., 2014; Peng & Saenko, 2018; Long et al., 2015; Sun & Saenko, 2016), adversarial alignment (Xu et al., 2019a; Liu & Tuzel, 2016; Tzeng et al., 2017; Liu et al., 2018a; Ganin & Lempitsky, 2015; Saito et al., 2018; Long et al., 2018), and reconstruction-based alignment (Yi et al., 2017; Zhu et al., 2017; Hoffman et al., 2018; Kim et al., 2017). While such models typically learn feature mapping from one domain (Φ(xs)) to another (Φ(xt)) or derive a joint representation across domains (Φ(xs)⊗ Φ(xt)), the developed models have limited capacities in deriving an optimal neural architecture specific for domain transfer.
To advance network designs, neural architecture search (NAS) automates the net architecture engineering process by reinforcement supervision (Zoph & Le, 2017) or through neuro-evlolution (Real et al., 2019a). Conventional NAS models aim to derive neural architecture α along with the network parameters w, by solving a bilevel optimization problem (Anandalingam & Friesz, 1992): Φα,w = arg minα Lval(w∗(α), α) s.t. w∗(α) = argminwLtrain(w,α), where Ltrain and Lval indicate the training and validation loss, respectively. While recent works demonstrate competitive performance on tasks such as image classification (Zoph et al., 2018; Liu et al., 2018c;b; Real et al., 2019b) and object detection (Zoph & Le, 2017), designs of existing NAS algorithms typically assume that the training and testing domain are sampled from the same distribution, neglecting the scenario where two data domains or multiple feature distributions are of interest.
To efficiently devise a neural architecture across different data domains, we propose a novel learning task called Neural Architecture Search for Domain Adaptation (NASDA). The ultimate goal of NASDA is to minimize the validation loss of the target domain (Ltval). We postulate that a solution to NASDA should not only minimize validation loss of the source domain (Lsval), but should also
reduce the domain gap between the source and target. To this end, we propose a new NAS learning schema:
Φα,w = argminαLsval(w∗(α), α) + disc(Φ∗(xs),Φ∗(xt)) (1) s.t. w∗(α) = argminw Lstrain(w,α) (2)
where Φ∗ = Φα,w∗(α), and disc(Φ∗(xs),Φ∗(xt)) denotes the domain discrepancy between the source and target. Note that in unsupervised domain adaptation, Lttrain and Ltval cannot be computed directly due to the lack of label in the target domain.
Inspired by the past works in NAS and unsupervised domain adaptation, we propose in this paper an instantiated NASDA model, which comprises of two training phases, as shown in Figure 1. The first is the neural architecture searching phase, aiming to derive an optimal neural architecture (α∗), following the learning schema of Equation 1,2. Inspired by Differentiable ARchiTecture Search (DARTS) (Liu et al., 2019a), we relax the search space to be continuous so that α can be optimized with respect to Lsval and disc(Φ(xs),Φ(xt)) by gradient descent. Specifically, we enhance the feature transferability by embedding the hidden representations of the task-specific layers to a reproducing kernel Hilbert space where the mean embeddings can be explicitly matched by minimizing disc(Φ(xs),Φ(xt)). We use multi-kernel Maximum Mean Discrepancy (MK-MMD) (Gretton et al., 2007) to evaluate the domain discrepancy.
The second training phase aims to learn a good feature generator with task-specific loss, based on the derived α∗ from the first phase. To establish this goal, we use the derived deep neural network (Φα∗ ) as the feature generator (G) and devise an adversarial training process between G and a batch of classifiers C. The high-level intuition is to first diversify C in the training process, and train G to generate features such that the diversified C can have similar outputs. The training process is similar to Maximum Classifier Discrepancy framework (MCD) (Saito et al., 2018) except that we extend the dual-classifier in MCD to an ensembling of multiple classifiers. Experiments on standard UDA benchmarks demonstrate the effectiveness of our derived NASDA model in achieving significant improvements over state-of-the-art methods.
Our contributions of this paper are highlighted as follows:
• We formulate a novel dual-objective task of Neural Architecture Search for Domain Adaptation (NASDA), which optimize neural architecture for unsupervised domain adaptation, concerning both source performance objective and transfer learning objective.
• We propose an instantiated NASDA model that comprises two training stages, aiming to derive optimal architecture parameters α∗ and feature extractor G, respectively. We are the first to show the effectiveness of MK-MMD in NAS process specified for domain adaptation.
• Extensive experiments on multiple cross-domain recognition tasks demonstrate that NASDA achieves significant improvements over traditional unsupervised domain adaptation models as well as state-of-the-art NAS-based methods.

heading: 2 RELATED WORK
text: Deep convolutional neural network has been dominating image recognition task. In recent years, many handcrafted architectures have been proposed, including VGG (Simonyan & Zisserman, 2014), ResNet (He et al., 2016), Inception (Szegedy et al., 2015), etc., all of which verifies the importance of human expertise in network design. Our work bridges domain adaptation and the emerging field of neural architecture search (NAS), a process of automating architecture engineering technique.
Neural Architecture Search Neural Architecture Search has become the mainstream approach to discover efficient and powerful network structures (Zoph & Le, 2017; Zoph et al., 2018). The automatically searched architectures have achieved highly competitive performance in tasks such as image classification (Liu et al., 2018c;b), object detection (Zoph et al., 2018), and semantic segmentation (Chen et al., 2018). Reinforce learning based NAS methods (Zoph & Le, 2017; Tan et al., 2019; Tan & Le, 2019) are usually computational intensive, thus hampering its usage with limited computational budget. To accelerate the search procedure, many techniques has been proposed and they mainly follow four directions: (1) estimating the actual performance with lower fidelities. Such lower fidelities include shorter training times (Zoph et al., 2018; Zela et al., 2018), training on a subset of the data (Klein et al., 2017), or on lower-resolution images. (2) estimating the performance based on the learning curve extrapolation. Domhan et al. (2015) propose to extrapolate initial learning curves and terminate those predicted to perform poorly. (3) initializing the novel architectures based on other well-trained architectures. Wei et al. (2016) introduce network morphisms to modify an architecture without changing the network objects, resulting in methods that only require a few GPU days (Elsken et al., 2017; Cai et al., 2018a; Jin et al., 2019; Cai et al., 2018b). (4) one-shot architecture search. One-shot NAS treats all architectures as different subgraphs of a supergraph and shares weights between architectures that have edges of this supergraph in common (Saxena & Verbeek, 2016; Liu et al., 2019b; Bender, 2018). DARTS (Liu et al., 2019a) places a mixture of candidate operations on each edge of the one-shot model and optimizes the weights of the candidate operations with a continuous relaxation of the search space. Inspired by DARTS (Liu et al., 2019a), our model employs differentiable architecture search to derive the optimal feature extractor for unsupervised domain adaptation.
Domain Adaptation Unsupervised domain adaptation (UDA) aims to transfer the knowledge learned from one or more labeled source domains to an unlabeled target domain. Various methods have been proposed, including discrepancy-based UDA approaches (Long et al., 2017; Tzeng et al., 2014; Ghifary et al., 2014; Peng & Saenko, 2018), adversary-based approaches (Liu & Tuzel, 2016; Tzeng et al., 2017; Liu et al., 2018a), and reconstruction-based approaches (Yi et al., 2017; Zhu et al., 2017; Hoffman et al., 2018; Kim et al., 2017). These models are typically designed to tackle single source to single target adaptation. Compared with single source adaptation, multi-source domain adaptation (MSDA) assumes that training data are collected from multiple sources. Originating from the theoretical analysis in (Ben-David et al., 2010; Mansour et al., 2009; Crammer et al., 2008), MSDA has been applied to many practical applications (Xu et al., 2018; Duan et al., 2012; Peng et al., 2019). Specifically, Ben-David et al. (2010) introduce an H∆H-divergence between the weighted combination of source domains and a target domain. These models are developed using the existing hand-crafted network architecture. This property limits the capacity and versatility of domain adaptation as the backbones to extract the features are fixed. In contrast, we tackle the UDA from a different perspective, not yet considered in the UDA literature. We propose a novel dual-objective model of NASDA, which optimize neural architecture for unsupervised domain adaptation. We are the first to show the effectiveness of MK-MMD in NAS process which is designed specifically for domain adaptation.

heading: 3 NEURAL ARCHITECTURE SEARCH FOR DOMAIN ADAPTATION
text: In unsupervised domain adaptation, we are given a source domain Ds = {(xsi ,ysi )} ns i=1 of ns labeled examples and a target domainDt = {xtj} nt j=1 of nt unlabeled examples. The source domain and target domain are sampled from joint distributions P (xs,ys) and Q(xt,yt), respectively. The goal of this paper is to leverage NAS to derive a deep networkG : x 7→ y, which is optimal for reducing the shifts in data distributions across domains, such that the target risk t (G) = E(xt,yt)∼Q [G (xt) 6= yt] is minimized. We will start by introducing some preliminary background in Section 3.1. We then describe how to incorporate the MK-MMD into the neural architecture searching framework in
Section 3.2. Finally, we introduce the adversarial training between our derived deep network and a batch of classifiers in Section 3.3. An overview of our model can be seen in Algorithm 1.

heading: 3.1 PRELIMINARY: DARTS
text: In this work, we leverage DARTS (Liu et al., 2019a) as our baseline framework. Our goal is to search for a robust cell and apply it to a network that is optimal to achieve domain alignment between Ds and Dt. Following Zoph et al. (2018), we search for a computation cell as the building block of the final architecture. The final convolutional network for domain adaptation can be stacked from the learned cell. A cell is defined as a directed acyclic graph (DAG) of L nodes, {xi}Ni=1, where each node x(i) is a latent representation and each directed edge e(i,j) is associated with some operation o(i,j) that transforms x(i). DARTS (Liu et al., 2019a) assumes that cells contain two input nodes and a single output node. To make the search space continuous, DARTS relaxes the categorical choice of a particular operation to a softmax over all possible operations and is thus formulated as:
ō(i,j)(x) = ∑ o∈O exp(α (i,j) o )∑ o′∈O exp(α (i,j) o′ ) o(x) (3)
where O denotes the set of candidate operations and i < j so that skip-connect can be applied. An intermediate node can be represented as xj = ∑ i<j o
(i,j)(xi). The task of architecture search then reduces to learning a set of continuous variables α = {α(i,j)}. At the end of search, a discrete architecture can be obtained by replacing each mixed operation ō(i,j) with the most likely operation, i.e., o∗ (i,j)
= argmaxo∈O α (i,j) o and α∗ = {o∗ (i,j)}.

heading: 3.2 SEARCHING NEURAL ARCHITECTURE
text: Denote by Ltrain and Lval the training loss and validation loss, respectively. Conventional neural architecture search models aim to derive Φα,w by solving a bilevel optimization problem (Anandalingam & Friesz, 1992): Φα,w = arg minα Lval(w∗(α), α) s.t. w∗(α) = argminwLtrain(w,α). While recent work (Zoph et al., 2018; Liu et al., 2018c) have show promising performance on tasks such as image classification and object detection, the existing models assume that the training data and testing data are sampled from the same distributions. Our goal is to jointly learn the architecture α and the weights w within all the mixed operations (e.g. weights of the convolution filters) so that the derived model Φw∗,α∗ can transfer knowledge fromDs toDt with some simple domain adapation guidence. Initialized by Equation 1, we leverage multi-kernel Maximum Mean Discrepancy (Gretton et al., 2007) to evaluate disc(Φ∗(xs),Φ∗(xt).
MK-MMD Denote by Hk be the Reproducing Kernel Hilbert Space (RKHS) endowed with a characteristic kernel k. The mean embedding of distribution p in Hk is a unique element µk(P ) such that Ex∼P f (x) = 〈f (x) , µk (P )〉Hk for all f ∈ Hk. The MK-MMD dk (P,Q) between probability distributions P and Q is defined as the RKHS distance between the mean embeddings of P and Q. The squared formulation of MK-MMD is defined as
d2k (P,Q) , ∥∥EP [Φα (xs)]−EQ [Φα (xt)]∥∥2Hk . (4)
In this paper, we consider the case of combining Gaussian kernels with injective functions fΦ, where k(x, x′) = exp(−‖fΦ(x) − fΦ(x)′‖2). Inspired by Long et al. (2015), the characteristic kernel associated with the feature map Φ, k (xs,xt) = 〈Φ (xs) ,Φ (xt)〉, is defined as the convex combination of n positive semidefinite kernels {ku},
K , { k =
n∑ u=1 βuku : n∑ u=1 βu = 1, βu > 0,∀u
} , (5)
where the constraints on {βu} are imposed to guarantee that the k is characteristic. In practice we use finite samples from distributions to estimate MMD distance. Given Xs = {xs1, · · · ,xsm} ∼ P and Xt = {xt1, · · · ,xtm} ∼ Q, one estimator of d2k(P,Q) is
d̂2k(P,Q) = 1( m 2 ) ∑ i 6=i′ k(xsi,x s′ i)− 2( m 2 ) ∑ i 6=j k(xsi ,x t j) + 1( m 2 ) ∑ j 6=j′ k(xtj ,x t′ j). (6)
Algorithm 1 Neural Architecture Search for Domain Adaptation Phase I: Searching Neural Architecture 1: Create a mixed operation o(i,j) parametrized by α(i,j) for each edge (i, j) 2: while not converged do 3: Update architecture α by ∂∂αL s val ( w − ξ ∂∂wL s train(w,α), α ) + λ ∂∂α ( d̂2k (Φ(x s),Φ(xt)) )
4: Update weights w by descending ∂∂wL s train(w,α) 5: end while 6: Derive the final architecture based on the learned α∗.
Phase II: Adversarial Training for Domain Adaptation 1: Stack feature generator G based on α∗, initialize classifiers C 2: while not converged do 3: Step one: Train G and C with Ls(xs,ys) = −E(xs,ys)∼Ds ∑K k=1 1[k=ys] log p(y
s|xs) 4: Step two: Fix G, train C with loss: Ls(xs,ys)− Ladv(xt)(Eq. 13) 5: Step three: Fix C, train G with loss: Ladv(xt) 6: end while
The merit of multi-kernel MMD lies in its differentiability such that it can be easily incorporated into the deep network. However, the computation of the d̂2k(P,Q) incurs a complexity of O(m
2), which is undesirable in the differentiable architecture search framework. In this paper, we use the unbiased estimation of MK-MMD (Gretton et al., 2012) which can be computed with linear complexity.
NAS for Domain Adaptation Denote by Lstrain and Lsval the training loss and validation loss on the source domain, respectively. Both losses are affected by the architecture α as well as by the weights w in the network. The goal for NASDA is to find α∗ that minimizes the validation loss Ltval(w∗, α∗) on the target domain, where the weights w∗ associated with the architecture are obtained by minimizing the training loss w∗ = argminw Lstrain(w,α∗). Due to the lack of labels in the target domain, it is prohibitive to compute Ltval directly, hampering the assumption of previous gradient-based NAS algorithms (Liu et al., 2019a; Chen et al., 2019). Instead, we derive α∗ by minimizing the validation loss Lsval(w∗, α∗) on the source domain plus the domain discrepancy, disc(Φ(xs),Φ(xt)), as shown in Equation 1.
Inspired by the gradient-based hyperparameter optimization (Franceschi et al., 2018; Pedregosa, 2016; Maclaurin et al., 2015), we set the architecture parameters α as a special type of hyperparameter. This implies a bilevel optimization problem (Anandalingam & Friesz, 1992) with α as the upper-level variable and w as the lower-level variable. In practice, we utilize the MK-MMD to evaluate the domain discrepancy. The optimization can be summarized as follows:
Φα,w = argminα ( Lsval(w∗(α), α) + λd̂2k ( Φ(xs),Φ(xt) ) ) (7)
s.t. w∗(α) = argminw Lstrain(w,α) (8)
where λ is the trade-off hyperparameter between the source validation loss and the MK-MMD loss.
Approximate Architecture Search Equation 7,8 imply that directly optimizing the architecture gradient is prohibitive due to the expensive inner optimization. Inspired by DARTS (Liu et al., 2019a), we approximate w∗(α) by adapting w using only a single training step, without solving the optimization in Equation 8 by training until convergence. This idea has been adopted and proven to be effective in meta-learning for model transfer (Finn et al., 2017), gradient-based hyperparameter tuning (Luketina et al., 2016) and unrolled generative adversarial networks. We therefore propose a simple approximation scheme as follows:
∂
∂α
( Lsval(w∗(α), α) + λd̂2k ( Φ(xs),Φ(xt) ) ) ≈ ∂ ∂α Lsval ( w − ξ ∂ ∂w Lstrain(w,α), α ) + λ ∂ ∂α ( d̂2k ( Φ(xs),Φ(xt) ) ) (9)
where w − ξ ∂∂wL s train(w,α) denotes weight for one-step forward model and ξ is the learning rate for a step of inner optimization. Note Equation 9 reduces to ∇αLval(w,α) if w is already a local optimum for the inner optimization and thus∇wLtrain(w,α) = 0.
The second term of Equation 9 can be computed directly with some forward and backward passes. For the first term, applying chain rule to the approximate architecture gradient yields
∂
∂α Lsval(w′, α)− ξ ( ∂2 ∂α∂w Lstrain(w,α) ∂ ∂w′ Lsval(w′, α) ) (10)
wherew′ = w−ξ ∂∂wLtrain(w,α). The expression above contains an expensive matrix-vector product in its second term. We leverage the central difference approximation to reduce the computation complexity. Specifically, let η be a small scalar and w± = w ± η ∂∂w′L s val(w ′, α). Then:
∂2
∂α∂w Lstrain(w,α)
∂
∂w′ Lsval(w′, α) ≈
∂ ∂αLtrain(w +, α)− ∂∂αLtrain(w −, α)
2η (11)
Evaluating the central difference only requires two forward passes for the weights and two backward passes for α, reducing the complexity from quadratic to linear.

heading: 3.3 ADVERSARIAL TRAINING FOR DOMAIN ADAPTATION
text: By neural architecture searching from Section 3.2, we have derived the optimal cell structure (α∗) for domain adaptation. We then stack the cells to derive our feature generator G. In this section, we describe how do we consolidate G by an adversarial training of G and the classifiers C. Assume C includes N independent classifiers {C(i)}Ni=1 and denote pi(y|x) as the K-way propabilistic outputs of C(i), where K is the category number.
The high-level intuition is to consolidate the feature generator G such that it can make the diversified C generate similar outputs. To this end, our training process include three steps: (1) train G and C on Ds to obtain task-specific features, (2) fix G and train C to make {C(i)}Ni=1 have diversified output, (3) fix C and train G to minimize the output discrepancy between C. Related techniques have been used in Saito et al. (2018); Kumar et al. (2018).
First, we train both G and C to classify the source samples correctly with cross-entropy loss. This step is crucial as it enables G and C to extract the task-specific features. The training objective is min G,C Ls(xs,ys) and the loss function is defined as follows:
Ls(xs,ys) = −E(xs,ys)∼Ds K∑ k=1 1[k=ys] log p(y s|xs) (12)
In the second step, we are aiming to diversify C. To establish this goal, we fix G and train C to increase the discrepancy of C’s output. To avoid mode collapse (e.g. C(1) outputs all zeros and C(2) output all ones), we add Ls(xs,ys) as a regularizer in the training process. The high-level intuition is that we do not expect C to forget the information learned in the first step in the training process. The training objective is min
C Ls(xs,ys)− Ladv(xt), where the adversarial loss is defined as:
Ladv(xt) = Ext∼Dt N−1∑ i=1 N∑ j=i+1 ‖(pi(y|xt)− pj(y|xt)‖1 (13)
In the last step, we are trying to consolidate the feature generator G by training G to extract generalizable representations such that the discrepancy of C’s output is minimized. To achieve this goal, we fix the diversified classifiers C and train G with the adversarial loss (defined in Equation 13). The training objective is min
G Ladv(xt).
4 EXPERIMENTS
We compare the proposed NASDA model with many stateof-the-art UDA baselines on multiple benchmarks. In the main paper, we only report major results; more details are provided in the supplementary material. All of our experiments are implemented in the PyTorch platform.
In the architecture search phase, we use λ=1 for all the searching experiments. We leverage the ReLU-Conv-BN order for convolutional operations, and each separable convolution is always applied twice. Our search space O includes the following operations: 3 × 3 and 5 × 5 separable convolutions, 3 × 3 and 5 × 5 dilated separable convolutions, 3 × 3 max pooling, identity, and zero. Our convolutional cell consists of N = 7 nodes. Cells located at the 13 and 2 3 of the total depth of the network are reduction cells. The architecture encoding therefore is
(αnormal, αreduce), where αnormal is shared by all the normal cells and αreduce is shared by all the reduction cells.

heading: 4.1 SETUP
text: Digits We investigate three digits datasets: MNIST, USPS, and Street View House Numbers (SVHN). We adopt the evaluation protocol of CyCADA (Hoffman et al., 2018) with three transfer tasks: USPS to MNIST (U→ M), MNIST to USPS (M→ U), and SVHN to MNIST (S→ M). We train our model using the training sets: MNIST (60,000), USPS (7,291), standard SVHN train (73,257).
STL→CIFAR10 Both CIFAR10 (Krizhevsky et al., 2009) and STL (Coates et al., 2011) are both 10-class image datasets. These two datasets contain nine overlapping classes. We remove the ‘frog’ class in CIFAR10 and the ‘monkey’ class in STL datasets as they have no equivalent in the other dataset, resulting in a 9-class problem. The STL images were down-scaled to 32×32 resolution to match that of CIFAR10.
SYN SIGNS→GTSRB We evaluated the adaptation from synthetic traffic sign dataset called SYN SIGNS (Moiseev et al., 2013) to real-world sign dataset called GTSRB (Stallkamp et al., 2011). These datasets contain 43 classes.
We compare our NASDA model with state-of-the-art DA methods: Deep Adaptation Network (DAN) (Long et al., 2015), Domain Adversarial Neural Network (DANN) (Ganin & Lempitsky, 2015), Domain Separation Network (DSN) (Bousmalis et al., 2016), Coupled Generative Adversarial Networks (CoGAN) (Liu & Tuzel, 2016), Maximum Classifier Discrepancy (MCD) (Saito et al., 2018), Generate to Adapt (G2A) (Sankaranarayanan et al., 2018), Stochastic Neighborhood Embedding (d-SNE) (Xu et al., 2019b), Associative Domain Adaptation (ASSOC) (Haeusser et al., 2017).

heading: 4.2 EMPIRICAL RESULTS
text: Neural Architecture Search Results We show the neural architecture search results in Figure 2. We can observe that our model contains more “avg_pool” and “5x5_conv” layer than other NAS model. This will make our model more generic as the average pooling method smooths out the image and hence the model is not congested with sharp features and domain-specific features. We also show that our NASDA model contains less parameters and takes less time to converge compared with state-of-the-art NAS architectures. Another interesting finding is that our NASDA contains more sequential connections in both Normal and Reduce cells when trained on MNIST→USPS. Unsupervised Domain Adaptation Results The UDA results for Digits and SYN SIGNS→GTSRB are reported in Table 2, with results of baselines directly reported from the original papers if the protocol is the same (numbers with ∗ indicates training on partial data). The NASDA model achieves a 98.4% average accuracy for Digits dataset, outperforming other baselines. For SYN SIGNS→GTSRB task, our model gets comparable results with state-of-the-art baselines. The results demonstrate the effectiveness of our NASDA model on small images.
The UDA results on the STL→CIFAR10 recognition task are reported in Table 1. Our model achieves a performance of 76.8%, outperforming all the baselines. To compare our search neural architecture with previous NAS models, we replace the neural architecture in G with other NAS models. Other training settings in the second phase are identical to our model. As such, we derive NASNet+Phase II (Zoph et al., 2018), AmoebaNet+Phase II (Shah et al., 2018) , DARTS+Phase II (Liu et al., 2019a), and PDARTS+Phase II (Chen et al., 2019) models. The results in Table 1 demonstrate that our model outperform other NAS based model by a large margin, which shows the effectiveness of our model in unsupervised domain adaptation. Specifically, we set PDARTS+Phase II as an ablation study to demonstrate the effectiveness of our task-specific design in learning domain-adaption aware features.
Analysis To dive deeper into the training process of our NASDA model, we plot the T-SNE embedding of the weights of C in USPS→MNIST in Figure 3. This is achieved by recording the weights of all the classifiers at each epoch. The black dot indicates epoch zero, which is the common starting point. The color from light to dark corresponds to the epoch number from small to large. The T-SNE plots clearly show that the classifiers are diverged from each other, demonstrating the effectiveness of the second step of our NASDA training described in Section 3.2.

heading: 5 CONCLUSION
text: In this paper, we first formulate a novel dual-objective task of Neural Architecture Search for Domain Adaptation (NASDA) to invigorate the design of transfer-aware network architectures. Towards tackling the NASDA task, we have proposed a novel learning framework that leverages MK-MMD to guide the neural architecture search process. Instead of aligning the features from existing handcrafted backbones, our model directly searches for the optimal neural architecture specific for domain adaptation. Furthermore, we have introduced the ways to consolidate the feature generator, which is stacked from the searched architecture, in order to boost the UDA performance. Extensive empirical evaluations on UDA benchmarks have demonstrated the efficacy of the proposed model against several state-of-the-art domain adaptation algorithms.

Review summaries:
This paper introduces an approach to search for the best network architecture for a domain adaptation task. This is achieved by following a differentiable architecture search strategy in which an additional loss function is included to account for the domain shift. Specifically, the loss function aims to minimize the discrepancy between feature representations from the two domains.

In the work, the authors aim at improving the transferability of domain adaptation models from the perspective of neural architecture search. It consists of two phases, in particular, the first phase searches a neural architecture for domain adaptation based on a famous differentiable NAS method named DARTs, and the second phase develops an adversarial training method for domain adaptation by extending MCD to a multiple classifiers version. The empirical study evaluates the performance on some UDA tasks and shows the effectiveness of the introduced method.

This work devises a two step process for searching optimal models for unsupervised domain adaptation. The first step involves a modification of DARTS where a discrepancy term between features for source and target domain is added to the negative-reward. The obtained feature transformer is then re-trained with an adversarial objective in order to ensure that it performs well across multiple classifiers.