Network Architecture Search for Domain Adaptation

1 INTRODUCTION . Supervised machine learning models ( Φ ) aim to minimize the empirical test error ( ( Φ ( x ) , y ) ) by optimizing Φ on training data ( x ) and ground truth labels ( y ) , assuming that the training and testing data are sampled i.i.d from the same distribution . While in practical , the training and testing data are typically collected from related domains under different distributions , a phenomenon known as domain shift ( or domain discrepancy ) ( Quionero-Candela et al. , 2009 ) . To avoid the cost of annotating each new test data , Unsupervised Domain Adaptation ( UDA ) tackles domain shift by transferring the knowledge learned from a rich-labeled source domain ( P ( xs , ys ) ) to the unlabeled target domain ( Q ( xt ) ) . Recently unsupervised domain adaptation research has achieved significant progress with techniques like discrepancy alignment ( Long et al. , 2017 ; Tzeng et al. , 2014 ; Ghifary et al. , 2014 ; Peng & Saenko , 2018 ; Long et al. , 2015 ; Sun & Saenko , 2016 ) , adversarial alignment ( Xu et al. , 2019a ; Liu & Tuzel , 2016 ; Tzeng et al. , 2017 ; Liu et al. , 2018a ; Ganin & Lempitsky , 2015 ; Saito et al. , 2018 ; Long et al. , 2018 ) , and reconstruction-based alignment ( Yi et al. , 2017 ; Zhu et al. , 2017 ; Hoffman et al. , 2018 ; Kim et al. , 2017 ) . While such models typically learn feature mapping from one domain ( Φ ( xs ) ) to another ( Φ ( xt ) ) or derive a joint representation across domains ( Φ ( xs ) ⊗ Φ ( xt ) ) , the developed models have limited capacities in deriving an optimal neural architecture specific for domain transfer . To advance network designs , neural architecture search ( NAS ) automates the net architecture engineering process by reinforcement supervision ( Zoph & Le , 2017 ) or through neuro-evlolution ( Real et al. , 2019a ) . Conventional NAS models aim to derive neural architecture α along with the network parameters w , by solving a bilevel optimization problem ( Anandalingam & Friesz , 1992 ) : Φα , w = arg minα Lval ( w∗ ( α ) , α ) s.t . w∗ ( α ) = argminwLtrain ( w , α ) , where Ltrain and Lval indicate the training and validation loss , respectively . While recent works demonstrate competitive performance on tasks such as image classification ( Zoph et al. , 2018 ; Liu et al. , 2018c ; b ; Real et al. , 2019b ) and object detection ( Zoph & Le , 2017 ) , designs of existing NAS algorithms typically assume that the training and testing domain are sampled from the same distribution , neglecting the scenario where two data domains or multiple feature distributions are of interest . To efficiently devise a neural architecture across different data domains , we propose a novel learning task called Neural Architecture Search for Domain Adaptation ( NASDA ) . The ultimate goal of NASDA is to minimize the validation loss of the target domain ( Ltval ) . We postulate that a solution to NASDA should not only minimize validation loss of the source domain ( Lsval ) , but should also reduce the domain gap between the source and target . To this end , we propose a new NAS learning schema : Φα , w = argminαLsval ( w∗ ( α ) , α ) + disc ( Φ∗ ( xs ) , Φ∗ ( xt ) ) ( 1 ) s.t . w∗ ( α ) = argminw Lstrain ( w , α ) ( 2 ) where Φ∗ = Φα , w∗ ( α ) , and disc ( Φ∗ ( xs ) , Φ∗ ( xt ) ) denotes the domain discrepancy between the source and target . Note that in unsupervised domain adaptation , Lttrain and Ltval can not be computed directly due to the lack of label in the target domain . Inspired by the past works in NAS and unsupervised domain adaptation , we propose in this paper an instantiated NASDA model , which comprises of two training phases , as shown in Figure 1 . The first is the neural architecture searching phase , aiming to derive an optimal neural architecture ( α∗ ) , following the learning schema of Equation 1,2 . Inspired by Differentiable ARchiTecture Search ( DARTS ) ( Liu et al. , 2019a ) , we relax the search space to be continuous so that α can be optimized with respect to Lsval and disc ( Φ ( xs ) , Φ ( xt ) ) by gradient descent . Specifically , we enhance the feature transferability by embedding the hidden representations of the task-specific layers to a reproducing kernel Hilbert space where the mean embeddings can be explicitly matched by minimizing disc ( Φ ( xs ) , Φ ( xt ) ) . We use multi-kernel Maximum Mean Discrepancy ( MK-MMD ) ( Gretton et al. , 2007 ) to evaluate the domain discrepancy . The second training phase aims to learn a good feature generator with task-specific loss , based on the derived α∗ from the first phase . To establish this goal , we use the derived deep neural network ( Φα∗ ) as the feature generator ( G ) and devise an adversarial training process between G and a batch of classifiers C. The high-level intuition is to first diversify C in the training process , and train G to generate features such that the diversified C can have similar outputs . The training process is similar to Maximum Classifier Discrepancy framework ( MCD ) ( Saito et al. , 2018 ) except that we extend the dual-classifier in MCD to an ensembling of multiple classifiers . Experiments on standard UDA benchmarks demonstrate the effectiveness of our derived NASDA model in achieving significant improvements over state-of-the-art methods . Our contributions of this paper are highlighted as follows : • We formulate a novel dual-objective task of Neural Architecture Search for Domain Adaptation ( NASDA ) , which optimize neural architecture for unsupervised domain adaptation , concerning both source performance objective and transfer learning objective . • We propose an instantiated NASDA model that comprises two training stages , aiming to derive optimal architecture parameters α∗ and feature extractor G , respectively . We are the first to show the effectiveness of MK-MMD in NAS process specified for domain adaptation . • Extensive experiments on multiple cross-domain recognition tasks demonstrate that NASDA achieves significant improvements over traditional unsupervised domain adaptation models as well as state-of-the-art NAS-based methods . 2 RELATED WORK . Deep convolutional neural network has been dominating image recognition task . In recent years , many handcrafted architectures have been proposed , including VGG ( Simonyan & Zisserman , 2014 ) , ResNet ( He et al. , 2016 ) , Inception ( Szegedy et al. , 2015 ) , etc. , all of which verifies the importance of human expertise in network design . Our work bridges domain adaptation and the emerging field of neural architecture search ( NAS ) , a process of automating architecture engineering technique . Neural Architecture Search Neural Architecture Search has become the mainstream approach to discover efficient and powerful network structures ( Zoph & Le , 2017 ; Zoph et al. , 2018 ) . The automatically searched architectures have achieved highly competitive performance in tasks such as image classification ( Liu et al. , 2018c ; b ) , object detection ( Zoph et al. , 2018 ) , and semantic segmentation ( Chen et al. , 2018 ) . Reinforce learning based NAS methods ( Zoph & Le , 2017 ; Tan et al. , 2019 ; Tan & Le , 2019 ) are usually computational intensive , thus hampering its usage with limited computational budget . To accelerate the search procedure , many techniques has been proposed and they mainly follow four directions : ( 1 ) estimating the actual performance with lower fidelities . Such lower fidelities include shorter training times ( Zoph et al. , 2018 ; Zela et al. , 2018 ) , training on a subset of the data ( Klein et al. , 2017 ) , or on lower-resolution images . ( 2 ) estimating the performance based on the learning curve extrapolation . Domhan et al . ( 2015 ) propose to extrapolate initial learning curves and terminate those predicted to perform poorly . ( 3 ) initializing the novel architectures based on other well-trained architectures . Wei et al . ( 2016 ) introduce network morphisms to modify an architecture without changing the network objects , resulting in methods that only require a few GPU days ( Elsken et al. , 2017 ; Cai et al. , 2018a ; Jin et al. , 2019 ; Cai et al. , 2018b ) . ( 4 ) one-shot architecture search . One-shot NAS treats all architectures as different subgraphs of a supergraph and shares weights between architectures that have edges of this supergraph in common ( Saxena & Verbeek , 2016 ; Liu et al. , 2019b ; Bender , 2018 ) . DARTS ( Liu et al. , 2019a ) places a mixture of candidate operations on each edge of the one-shot model and optimizes the weights of the candidate operations with a continuous relaxation of the search space . Inspired by DARTS ( Liu et al. , 2019a ) , our model employs differentiable architecture search to derive the optimal feature extractor for unsupervised domain adaptation . Domain Adaptation Unsupervised domain adaptation ( UDA ) aims to transfer the knowledge learned from one or more labeled source domains to an unlabeled target domain . Various methods have been proposed , including discrepancy-based UDA approaches ( Long et al. , 2017 ; Tzeng et al. , 2014 ; Ghifary et al. , 2014 ; Peng & Saenko , 2018 ) , adversary-based approaches ( Liu & Tuzel , 2016 ; Tzeng et al. , 2017 ; Liu et al. , 2018a ) , and reconstruction-based approaches ( Yi et al. , 2017 ; Zhu et al. , 2017 ; Hoffman et al. , 2018 ; Kim et al. , 2017 ) . These models are typically designed to tackle single source to single target adaptation . Compared with single source adaptation , multi-source domain adaptation ( MSDA ) assumes that training data are collected from multiple sources . Originating from the theoretical analysis in ( Ben-David et al. , 2010 ; Mansour et al. , 2009 ; Crammer et al. , 2008 ) , MSDA has been applied to many practical applications ( Xu et al. , 2018 ; Duan et al. , 2012 ; Peng et al. , 2019 ) . Specifically , Ben-David et al . ( 2010 ) introduce an H∆H-divergence between the weighted combination of source domains and a target domain . These models are developed using the existing hand-crafted network architecture . This property limits the capacity and versatility of domain adaptation as the backbones to extract the features are fixed . In contrast , we tackle the UDA from a different perspective , not yet considered in the UDA literature . We propose a novel dual-objective model of NASDA , which optimize neural architecture for unsupervised domain adaptation . We are the first to show the effectiveness of MK-MMD in NAS process which is designed specifically for domain adaptation . 3 NEURAL ARCHITECTURE SEARCH FOR DOMAIN ADAPTATION . In unsupervised domain adaptation , we are given a source domain Ds = { ( xsi , ysi ) } ns i=1 of ns labeled examples and a target domainDt = { xtj } nt j=1 of nt unlabeled examples . The source domain and target domain are sampled from joint distributions P ( xs , ys ) and Q ( xt , yt ) , respectively . The goal of this paper is to leverage NAS to derive a deep networkG : x 7→ y , which is optimal for reducing the shifts in data distributions across domains , such that the target risk t ( G ) = E ( xt , yt ) ∼Q [ G ( xt ) 6= yt ] is minimized . We will start by introducing some preliminary background in Section 3.1 . We then describe how to incorporate the MK-MMD into the neural architecture searching framework in Section 3.2 . Finally , we introduce the adversarial training between our derived deep network and a batch of classifiers in Section 3.3 . An overview of our model can be seen in Algorithm 1 .

Summary: In the work, the authors aim at improving the transferability of domain adaptation models from the perspective of neural architecture search. It consists of two phases, in particular, the first phase searches a neural architecture for domain adaptation based on a famous differentiable NAS method named DARTs, and the second phase develops an adversarial training method for domain adaptation by extending MCD to a multiple classifiers version. The empirical study evaluates the performance on some UDA tasks and shows the effectiveness of the introduced method.