Network Architecture Search for Domain Adaptation

1 INTRODUCTION . Supervised machine learning models ( Φ ) aim to minimize the empirical test error ( ( Φ ( x ) , y ) ) by optimizing Φ on training data ( x ) and ground truth labels ( y ) , assuming that the training and testing data are sampled i.i.d from the same distribution . While in practical , the training and testing data are typically collected from related domains under different distributions , a phenomenon known as domain shift ( or domain discrepancy ) ( Quionero-Candela et al. , 2009 ) . To avoid the cost of annotating each new test data , Unsupervised Domain Adaptation ( UDA ) tackles domain shift by transferring the knowledge learned from a rich-labeled source domain ( P ( xs , ys ) ) to the unlabeled target domain ( Q ( xt ) ) . Recently unsupervised domain adaptation research has achieved significant progress with techniques like discrepancy alignment ( Long et al. , 2017 ; Tzeng et al. , 2014 ; Ghifary et al. , 2014 ; Peng & Saenko , 2018 ; Long et al. , 2015 ; Sun & Saenko , 2016 ) , adversarial alignment ( Xu et al. , 2019a ; Liu & Tuzel , 2016 ; Tzeng et al. , 2017 ; Liu et al. , 2018a ; Ganin & Lempitsky , 2015 ; Saito et al. , 2018 ; Long et al. , 2018 ) , and reconstruction-based alignment ( Yi et al. , 2017 ; Zhu et al. , 2017 ; Hoffman et al. , 2018 ; Kim et al. , 2017 ) . While such models typically learn feature mapping from one domain ( Φ ( xs ) ) to another ( Φ ( xt ) ) or derive a joint representation across domains ( Φ ( xs ) ⊗ Φ ( xt ) ) , the developed models have limited capacities in deriving an optimal neural architecture specific for domain transfer . To advance network designs , neural architecture search ( NAS ) automates the net architecture engineering process by reinforcement supervision ( Zoph & Le , 2017 ) or through neuro-evlolution ( Real et al. , 2019a ) . Conventional NAS models aim to derive neural architecture α along with the network parameters w , by solving a bilevel optimization problem ( Anandalingam & Friesz , 1992 ) : Φα , w = arg minα Lval ( w∗ ( α ) , α ) s.t . w∗ ( α ) = argminwLtrain ( w , α ) , where Ltrain and Lval indicate the training and validation loss , respectively . While recent works demonstrate competitive performance on tasks such as image classification ( Zoph et al. , 2018 ; Liu et al. , 2018c ; b ; Real et al. , 2019b ) and object detection ( Zoph & Le , 2017 ) , designs of existing NAS algorithms typically assume that the training and testing domain are sampled from the same distribution , neglecting the scenario where two data domains or multiple feature distributions are of interest . To efficiently devise a neural architecture across different data domains , we propose a novel learning task called Neural Architecture Search for Domain Adaptation ( NASDA ) . The ultimate goal of NASDA is to minimize the validation loss of the target domain ( Ltval ) . We postulate that a solution to NASDA should not only minimize validation loss of the source domain ( Lsval ) , but should also reduce the domain gap between the source and target . To this end , we propose a new NAS learning schema : Φα , w = argminαLsval ( w∗ ( α ) , α ) + disc ( Φ∗ ( xs ) , Φ∗ ( xt ) ) ( 1 ) s.t . w∗ ( α ) = argminw Lstrain ( w , α ) ( 2 ) where Φ∗ = Φα , w∗ ( α ) , and disc ( Φ∗ ( xs ) , Φ∗ ( xt ) ) denotes the domain discrepancy between the source and target . Note that in unsupervised domain adaptation , Lttrain and Ltval can not be computed directly due to the lack of label in the target domain . Inspired by the past works in NAS and unsupervised domain adaptation , we propose in this paper an instantiated NASDA model , which comprises of two training phases , as shown in Figure 1 . The first is the neural architecture searching phase , aiming to derive an optimal neural architecture ( α∗ ) , following the learning schema of Equation 1,2 . Inspired by Differentiable ARchiTecture Search ( DARTS ) ( Liu et al. , 2019a ) , we relax the search space to be continuous so that α can be optimized with respect to Lsval and disc ( Φ ( xs ) , Φ ( xt ) ) by gradient descent . Specifically , we enhance the feature transferability by embedding the hidden representations of the task-specific layers to a reproducing kernel Hilbert space where the mean embeddings can be explicitly matched by minimizing disc ( Φ ( xs ) , Φ ( xt ) ) . We use multi-kernel Maximum Mean Discrepancy ( MK-MMD ) ( Gretton et al. , 2007 ) to evaluate the domain discrepancy . The second training phase aims to learn a good feature generator with task-specific loss , based on the derived α∗ from the first phase . To establish this goal , we use the derived deep neural network ( Φα∗ ) as the feature generator ( G ) and devise an adversarial training process between G and a batch of classifiers C. The high-level intuition is to first diversify C in the training process , and train G to generate features such that the diversified C can have similar outputs . The training process is similar to Maximum Classifier Discrepancy framework ( MCD ) ( Saito et al. , 2018 ) except that we extend the dual-classifier in MCD to an ensembling of multiple classifiers . Experiments on standard UDA benchmarks demonstrate the effectiveness of our derived NASDA model in achieving significant improvements over state-of-the-art methods . Our contributions of this paper are highlighted as follows : • We formulate a novel dual-objective task of Neural Architecture Search for Domain Adaptation ( NASDA ) , which optimize neural architecture for unsupervised domain adaptation , concerning both source performance objective and transfer learning objective . • We propose an instantiated NASDA model that comprises two training stages , aiming to derive optimal architecture parameters α∗ and feature extractor G , respectively . We are the first to show the effectiveness of MK-MMD in NAS process specified for domain adaptation . • Extensive experiments on multiple cross-domain recognition tasks demonstrate that NASDA achieves significant improvements over traditional unsupervised domain adaptation models as well as state-of-the-art NAS-based methods . 2 RELATED WORK . Deep convolutional neural network has been dominating image recognition task . In recent years , many handcrafted architectures have been proposed , including VGG ( Simonyan & Zisserman , 2014 ) , ResNet ( He et al. , 2016 ) , Inception ( Szegedy et al. , 2015 ) , etc. , all of which verifies the importance of human expertise in network design . Our work bridges domain adaptation and the emerging field of neural architecture search ( NAS ) , a process of automating architecture engineering technique . Neural Architecture Search Neural Architecture Search has become the mainstream approach to discover efficient and powerful network structures ( Zoph & Le , 2017 ; Zoph et al. , 2018 ) . The automatically searched architectures have achieved highly competitive performance in tasks such as image classification ( Liu et al. , 2018c ; b ) , object detection ( Zoph et al. , 2018 ) , and semantic segmentation ( Chen et al. , 2018 ) . Reinforce learning based NAS methods ( Zoph & Le , 2017 ; Tan et al. , 2019 ; Tan & Le , 2019 ) are usually computational intensive , thus hampering its usage with limited computational budget . To accelerate the search procedure , many techniques has been proposed and they mainly follow four directions : ( 1 ) estimating the actual performance with lower fidelities . Such lower fidelities include shorter training times ( Zoph et al. , 2018 ; Zela et al. , 2018 ) , training on a subset of the data ( Klein et al. , 2017 ) , or on lower-resolution images . ( 2 ) estimating the performance based on the learning curve extrapolation . Domhan et al . ( 2015 ) propose to extrapolate initial learning curves and terminate those predicted to perform poorly . ( 3 ) initializing the novel architectures based on other well-trained architectures . Wei et al . ( 2016 ) introduce network morphisms to modify an architecture without changing the network objects , resulting in methods that only require a few GPU days ( Elsken et al. , 2017 ; Cai et al. , 2018a ; Jin et al. , 2019 ; Cai et al. , 2018b ) . ( 4 ) one-shot architecture search . One-shot NAS treats all architectures as different subgraphs of a supergraph and shares weights between architectures that have edges of this supergraph in common ( Saxena & Verbeek , 2016 ; Liu et al. , 2019b ; Bender , 2018 ) . DARTS ( Liu et al. , 2019a ) places a mixture of candidate operations on each edge of the one-shot model and optimizes the weights of the candidate operations with a continuous relaxation of the search space . Inspired by DARTS ( Liu et al. , 2019a ) , our model employs differentiable architecture search to derive the optimal feature extractor for unsupervised domain adaptation . Domain Adaptation Unsupervised domain adaptation ( UDA ) aims to transfer the knowledge learned from one or more labeled source domains to an unlabeled target domain . Various methods have been proposed , including discrepancy-based UDA approaches ( Long et al. , 2017 ; Tzeng et al. , 2014 ; Ghifary et al. , 2014 ; Peng & Saenko , 2018 ) , adversary-based approaches ( Liu & Tuzel , 2016 ; Tzeng et al. , 2017 ; Liu et al. , 2018a ) , and reconstruction-based approaches ( Yi et al. , 2017 ; Zhu et al. , 2017 ; Hoffman et al. , 2018 ; Kim et al. , 2017 ) . These models are typically designed to tackle single source to single target adaptation . Compared with single source adaptation , multi-source domain adaptation ( MSDA ) assumes that training data are collected from multiple sources . Originating from the theoretical analysis in ( Ben-David et al. , 2010 ; Mansour et al. , 2009 ; Crammer et al. , 2008 ) , MSDA has been applied to many practical applications ( Xu et al. , 2018 ; Duan et al. , 2012 ; Peng et al. , 2019 ) . Specifically , Ben-David et al . ( 2010 ) introduce an H∆H-divergence between the weighted combination of source domains and a target domain . These models are developed using the existing hand-crafted network architecture . This property limits the capacity and versatility of domain adaptation as the backbones to extract the features are fixed . In contrast , we tackle the UDA from a different perspective , not yet considered in the UDA literature . We propose a novel dual-objective model of NASDA , which optimize neural architecture for unsupervised domain adaptation . We are the first to show the effectiveness of MK-MMD in NAS process which is designed specifically for domain adaptation . 3 NEURAL ARCHITECTURE SEARCH FOR DOMAIN ADAPTATION . In unsupervised domain adaptation , we are given a source domain Ds = { ( xsi , ysi ) } ns i=1 of ns labeled examples and a target domainDt = { xtj } nt j=1 of nt unlabeled examples . The source domain and target domain are sampled from joint distributions P ( xs , ys ) and Q ( xt , yt ) , respectively . The goal of this paper is to leverage NAS to derive a deep networkG : x 7→ y , which is optimal for reducing the shifts in data distributions across domains , such that the target risk t ( G ) = E ( xt , yt ) ∼Q [ G ( xt ) 6= yt ] is minimized . We will start by introducing some preliminary background in Section 3.1 . We then describe how to incorporate the MK-MMD into the neural architecture searching framework in Section 3.2 . Finally , we introduce the adversarial training between our derived deep network and a batch of classifiers in Section 3.3 . An overview of our model can be seen in Algorithm 1 .

Summary: This work devises a two step process for searching optimal models for unsupervised domain adaptation. The first step involves a modification of DARTS where a discrepancy term between features for source and target domain is added to the negative-reward. The obtained feature transformer is then re-trained with an adversarial objective in order to ensure that it performs well across multiple classifiers.