Generalized Domain Adaptation with Covariate and Label Shift CO-ALignment

1 INTRODUCTION . The success of deep learning models is highly dependent on the assumption that the training and testing data are i.i.d and sampled from the same distribution . In reality , they are typically collected from different but related domains , leading to a phenomenon known as domain shift ( QuioneroCandela et al. , 2009 ) . To bridge the domain gap , Unsupervised Domain Adaptation ( UDA ) transfers the knowledge learned from a labeled source domain to an unlabeled target domain by statistical distribution alignment ( Long et al. , 2015 ; Tzeng et al. , 2014a ) or adversarial alignment ( Tzeng et al. , 2017 ; Ganin & Lempitsky , 2015a ; Saito et al. , 2018 ) . Though recent UDA work has made great progress , it has mostly failed to address the case of Shift in Label Distribution , i.e. , a changing prior over the labels . Denote the input data as x and output labels as y , and let the source and target domain be characterized by probability distributions p and q , respectively . The majority of methods assume that the conditional label distribution is invariant ( p ( y|x ) = q ( y|x ) ) , and only the Shift in Feature Distribution ( SFD ) ( p ( x ) 6= q ( x ) ) needs to be tackled , neglecting any potential Shift in Label Distribution ( SLD ) ( p ( y ) 6= q ( y ) ) . However , recent theoretical work ( Zhao et al. , 2019a ) has demonstrated that tackling SLD is crucial to solving the domain adaptation problem . SLD also occurs in real applications ; for example , an autonomous driving system should be able to handle changing frequencies of pedestrians and cars when adapting from a rural to a downtown area . In this paper , we propose Generalized Domain Adaptation ( GDA ) , a more challenging but practical domain adaptation setting where the conditional Shift in Feature Distribution and Shift in Label Distribution are required to be minimized simultaneously . Specifically , in addition to the Covariate Shift assumption ( p ( x ) 6= q ( x ) , p ( y|x ) = q ( y|x ) ) , we further assume p ( x|y ) 6= q ( x|y ) and p ( y ) 6= q ( y ) . The main challenges of GDA are : ( 1 ) SLD exists between the source and target domain , which hampers the effectiveness of mainstream domain adaptation methods that only minimize SFD , ( 2 ) aligning the conditional feature distributions ( p ( x|y ) , q ( x|y ) ) is difficult in the presence of SLD , and ( 3 ) when the data in one or both of the domains are unequally distributed across different categories , it is difficult to train an unbiased classifier . An overview of GDA is shown in Figure 1 . Mainstream unsupervised domain adaptation aligns the marginal feature distributions of two domains by methods that include minimizing the Maximum Mean Discrepancy ( Long et al. , 2015 ; Tzeng et al. , 2014a ) , aligning high-order moments ( Zellinger et al. , 2017 ; Peng et al. , 2019a ) , or adversarial training ( Tzeng et al. , 2017 ; Ganin & Lempitsky , 2015a ) . These methods have achieved state-of-the-art performance on several domain adaptation benchmarks ( Saenko et al. , 2010 ; Venkateswara et al. , 2017 ; Peng et al. , 2017 ) which have significant SFD but limited SLD . These models are limited when applied to the GDA task as they only align the feature distribution , ignoring the real issue of SLD which exists in many real-world applications . Another line of works ( Lipton et al. , 2018a ; Azizzadenesheli et al. , 2019a ) assume that only SLD exists ( p ( y ) 6= q ( y ) ) between two domains and the conditional feature distribution is invariant ( p ( x|y ) = q ( x|y ) ) . These methods have achieved good performance when the data in both domains are sampled from the same data distributions but under different label distributions . However , these models can not handle the GDA task as the label distribution is not well aligned . Taking steps towards minimizing the SFD and SLD simultaneously , several works ( Zhang et al. , 2013 ; Gong et al. , 2016 ; Wu et al. , 2019 ) provide a theoretical analysis with additional constraints on distributions p and q , such as a linearity assumption between p ( x|y ) and q ( x|y ) ( Zhang et al. , 2013 ) , which does not necessarily hold in many real applications . In addition , no practical algorithm which can solve real-world cross-domain problems has been proposed by these papers . We postulate that it is essential to align the conditional feature distributions as well as the label distributions to tackle the GDA task . In this work , we address GDA with SFD and SLD CO-ALignment ( COAL ) . Specifically , our approach diminishes the negative effect of SFD and SLD with prototypebased conditional distribution alignment and label distribution estimation , respectively . First , to reduce the SFD in the context of SLD , it is essential to align the conditional rather than marginal feature distributions , to avoid the negative transfer effects which are caused by matching marginal feature distributions , according to a theoretical proof from Zhao et al . ( 2019a ) ( illustrated in Figure 2 ) . To this end , we propose a prototype-based method to align the conditional feature distributions of the two domains . The source prototypes are computed by learning a similarity-based classifier and the target prototypes are estimated by a minimax entropy algorithm ( Saito et al. , 2019b ) . Second , we align the label distributions in the context of SFD by estimating the target label distribution . We incorporate feature distribution and label distribution alignment into an end-to-end deep learning framework , as illustrated in Figure 1 . Comprehensive experiments on standard crossdomain recognition benchmarks demonstrate that COAL achieves significant improvements over the state-of-the-art methods on the task of GDA . The main contributions of this paper are highlighted as follows : ( 1 ) to the best of our knowledge , we provide the first practical solution and set of benchmarks for joint SFD and SLD in deep learning ; ( 2 ) we develop an end-to-end CO-ALignment ( COAL ) framework which leverages conditional prototype-based alignment to solve the problem caused by SFD and label distribution estimation to solve the problem caused by the SLD ; and ( 3 ) we deliver extensive experiments and analysis to show the effectiveness of COAL , and importantly , we empirically demonstrate that state-of-the-art methods fail to align conditional feature distribution in the presence of label distribution . 2 RELATED WORK . Unsupervised Domain Adaptation Domain adaptation aims to transfer the knowledge learned from one or more source domains to a target domain . Recently , many unsupervised domain adaptation methods have been proposed . These methods can be taxonomically divided into three categories ( Wang & Deng , 2018 ) . The first category is the discrepancy-based approach , which leverages different measures to align the marginal feature distributions between source and target domains . Gradient for labeled and pseudo-labeled samples Commonly used measures include Maximum Mean Discrepancy ( MMD ) ( Long et al. , 2017 ; Tzeng et al. , 2014b ) , H-divergence ( Ben-David et al. , 2010 ) , Kullback-Lerbler ( KL ) divergence ( Zhuang et al. , 2015 ) , and Wasserstein distance ( Lee & Raginsky , 2017 ; Shen et al. , 2017 ) . The second category is the adversarial-based approach ( Tzeng et al. , 2017 ; Liu & Tuzel , 2016 ; Peng et al. , 2019b ) which uses a domain discriminator to encourage domain confusion via an adversarial objective . The third category is the reconstruction-based approach . Data are reconstructed in the new domain by an encoder-decoder ( Bousmalis et al. , 2016 ; Ghifary et al. , 2016 ) or a GAN discriminator , such as dual-GAN ( Yi et al. , 2017 ) , cycle-GAN ( Zhu et al. , 2017 ) , disco-GAN ( Kim et al. , 2017 ) , and CyCADA ( Hoffman et al. , 2018 ) . However , these methods mainly consider aligning the marginal distributions to decrease feature shift , neglecting label shift . To the best of our knowledge , we are the first the propose an end-to-end deep model to tackle both feature and label shift between the source and target domains . Learning with Label Shift Despite its wide applicability , learning under label shift remains underexplored . Existing works tackle this challenge by importance reweighting or target distribution estimation . Specifically , Zhang et al . ( 2013 ) exploit importance reweighting to enhance knowledge transfer under label shift . Recently , Lipton et al . ( 2018b ) introduce a test distribution estimator to detect and correct for label shift . Based on it , Azizzadenesheli et al . ( 2019b ) propose a regularized algorithm to better estimate the target label distribution . These methods assume that the source and target domains share the same generative distributions and only differ in the marginal label distribution . In this work , we explore transfer learning between domains under domain and label shifts . In this direction , Zhao et al . ( 2019b ) introduce a theoretical analysis to show that only learning domain-invariant features is not sufficient to solve domain adaptation task since the label priors are not aligned . In related work , Wu et al . ( 2019 ) propose asymmetrically-relaxed distribution alignment to overcomes the limitations of standard domain adaptation algorithms which aims to extract domain-invariant representations . Panareda Busto & Gall ( 2017 ) propose open set domain adaptation where the categories in the training domain and testing domain are not fully overlapped . Cao et al . ( 2018b ) introduce partial domain adaptation where the categories in the target domain are a subset of those in the source domain . Domain adaptation with self-training Self-training methods utilize pseudo-labels to compensate for the lack of categorical information in the target domain . The intuition is to assign pseudo-labels to unlabeled samples based on the predictions of one or more classifiers . Long et al . ( 2013 ) jointly align the marginal and conditional distributions using an MMD loss with pseudo-label refinement . Saito et al . ( 2017 ) leverage an asymmetric tri-training strategy to assign pseudo-labels to the unlabeled target domain . Xie et al . ( 2018 ) propose to assign pseudo-labels to all target samples and use them to achieve semantic alignment across domains . Zhang et al . ( 2018 ) progressively enlarge the training data with pseudo-labeled target samples assigned by the classifier from the previous training epoch and retrain the model with the enlarged training set . Recently , Chen et al . ( 2019a ) propose to progressively label the target samples and align the prototypes of source domain and target domain to achieve domain alignment . However , to the best of our knowledge , self-training has not been applied for DA with label shift . 3 CO-ALIGNMENT OF FEATURE AND LABEL DISTRIBUTION . In domain adaptation with shift in the feature and label distribution , we are given a source domain DS = { ( xsi , ysi ) Ns i=1 } with Ns labeled examples , and a target domain DT = { ( xti ) Nt i=1 } with Nt unlabeled examples . We assume that p ( x|y ) 6= q ( x|y ) and p ( y ) 6= q ( y ) . We aim to construct a end-to-end deep neural network which is able to transfer the knowledge learned from DS to DT , and train a classifier y = θ ( x ) which can minimize task risk T ( θ ) = Pr ( x , y ) ∼q [ θ ( x ) 6= y ] . Previous works either focus on aligning the marginal feature distributions ( Long et al. , 2015 ; Tzeng et al. , 2017 ) or aligning the label distributions ( Lipton et al. , 2018b ) . These approaches are not able to fully tackle the domain adaptation with shift in the feature and label distribution as they only align one of the two marginal distributions . In this work , we tackle GDA with prototype-based conditional alignment and label distribution estimation .

Summary: In this work, the authors proposed a method to address the covariate shift and label shift problems simultaneously. In detail, the prototype-based conditional alignment and self-training based label distribution estimation is utilized. Empirically evaluation is conducted on three datasets to show the superiority of the proposed method. However, the work suffers the following weaknesses: