Wildly Unsupervised Domain Adaptation and Its Powerful and Efficient Solution

1 INTRODUCTION . Domain adaptation ( DA ) aims to learn a discriminative classifier in the presence of a shift between training data in source domain and test data in target domain ( Ben-David et al. , 2010 ; Ganin and Lempitsky , 2015 ; Xiao and Guo , 2015 ; Zhang et al. , 2015 ; 2013 ) . Currently , DA can be divided into three categories : supervised DA ( Tzeng et al. , 2015 ) , semi-supervised DA ( Guo and Xiao , 2012 ) and unsupervised DA ( UDA ) ( Saito et al. , 2017 ) . When the number of labeled data is few in target domain , supervised DA is also known as few-shot DA ( Motiian et al. , 2017 ) . Since unlabeled data in target domain can be easily obtained , UDA exhibits the greatest potential in the real world ( Ganin and Lempitsky , 2015 ; Ganin et al. , 2016 ; Gong et al. , 2012 ; 2016 ; Long et al. , 2015 ; Saito et al. , 2017 ; 2018 ) . UDA methods train with clean labeled data in source domain ( i.e. , clean source data ) and unlabeled data in target domain ( i.e. , unlabeled target data ) to obtain classifiers for the target domain ( TD ) , which mainly consist of three orthogonal techniques : integral probability metrics ( IPM ) ( Ghifary et al. , 2017 ; Gong et al. , 2016 ; Gretton et al. , 2012 ; Lee and Raginsky , 2018 ; Long et al. , 2015 ) , adversarial training ( Ganin et al. , 2016 ; Gong et al. , 2018 ; Hoffman et al. , 2018 ; Li et al. , 2018 ; Saito et al. , 2018 ; Tzeng et al. , 2017 ) and pseudo labeling ( Saito et al. , 2017 ) . Compared to IPM- and adversarial-training-based methods , the pseudolabeling-based method ( i.e. , asymmetric tri-training domain adaptation ( ATDA ) ( Saito et al. , 2017 ) ) can construct a high-quality target-specific representation , providing a better classification performance . However , in the wild , the data volume of source domain tends to be large . To avoid the expensive labeling cost , labeled data in source domain normally come from amateur annotators or the Internet ( Lee et al. , 2018 ; Schroff et al. , 2011 ; Tommasi and Tuytelaars , 2014 ) . This brings us a new , more realistic and more challenging problem , wildy unsupervised domain adaptation ( abbreviated as WUDA , Figure 1 ) . This adaptation aims to transfer knowledge from noisy labeled data in source domain ( P̃s , i.e. , noisy source data ) to unlabeled target data ( Pxt ) . Unfortunately , existing UDA methods share an implicit assumption that there are no noisy source data . Namely , these methods focus on transferring knowledge from clean source data ( Ps ) to unlabeled target data ( Pxt ) . Therefore , these methods can not well handle the WUDA . To validate this fact , we empirically reveal the deficiency of existing UDA methods ( Figure 2 ) . To improve these methods , a straightforward solution is a two-step approach . In Figure 1 , we can first use label-noise algorithms to train a model on noisy source data , then leverage this trained model to assign pseudo labels for noisy source data . Via UDA methods , we can transfer knowledge from pseudo-labeled source data ( P̂s ) to unlabeled target data ( Pxt ) . Nonetheless , pseudo-labeled source data are still noisy , and such two-step approach may not eliminate noise effects . To circumvent the issue of two-step approach , we present a robust one-step approach called Butterfly . In high level , Butterfly directly transfers knowledge from P̃s to Pxt , and uses the transferred knowledge to construct target-specific representations . In low level , Butterfly maintains four networks dividing two branches ( Figure 3 ) : Two networks in Branch-I are jointly trained on noisy source data and pseudo-labeled target data ( data in mixture domain ( MD ) ) ; while two networks in Branch-II are trained on pseudo-labeled target data . The reason why Butterfly can be robust takes root in the dual-checking principle ( DCP ) : Butterfly checks high-correctness data out , from not only the data in MD but also the pseudo-labeled target data . After crosspropagating these high-correctness data , Butterfly can obtain high-quality domain-invariant representations ( DIR ) and target-specific representations ( TSR ) simultaneously in an iterative manner . If we only check data in MD ( i.e. , single checking ) , the error existed in pseudo-labeled target data will accumulate , leading to low-quality DIR and TSR . We conduct experiments on simulated WUDA tasks , including 4 MNIST-to-SYND tasks , 4 SYND-to-MNIST tasks and 24 human-sentiment tasks . Besides , we conduct experiments on 3 real-world WUDA tasks . Empirical results demonstrate that Butterfly can robustly transfer knowledge from noisy source data to unlabeled target data . Meanwhile , Butterfly performs much better than existing UDA methods when source domain ( SD ) suffers the extreme ( e.g. , 45 % ) noise . 2 WILDLY UNSUPERVISED DOMAIN ADAPTATION . In this section , we first define a new , more realistic and more challenging setting called wildly unsupervised domain adaptation ( WUDA ) , and explain the nature of WUDA . Then , we empirically show that representative UDA methods can not handle WUDA well , which motivates us to propose Butterfly ( see Section 3 ) . Definition 1 ( Wildly Unsupervised Domain Adaptation ) . Let Xt be a multivariate random variable defined on the space X with respective a probability density pxt , ( Xs , Ys ) be a multivariate random variable defined on the space X ×C with respective a probability density p̃s , where C = { 1 , . . . , K } . Let pxs be the marginal probability density of p̃s . Given i.i.d . data D̃s = { ( xsi , ỹsi ) } nsi=1 and Dt = { xti } nt i=1 drawn from P̃s and Pxt , in wildly unsupervised domain adaptation , we aim to train with D̃s and Dt to accurately annotate data drawn from Pxt , where pxs 6= pxt . Remark 1 . In Definition 1 , D̃s is noisy source data , Dt is unlabeled target data , and P̃s and Pxt are two probability measures corresponding to densities p̃s ( xs , ỹs ) and pxt ( xt ) . Nature of WUDA . Specifically , there are five distributions involved in WUDA setting : 1 ) a marginal distribution on source data , i.e. , pxs in Definition 1 ; 2 ) a marginal distribution on target data , i.e. , pxt in Definition 1 ; 3 ) an incorrect conditional distribution of label given xs , q ( ys|xs ) ; 4 ) a correct conditional distribution of label given xs , p ( ys|xs ) and 5 ) a correct conditional distribution of label given xt , p ( yt|xt ) . Based on Definition 1 and Appendix A.2 , noisy source data D̃s are drawn from p̃s ( xs , ys ) = pxs ( xs ) ( ( 1− ρ ) p ( ys|xs ) + ρq ( ys|xs ) ) , where ρ is the noise rate in source data . Namely , source data D̃s are mixture of correct source data from pxs ( xs ) p ( ys|xs ) and incorrect data from pxs ( xs ) q ( ys|xs ) . Target data Dt are drawn from pxt . In WUDA setting , we aim to train a classifier with D̃s and Dt . This classifier is expected to accurately annotate data from pxt , i.e. , to accurately simulate distribution 5 ) . WUDA ruins representative UDA methods . We take a simple example to illustrate why WUDA ruins representative UDA methods . We corrupt source data using symmetry flipping ( Patrini et al. , 2017 ) and pair flipping ( Han et al. , 2018 ) ( Appendix B ) . Namely , the corrupted source data ( D̃s in Definition 1 ) are drawn from P̃s whose density is p̃s ( xs , ys ) = pxs ( xs ) ( ( 1− ρ ) p ( ys|xs ) + ρq ( ys|xs ) ) . We draw the target data Dt from Pxt whose density is pxt . To instantiate source and target data , we leverage MNIST and SYND ( Figure 6 in Appendix C ) , respectively . Thus , we first construct two WUDA tasks with symmetry-flip noise : corrupted SYND→MNIST ( S→M ) and corrupted MNIST→SYND ( M→S ) . In Figure 2 ( a ) , we report target-domain accuracy of representative UDA methods , when the noise rate ρ of SD changes from 5 % to 70 % . It is clear that target-domain accuracy of these representative UDA methods drops quickly when ρ increases . This means that WUDA ruins representative UDA methods . Then , we construct another two WUDA tasks with pair-flip noise . In Figure 2 ( b ) , we report target-domain accuracy , when the noise rate ρ of SD changes from 5 % to 45 % . Again , WUDA still ruins representative UDA methods . Note that pair-flip noise is much harder than symmetry-flip noise , and its noise rate can not be over 50 % in practice ( Han et al. , 2018 ) . However , the proposed Butterfly network ( abbreviated as B-Net , Figure 3 ) performs robustly when ρ increases ( blue lines in Figure 2 ) . In following sections , we will introduce Butterfly framework , and explain why Butterfly achieves better target-domain accuracy consistently . 3 BUTTERFLY : TOWARDS ROBUST ONE-STEP APPROACH . To realize a robust WUDA approach , we propose a Butterfly framework ( a one-step approach , Algorithm 1 ) , which trains four networks dividing into two branches ( Figure 3 ) . By using DCP , Branch-I checks which data is correct in MD ; while Branch-II checks which pseudo-labeled target data is correct . To ensure these checked data highly-correct , we apply the small-loss trick based on memorization effects of deep learning ( Arpit et al. , 2017 ) . After cross-propagating these checked data ( Bengio , 2014 ) , Butterfly can obtain high-quality DIR and TSR simultaneously in an iterative manner . Loss function in Butterfly . Four networks trained by Butterfly share the same loss function but with different inputs . L ( θ , u ; F , D ) = 1∑n i=1 ui n∑ i=1 ui ` ( F ( xi ) , y̌i ) , ( 1 ) where n is the batch size , and F represents a network ( e.g. , F1 , F2 , Ft1 and Ft2 ) . D = { ( xi , y̌i ) } ni=1 is a mini-batch for training a network , where { xi , y̌i } ni=1 could be data in MD or TD ( Figure 3 ) , and θ represents parameters of F and u = [ u1 , ... , un ] T is an n-by-1 vector whose elements equal 0 or 1 . ` ( · , · ) is the cross-entropy loss . For two networks in Branch-I , following Saito et al . ( 2017 ) , we also add a regularizer |θTf11θf21| in their loss functions , where θf11 and θf21 are weights of the first fully-connect layer of F1 and F2 . With this regularizer , F1 and F2 will learn from different features . Nature of the loss L. In loss function L , we have n instances : ( xi , y̌i ) , where i = 1 , 2 , . . . , n. For the ith instance , we will compute its cross-entropy loss ( i.e. , ` ( F ( xi ) , y̌i ) ) , and we will denote this instance as “ selected ” if ui = 1 . Thus , the nature of L is actually the average value of cross-entropy loss of these “ selected ” instances . Note that , we need to set a constrain to prevent ∑n i=1 ui = 0 in L , which means that we need to select at least one instance to compute L. Training procedure of Butterfly . For two networks in each branch , they will first check high-correctness data out and then cross update their parameters using these data . Based on loss function defined in Eq . ( 1 ) , entire training procedures of Butterfly are shown in Algorithm 1 . First , we initialize training data for two branches ( D̃ for Branch-I and D̃lt for Branch-II ) , four networks ( F1 , F2 , Ft1 and Ft2 ) and the number of pseudo labels ( line 2 ) . In the first epoch ( T = 1 ) , following Saito et al . ( 2017 ) , D̃lt is the same with D̃s ( i.e. , we use noisy source data as pseudo-labeled target data ) since we can not annotate pseudo labels for target data when T = 1 . After mini-batch Ď is fetched from D̃ ( line 4 ) , F1 and F2 check high-correctness data out and update their parameters ( lines 5 ) using Algorithm 2 . Using similar procedures , Ft1 and Ft2 also update their parameters using Algorithm 2 ( lines 6-7 ) . In each epoch , after Nmax mini-batch updating , we randomly select nlt unlabeled target data and assign them pseudo labels using F1 and F2 ( lines 8 ) . Following Saito et al . ( 2017 ) , the Labeling function in Algorithm 1 ( line 8 ) assigns pseudo labels for unlabeled target data , when predictions of F1 and F2 agree and at least one of them is confident about their predictions ( probability above 0.9 or 0.95 ) . Using this function , we can obtain the pseudo-labeled target data D̃lt for training Branch-II in the next epoch . Then , we merge D̃ l t and D̃s to be D̃ for training Branch-I in the next epoch ( line 9 ) . Finally , we update nlt , R ( T ) and Rt ( T ) in lines 10-11 according to Saito et al . ( 2017 ) and Han et al . ( 2018 ) . Note that R ( T ) is a piecewise-defined linear function . Namely , when T ≥ Tk , R ( T ) = 1− τ ; when T ≤ Tk , R ( T ) = 1− T/Tk × τ . In Algorithm 1 , we use τ to represent the noise rate ( i.e. , the ratio of data with incorrect labels ) in MD and use τt to represent the noise rate in TD . However , in WUDA , we can not obtain the ground-truth τ and τt . Thus , we regard τ and τt as hyper-parameters . Note that τ and τt are robustly set to 0.4 and 0.05 in experiments . Checking process in Butterfly . In Algorithm 2 , we first obtain four inputs : 1 ) networks F1 and F2 , and 2 ) a mini-batch D , and 3 ) learning rate η and 4 ) remember rate α ( line 1 ) . Then , we will obtain the best u1 by solving a minimization problem ( line 2 ) . L represents the loss function defined in Eq . ( 1 ) . θ1 represents the parameters of the network F1 . Similarly we will obtain the best u2 ( line 3 ) . θ2 represents the parameters of the network F2 . Next , θ1 and θ2 are updated using gradient descent , where the gradients are computed using a given optimizer ( lines 4-5 ) . Finally , we substitute the updated θ1 into F1 and the updated θ2 into F2 and output F1 and F2 ( line 6 ) . Solution to minimization problems in Algorithm 2 . In line 2 or 3 in Algorithm 2 , we need to solve a minimization problem : minu′:1u′ > α|D| L ( θ , u′ ; F , D ) and return the best u′ as u ( u1 in line 2 and u2 in line 3 ) . In this paragraph , we will show how to quickly solve this problem using a sorting algorithm . Recall the nature of the loss L , we know L is the average value of cross-entropy losses of “ selected ” instances , and 1u′ is the number of these “ selected ” instances . Thus this minimization problem is equivalent to “ given a fixed F ( F1 or F2 ) and n instances in D , how to select at least k instances such that L is minimized ” , where k = dα|D|e . To solve this problem , we first use a sorting algorithm ( top_k function in TensorFlow ) to sort these n instances according to their cross-entropy losses ` ( F1 ( xi ) , y̌i ) . Then , we select k instance with the smallest cross-entropy losses . Finally , let ui of these k instances be 1 and ui of the other instances be 0 , and we can get the best u = [ u1 , . . . , un ] . The average value of cross-entropy losses of these k instances is the minimized value of L ( θ , u′ ; F , D ) under the constrain 1u′ > α|D| .

Summary: This paper proposes a new problem setting in the domain adaptation field. Since it is impossible to obtain perfectly clean labeled source data in the real world, existing UDA methods cannot well handle real-world data. However, in wildly unsupervised domain adaptation (WUDA), we do not need a perfectly clean source data, which means that WUDA problem is a more general and realistic problem than existing ones.