Understanding the Limits of Unsupervised Domain Adaptation via Data Poisoning

1 Introduction . The problem of domain adaptation ( DA ) arises when the training and the test data distributions are different , violating the common assumption of supervised learning . In this paper , we focus on unsupervised DA ( UDA ) , which is a special case of DA when no labeled information from the target domain is available . This setting is useful for applications where obtaining large-scale well-curated datasets is both time-consuming and costly . The seminal works [ 1 , 2 ] proved an upper bound on a classifier ’ s target domain error in the UDA setting leading to several algorithms for learning in this setting . Many of these algorithms rely on learning a domain invariant representation by minimizing the error on the source domain and a divergence measure between the marginal feature distributions of the source and target domains . Popular divergence measures include total variation distance , JensenShannon divergence [ 9 , 32 , 37 ] , Wasserstein distance [ 29 , 15 , 7 ] , and maximum mean discrepancy [ 19 , 17 , 20 ] . The success of these algorithms is argued in terms of minimization of the upper bound proposed in [ 1 ] along with an improvement in the target domain accuracy on benchmark UDA tasks . Despite this success on benchmark datasets , some works [ 9 , 16 , 36 , 33 ] have presented evidence of the failure of these methods in different scenarios . Recent works have explained this apparent failure of UDA methods by proposing new upper bounds [ 36 , 6 , 14 , 34 ] on the target domain error while 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . others have demonstrated the cause of failure using experiments showing that learning a domain invariant representation that minimizes the source domain error can cause an increase in the error of the ideal joint hypothesis [ 16 ] . To provably explain this apparent failure of learning in the UDA setting we propose a lower bound on the target domain error . Our lower bound provides a necessary condition for successful learning in the UDA setting , complementing the existing upper bound of [ 1 ] and is dependent on the difference between the labeling functions of source and target domain data induced by the representation map . For cases where the induced labeling functions match on the source and the target domain data ( i.e. , favorable case ) , the success of UDA is explained using the upper bounds proposed by previous works [ 1 , 36 , 14 ] . For a representation that aligns the source and the target domain data and minimizes the error on the source but induces labeling functions that don ’ t agree on the source and the target domain data ( i.e. , unfavorable case ) , our lower bound explains the failure of UDA . Our analysis brings to light yet another case ( i.e. , ambiguous case ) of data distributions where success and failure of UDA are equally likely . This happens due to the lack of label information from the target domain . This case opens doors for adversarial attacks against UDA methods since a small amount of misinformation about the target domain labels can lead the UDA methods into producing a representation similar to the unfavorable case , incurring a significant increase in the target domain error . Motivated from this analysis of UDA methods under different data distributions , we evaluate the extent to which the performance of current UDA methods can suffer in presence of a small amount of adversarially crafted data . For this purpose , we propose novel data poisoning attacks , using mislabeled and clean-label points . We evaluate the effect of our poisoning attacks on popular UDA methods using benchmark datasets , where they were previously shown to be very effective . We find that our poisoning attacks cause UDA methods to either align incorrect classes from the two domains or prevent correct classes from being very close in the representation space . Both of these lead to the failure of UDA methods at reducing target domain error . With just 10 % poison data in the source domain , target domain accuracy for current UDA methods is significantly reduced , dropping to almost 0 % in some cases . This dramatic failure of UDA methods demonstrates their limits and suggests that the future UDA methods must be evaluated in adversarial settings along with evaluation on benchmark datasets to truly gauge their effectiveness at learning under the UDA setting . Our main contributions are summarized as follows : • We prove a lower bound on the target domain error that provides a necessary condition for successful learning in the UDA setting . Our bound shows the failure of learning a domain invariant representation while minimizing the source domain error at guaranteeing target generalization . • We present example data distributions where UDA succeeds , fails , and where success and failure are equally likely . This sensitivity of UDA methods to the data distribution brings to light a new vulnerability of UDA methods to adversarial attacks such as data poisoning . • To concretely understand the extent of this vulnerability of UDA methods , we propose novel data poisoning attacks using clean-label and mislabeled data . Our results show a dramatic failure of current UDA methods at target generalization in presence of poisoned data . Thus , our poisoning attacks can provide better insights into the robustness of UDA methods than those obtained from performance evaluation on benchmark datasets . 2 Background and related work . Analysis of unsupervised domain adaptation : Several previous works have studied the problem of UDA and have provided conditions under which UDA is possible [ 2 , 1 , 21 , 22 , 8 , 3 , 34 ] . [ 1 , 2 ] proposed an upper bound on the target domain error which has inspired many UDA algorithms . Recent works [ 36 , 6 , 14 ] have improved the upper bounds on target domain error and have proposed a lower bound dependent on the labeling functions in the input space . Using this lower bound , the failure of learning in the UDA setting was explained in the case when marginal label distributions are different between the two domains . Our lower bound , on the other hand , is dependent on the labeling functions for the source and target domains , induced by the representation . This suggests that if a representation induces labeling functions that disagree on source and target domains then UDA provably fails even if the representation is domain invariant and minimizes error on the source domain . Thus our lower bound directly explains the observations of failure of UDA in many previous works [ 9 , 16 , 33 , 36 , 34 ] . A detailed comparison of our work with other works analyzing the failure of learning in the UDA setting is present in Appendix G. Algorithms for UDA : Many algorithms [ 9 , 32 , 18 , 37 , 30 , 10 ] for UDA learn a domain invariant representation while minimizing error on the source domain . A popular adversarial approach to domain adaptation is DANN [ 9 ] which uses a discriminator to distinguish points from source and target domains based on their representations . Another popular method CDAN [ 18 ] , uses classifier output along with representations to identify the domains of the points . IW-DAN and IW-CDAN [ 6 ] were recently proposed as extensions of the original DANN and CDAN with an importance weighting scheme to minimize the mismatch between the labeling distributions of the two domains . A different approach MCD [ 27 ] , makes use of two task-specific classifiers as discriminators to align the two domains . This method adversarially trains the representation to minimize the disagreement between the two classifiers on the target domain data ( classifier discrepancy ) while training the classifiers to maximize this discrepancy . Another recent approach , SSL [ 35 ] uses self-supervised learning tasks ( e.g . rotation angle prediction ) to better align the two domains . In this work , we study the effect of poisoning on these methods as they have been shown to be effective at various UDA tasks . Data poisoning : Data poisoning [ 4 , 25 , 12 , 5 , 13 , 31 , 24 ] is a training time attack where the attacker has access to the data which will be used by the victim for training . Most works [ 38 , 26 , 28 , 23 , 11 ] have considered data poisoning in a fully supervised setting where train and test sets are drawn from the same underlying data distribution ( single domain setting ) . These works either target the classification of a single test point by adding a large number of poisoned data or require modifying all points in a class to affect the model ’ s performance on that class after retraining . Our work studies the effect of poisoning on the entire target domain in the UDA setting using popular UDA algorithms for training . The success of our poisoning attacks at significantly reducing the target domain accuracy with a small amount of poisoned data shows the ease of poisoning in the UDA setting . In contrast , poisoning leads to a small decrease in the overall test accuracy in the single domain setting , especially when using state-of-the-art classifiers such as deep neural networks . 3 When does learning fail in the unsupervised domain adaption setting ? . Notations and settings : Let X be the data domain and D be the distribution over X with the corresponding pdf p ( x ) . We assume there is a deterministic labeling function f : X → [ 0 , 1 ] for the given binary classification task . The f ( x ) can be interpreted as Pr [ y = 1|x ] . Let g : X → Z denote the representation map that maps an input instance x to its features where Z is called feature or representation space . Let h : Z → [ 0 , 1 ] be a hypothesis for binary classification on the representation space . Note that the representation map g induces a distribution over Z denoted by PrD̃ [ B ] : = PrD [ g −1 ( B ) ] and the corresponding density function p̃ ( z ) on Z [ 2 ] . The g also induces the labeling function f̃ ( z ) : = ED [ f ( x ) |g ( x ) = z ] , ( 1 ) for any B such that g−1 ( B ) is D-measurable . The misclassification error e ( h ) w.r.t . the induced labeling function is e ( h ) = Ez∼D̃ [ |f̃ ( z ) − h ( z ) | ] where D̃ is the induced distribution over Z . Similarly , we define e ( f̃ , f̃ ′ ) = Ez∼D̃ [ |f̃ ( z ) − f̃ ′ ( z ) | ] and e ( h , h′ ) = Ez∼D̃ [ |h ( z ) − h′ ( z ) | ] . The distributions p̃ and the labeling functions f̃ for the source and the target domains will be written as p̃S , p̃T , f̃S and f̃T , respectively . The total variation distance is D1 ( p̃ , p̃′ ) = ∫ Z |p̃ ( z ) − p̃ ′ ( z ) |dz . 3.1 Lower bound on the target domain error . Most adversarial DA methods learn a domain invariant representation g : X → Z by minimizing error on the source domain and penalizing the mismatch between the marginal source and target distributions since the conditional distribution p̃T ( z|y ) for target domain is unavailable in the UDA setting . Some works [ 33 , 36 , 16 ] have shown this to be insufficient at guaranteeing target generalization . Recent works have proposed a new upper bound [ 36 ] or argued failure in terms of the upper bound in [ 1 , 2 ] ( eT ( h ) ≤ min { eT ( f̃S , f̃T ) , eS ( f̃S , f̃T ) } + eS ( h ) +D1 ( p̃S , p̃T ) ) being large . However , a large upper bound does not guarantee failure . Thus , we prove a simple lower bound on the target domain error which shows the necessary condition for the success of learning in the UDA setting and also explains the failure of current UDA methods at guaranteeing target generalization . Theorem 1 . Let H be the hypothesis class and G be the class representation maps . Then , for all h ∈ H and g ∈ G , eT ( h ) ≥ max { eS ( f̃S , f̃T ) , eT ( f̃S , f̃T ) } − eS ( h ) −D1 ( p̃S , p̃T ) . ( 2 ) The proof is in Appendix A . Even though our bound depends on the total variation distance which is a strict measure of distance and is difficult to estimate , the bound still explains the limitations of UDA . Extending the bound to different divergence metrics is left as future work . The following corollary is immediate from Theorem 1 . Corollary 1.1 . For all h ∈ H , g ∈ G , eT ( h ) ≥ max { eS ( f̃S , f̃T ) , eT ( f̃S , f̃T ) } − eS ( h ) − √ 0.5DKL ( p̃S ||p̃T ) . This is due to the Pinsker ’ s inequality : D1 ( p , p′ ) ≤ √ 0.5DKL ( p||p′ ) . The interpretation of Eq . 2 is as follows . Since fT is not observable , the goal of UDA is to minimize the observable source classification error eS ( h ) and the domain mismatch D1 ( p̃S , p̃T ) ( or other metrics such as dH∆H ( p̃s , p̃T ) ) : min g , h es ( h ) +D1 ( p̃S , p̃T ) . ( 3 ) This leads to maximization of the second and the third term in the RHS of Eq . 2 . With overparameterized models and large datasets , minimizing the empirical estimates of the quantities in Eq . 3 can drive eS ( h ) ' 0 and D1 ( p̃S , p̃T ) ' 0 . Consequently , eT ( h ) ≥ max { eS ( f̃S , f̃T ) , eT ( f̃S , f̃T ) } . If f̃S and f̃T disagree on the source and target domains in the representation space , target domain error eT ( h ) will be provably large . Corollary 1.2 . For all h ∈ H and g ∈ G , |eT ( h ) −eS ( f̃S , f̃T ) | ≤ eS ( h ) +D1 ( p̃S , p̃T ) , and |eT ( h ) −eT ( f̃S , f̃T ) | ≤ eS ( h ) +D1 ( p̃S , p̃T ) . This is obtained by combining the upper and the lower bounds ( Appendix A ) . Thus a UDA method can only guarantee the target error eT ( h ) to be close to the labeling function mismatch e ( f̃S , f̃T ) . Whether e ( f̃S , f̃T ) becomes larger or smaller after solving Eq . 3 is data/model dependent . For concreteness , we analyze the sensitivity of the performance of UDA methods to different data distributions using the following illustrative examples .

Summary: The paper addresses the vulnerability of domain adaptation to adversarial attacks. First, a lower bound on the target risk of a given classifier is derived, which provides necessary conditions for the success of domain adaptation methods. Then, 3 examples on 2D data are given and analyzed to show the potential failure of DA methods aiming at aligning both domains while remaining good for classification on the source one. The last example describes an ambiguous situation for which success and failure of adaptation are equally likely, from which the authors have the inspiration that adding a few poison data can easily favor the bad solution. They concretize this idea by proposing 3 data poisoning strategies and they show their ability to deteriorate the performance of previously proposed DA methods.