Understanding the Limits of Unsupervised Domain Adaptation via Data Poisoning

Abstract
Unsupervised domain adaptation (UDA) enables cross-domain learning without target domain labels by transferring knowledge from a labeled source domain whose distribution differs from that of the target. However, UDA is not always successful and several accounts of ‘negative transfer’ have been reported in the literature. In this work, we prove a simple lower bound on the target domain error that complements the existing upper bound. Our bound shows the insufficiency of minimizing source domain error and marginal distribution mismatch for a guaranteed reduction in the target domain error, due to the possible increase of induced labeling function mismatch. This insufficiency is further illustrated through simple distributions for which the same UDA approach succeeds, fails, and may succeed or fail with an equal chance. Motivated from this, we propose novel data poisoning attacks to fool UDA methods into learning representations that produce large target domain errors. We evaluate the effect of these attacks on popular UDA methods using benchmark datasets where they have been previously shown to be successful. Our results show that poisoning can significantly decrease the target domain accuracy, dropping it to almost 0% in some cases, with the addition of only 10% poisoned data in the source domain. The failure of these UDA methods demonstrates their limitations at guaranteeing cross-domain generalization consistent with our lower bound. Thus, evaluating UDA methods in adversarial settings such as data poisoning provides a better sense of their robustness to data distributions unfavorable for UDA.

1 Introduction
The problem of domain adaptation (DA) arises when the training and the test data distributions are different, violating the common assumption of supervised learning. In this paper, we focus on unsupervised DA (UDA), which is a special case of DA when no labeled information from the target domain is available. This setting is useful for applications where obtaining large-scale well-curated datasets is both time-consuming and costly. The seminal works [1, 2] proved an upper bound on a classifier’s target domain error in the UDA setting leading to several algorithms for learning in this setting. Many of these algorithms rely on learning a domain invariant representation by minimizing the error on the source domain and a divergence measure between the marginal feature distributions of the source and target domains. Popular divergence measures include total variation distance, JensenShannon divergence [9, 32, 37], Wasserstein distance [29, 15, 7], and maximum mean discrepancy [19, 17, 20]. The success of these algorithms is argued in terms of minimization of the upper bound proposed in [1] along with an improvement in the target domain accuracy on benchmark UDA tasks.
Despite this success on benchmark datasets, some works [9, 16, 36, 33] have presented evidence of the failure of these methods in different scenarios. Recent works have explained this apparent failure of UDA methods by proposing new upper bounds [36, 6, 14, 34] on the target domain error while
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
others have demonstrated the cause of failure using experiments showing that learning a domain invariant representation that minimizes the source domain error can cause an increase in the error of the ideal joint hypothesis [16]. To provably explain this apparent failure of learning in the UDA setting we propose a lower bound on the target domain error. Our lower bound provides a necessary condition for successful learning in the UDA setting, complementing the existing upper bound of [1] and is dependent on the difference between the labeling functions of source and target domain data induced by the representation map. For cases where the induced labeling functions match on the source and the target domain data (i.e., favorable case), the success of UDA is explained using the upper bounds proposed by previous works [1, 36, 14]. For a representation that aligns the source and the target domain data and minimizes the error on the source but induces labeling functions that don’t agree on the source and the target domain data (i.e., unfavorable case), our lower bound explains the failure of UDA. Our analysis brings to light yet another case (i.e., ambiguous case) of data distributions where success and failure of UDA are equally likely. This happens due to the lack of label information from the target domain. This case opens doors for adversarial attacks against UDA methods since a small amount of misinformation about the target domain labels can lead the UDA methods into producing a representation similar to the unfavorable case, incurring a significant increase in the target domain error.
Motivated from this analysis of UDA methods under different data distributions, we evaluate the extent to which the performance of current UDA methods can suffer in presence of a small amount of adversarially crafted data. For this purpose, we propose novel data poisoning attacks, using mislabeled and clean-label points. We evaluate the effect of our poisoning attacks on popular UDA methods using benchmark datasets, where they were previously shown to be very effective. We find that our poisoning attacks cause UDA methods to either align incorrect classes from the two domains or prevent correct classes from being very close in the representation space. Both of these lead to the failure of UDA methods at reducing target domain error. With just 10% poison data in the source domain, target domain accuracy for current UDA methods is significantly reduced, dropping to almost 0% in some cases. This dramatic failure of UDA methods demonstrates their limits and suggests that the future UDA methods must be evaluated in adversarial settings along with evaluation on benchmark datasets to truly gauge their effectiveness at learning under the UDA setting.
Our main contributions are summarized as follows:
• We prove a lower bound on the target domain error that provides a necessary condition for successful learning in the UDA setting. Our bound shows the failure of learning a domain invariant representation while minimizing the source domain error at guaranteeing target generalization.
• We present example data distributions where UDA succeeds, fails, and where success and failure are equally likely. This sensitivity of UDA methods to the data distribution brings to light a new vulnerability of UDA methods to adversarial attacks such as data poisoning.
• To concretely understand the extent of this vulnerability of UDA methods, we propose novel data poisoning attacks using clean-label and mislabeled data. Our results show a dramatic failure of current UDA methods at target generalization in presence of poisoned data. Thus, our poisoning attacks can provide better insights into the robustness of UDA methods than those obtained from performance evaluation on benchmark datasets.

2 Background and related work
Analysis of unsupervised domain adaptation: Several previous works have studied the problem of UDA and have provided conditions under which UDA is possible [2, 1, 21, 22, 8, 3, 34]. [1, 2] proposed an upper bound on the target domain error which has inspired many UDA algorithms. Recent works [36, 6, 14] have improved the upper bounds on target domain error and have proposed a lower bound dependent on the labeling functions in the input space. Using this lower bound, the failure of learning in the UDA setting was explained in the case when marginal label distributions are different between the two domains. Our lower bound, on the other hand, is dependent on the labeling functions for the source and target domains, induced by the representation. This suggests that if a representation induces labeling functions that disagree on source and target domains then UDA provably fails even if the representation is domain invariant and minimizes error on the source domain. Thus our lower bound directly explains the observations of failure of UDA in many previous
works [9, 16, 33, 36, 34]. A detailed comparison of our work with other works analyzing the failure of learning in the UDA setting is present in Appendix G.
Algorithms for UDA: Many algorithms [9, 32, 18, 37, 30, 10] for UDA learn a domain invariant representation while minimizing error on the source domain. A popular adversarial approach to domain adaptation is DANN [9] which uses a discriminator to distinguish points from source and target domains based on their representations. Another popular method CDAN[18], uses classifier output along with representations to identify the domains of the points. IW-DAN and IW-CDAN [6] were recently proposed as extensions of the original DANN and CDAN with an importance weighting scheme to minimize the mismatch between the labeling distributions of the two domains. A different approach MCD [27], makes use of two task-specific classifiers as discriminators to align the two domains. This method adversarially trains the representation to minimize the disagreement between the two classifiers on the target domain data (classifier discrepancy) while training the classifiers to maximize this discrepancy. Another recent approach, SSL [35] uses self-supervised learning tasks (e.g. rotation angle prediction) to better align the two domains. In this work, we study the effect of poisoning on these methods as they have been shown to be effective at various UDA tasks.
Data poisoning: Data poisoning [4, 25, 12, 5, 13, 31, 24] is a training time attack where the attacker has access to the data which will be used by the victim for training. Most works [38, 26, 28, 23, 11] have considered data poisoning in a fully supervised setting where train and test sets are drawn from the same underlying data distribution (single domain setting). These works either target the classification of a single test point by adding a large number of poisoned data or require modifying all points in a class to affect the model’s performance on that class after retraining. Our work studies the effect of poisoning on the entire target domain in the UDA setting using popular UDA algorithms for training. The success of our poisoning attacks at significantly reducing the target domain accuracy with a small amount of poisoned data shows the ease of poisoning in the UDA setting. In contrast, poisoning leads to a small decrease in the overall test accuracy in the single domain setting, especially when using state-of-the-art classifiers such as deep neural networks.

3 When does learning fail in the unsupervised domain adaption setting?
Notations and settings: Let X be the data domain and D be the distribution over X with the corresponding pdf p(x). We assume there is a deterministic labeling function f : X → [0, 1] for the given binary classification task. The f(x) can be interpreted as Pr[y = 1|x]. Let g : X → Z denote the representation map that maps an input instance x to its features where Z is called feature or representation space. Let h : Z → [0, 1] be a hypothesis for binary classification on the representation space. Note that the representation map g induces a distribution over Z denoted by PrD̃[B] := PrD[g
−1(B)] and the corresponding density function p̃(z) on Z [2]. The g also induces the labeling function
f̃(z) := ED[f(x)|g(x) = z], (1)
for any B such that g−1(B) is D-measurable. The misclassification error e(h) w.r.t. the induced labeling function is e(h) = Ez∼D̃[|f̃(z) − h(z)|] where D̃ is the induced distribution over Z . Similarly, we define e(f̃ , f̃ ′) = Ez∼D̃[|f̃(z) − f̃ ′(z)|] and e(h, h′) = Ez∼D̃[|h(z) − h′(z)|]. The distributions p̃ and the labeling functions f̃ for the source and the target domains will be written as p̃S , p̃T , f̃S and f̃T , respectively. The total variation distance is D1(p̃, p̃′) = ∫ Z |p̃(z)− p̃ ′(z)|dz.

3.1 Lower bound on the target domain error
Most adversarial DA methods learn a domain invariant representation g : X → Z by minimizing error on the source domain and penalizing the mismatch between the marginal source and target distributions since the conditional distribution p̃T (z|y) for target domain is unavailable in the UDA setting. Some works [33, 36, 16] have shown this to be insufficient at guaranteeing target generalization. Recent works have proposed a new upper bound [36] or argued failure in terms of the upper bound in [1, 2] (eT (h) ≤ min{eT (f̃S , f̃T ), eS(f̃S , f̃T )}+ eS(h) +D1(p̃S , p̃T )) being large. However, a large upper bound does not guarantee failure. Thus, we prove a simple lower bound on the target domain error which shows the necessary condition for the success of learning in the UDA setting and also explains the failure of current UDA methods at guaranteeing target generalization.
Theorem 1. Let H be the hypothesis class and G be the class representation maps. Then, for all h ∈ H and g ∈ G,
eT (h) ≥ max{eS(f̃S , f̃T ), eT (f̃S , f̃T )} − eS(h)−D1(p̃S , p̃T ). (2)
The proof is in Appendix A. Even though our bound depends on the total variation distance which is a strict measure of distance and is difficult to estimate, the bound still explains the limitations of UDA. Extending the bound to different divergence metrics is left as future work. The following corollary is immediate from Theorem 1. Corollary 1.1. For all h ∈ H, g ∈ G,
eT (h) ≥ max{eS(f̃S , f̃T ), eT (f̃S , f̃T )} − eS(h)− √ 0.5DKL(p̃S ||p̃T ).
This is due to the Pinsker’s inequality: D1(p, p′) ≤ √
0.5DKL(p||p′). The interpretation of Eq. 2 is as follows. Since fT is not observable, the goal of UDA is to minimize the observable source classification error eS(h) and the domain mismatch D1(p̃S , p̃T ) (or other metrics such as dH∆H(p̃s, p̃T )):
min g,h es(h) +D1(p̃S , p̃T ). (3)
This leads to maximization of the second and the third term in the RHS of Eq. 2. With overparameterized models and large datasets, minimizing the empirical estimates of the quantities in Eq. 3 can drive eS(h) ' 0 and D1(p̃S , p̃T ) ' 0. Consequently, eT (h) ≥ max{eS(f̃S , f̃T ), eT (f̃S , f̃T )}. If f̃S and f̃T disagree on the source and target domains in the representation space, target domain error eT (h) will be provably large. Corollary 1.2. For all h ∈ H and g ∈ G,
|eT (h)−eS(f̃S , f̃T )| ≤ eS(h) +D1(p̃S , p̃T ), and |eT (h)−eT (f̃S , f̃T )| ≤ eS(h) +D1(p̃S , p̃T ).
This is obtained by combining the upper and the lower bounds (Appendix A). Thus a UDA method can only guarantee the target error eT (h) to be close to the labeling function mismatch e(f̃S , f̃T ). Whether e(f̃S , f̃T ) becomes larger or smaller after solving Eq. 3 is data/model dependent. For concreteness, we analyze the sensitivity of the performance of UDA methods to different data distributions using the following illustrative examples.

3.2 Illustrative examples showing the sensitivity of UDA methods to data distributions
Assume mixture-of-Gaussian distributions pS(x) and pT (x) for the source and the target domains in the input space, shown as blue and green blobs in Fig. 1. We consider a linear representation map g(x) = uTx from the input space X ⊂ R2 to the feature space Z ⊂ R (dashed line) that minimizes the source classification loss plus the marginal mismatch loss in Eq. 3. The optimal solution g to the minimization problem can be found accurately using a mix of analytical and
numerical optimizations (detailed in Appendix B). We demonstrate the performance of UDA on three different data distributions. In Case 1 (favorable case), the true labeling function for source and target is fS((x1, x2)) = fT ((x1, x2)) = I[x2 ≥ 0], that is, f is 1 in the upper halfspace and 0 in the bottom halfspace. One can verify (Appendix B) that the optimal u is the vertical direction (u = [0, 1]T ) and the best hypothesis is h(z) = I[z ≥ 0], in which case eS(h) = 0 and D1(p̃S , p̃T ) = 0. That is, perfect source classification and a perfect alignment of the marginals are achieved. Furthermore, the true labeling function in Z is f̃S(z) = f̃T (z) = I[z ≥ 0] (from Eq. 1) and therefore e(f̃S , f̃T ) = 0 as well as the target loss eT (h) = 0. In other words, the representation that minimizes Eq. 3 simultaneously minimizes e(f̃S , f̃T ), achieving the goal of reducing eT (h). However in Case 2 (unfavorable case), the true labeling function for target is upside-down fT ((x1, x2)) = I[x2 ≤ 0]. Since UDA does not use the target label nor the true labeling function, the optimal g is exactly the same as Case 1 (u = [0, 1]T ), in which case we still have eS(h) = 0 and D1(p̃S , p̃T ) = 0, but the labeling function mismatch becomes e(f̃S , f̃T ) = 1 as well as eT (h) = 1 which is the worst case. In other words, the representation that minimizes Eq. 3 simultaneously maximizes e(f̃S , f̃T ), totally failing at the goal of reducing eT (h). Case 3 exemplifies an ambiguous case. The optimal projection minimizing eS is still the vertical direction u = [0, 1]T but the optimal projection minimizing the alignment loss D1(p̃S , p̃T ) can be either of the ±45◦ directions (u = [±1/ √ 2, 1/ √
2]T ) with no preference of one over the other. Therefore the optimal solution u for Eq. 3 that trades off the source error and the mismatch loss has two equal-valued global solutions [±u1, u2]T for some u1, u2. One solution (Fig. 1, Case 3, left) yields a small e(f̃S , f̃T ) and the other (Fig. 1, Case 3, right) yields a large e(f̃S , f̃T ). As can be intuitively seen, UDA is successful in the former but fails in the latter, and which equal-valued solution will be chosen is undetermined. A similar example of the failure of UDA methods due to the presence of two global optima was presented in [14]. Unlike their example, we use mixture-of-Gaussian distributions and analytically compute the representation that will be obtained from the minimization of Eq. 3. Using the obtained representations, we evaluate the quantities appearing in our lower bound and show that it is predictive of the performance of UDA methods on different data distributions. Details of the analysis and the results are in Appendix B.
Motivated from this extreme sensitivity of the performance of UDA methods on simple data distributions, we set out to explore the effects of small changes in real-world data distributions (via data poisoning) on state-of-the-art UDA methods. The results of our poisoning attacks in the next section suggest that slight changes in the data distributions could have a drastic impact on the target domain generalization performance of state-of-the-art UDA methods. Our findings highlight the importance of using adversarial settings (such as evaluating the performance of UDA methods in presence of poisoned data) to gauge the effectiveness of future UDA methods at learning in the UDA setting.

4 Breaking unsupervised domain adaptation methods with data poisoning
In this section, we present our novel data poisoning attacks to evaluate the ease with which current UDA methods can be fooled into producing a representation that leads to a large error on the target domain1. We propose three methods to generate poisoned data which will be added to the clean source domain data. The first poisoning attack uses mislabeled data as poisons. Under this attack, we evaluate two approaches (a) adding mislabeled source domain data (wrong-label correct-domain poisoning) and (b) adding mislabeled target domain data (wrong-label incorrect-domain poisoning) as poisons. The second attack adds images from the source domain watermarked with images from the target domain with incorrect labels (watermarking attack) as poisons. The last poisoning attack uses poisoned data with clean labels. We evaluate two approaches for this attack (a) using source domain data (clean-label correct-domain poisoning) and (b) using target domain data (clean-label wrong-domain poisoning) to initialize the poison data. The intuitive pictures of how these poisoning attacks hurt UDA methods are shown in Figs. 4, 5, and 6 (details in Appendix C). We evaluate the effect of poisoning on popular UDA methods namely, DANN[9], CDAN[18], MCD[27], SSL[35] (with rotation-angle prediction task), IW-DAN[6], and IW-CDAN[6]. We compare the difference in the target accuracy attainable by these methods when using clean versus poisoned data. Two benchmark datasets are used in our experiments, namely Digits and Office-31. We evaluate four tasks using SVHN, MNIST, MNIST_M, and USPS datasets under Digits and six tasks under the Office-31 using Amazon (A), DSLR (D), and Webcam (W) datasets. Additional experiments evaluating the
1Our code can be found at https://github.com/akshaymehra24/LimitsOfUDA.
(a) DANN trained on clean data (b) DANN trained on poisoned data (c) CDAN trained on clean data (d) CDAN trained on poisoned data
Figure 2: (Best viewed in color.) t-SNE embedding of the data in the representation space (for MNIST→ USPS task) learned using DANN and CDAN on clean and poisoned source domain data. Without poisoning, correct classes (data from source class 2 is zoomed in) from two domains are aligned ((a) and (c)). The presence of poisoned data fools the methods into aligning incorrect classes from the two domains ((b) and (d)). The mismatch between the source and target classes is dependent on the labels of the poison data (due to which the target class 1 is aligned to the source class 2).
performance of our poisoning attacks against other popular UDA methods are present in Appendix F. For all experiments, we train UDA algorithms using neural networks whose architectures are similar to those used by previous works (see Appendix H for details).

4.1 Poisoning using mislabeled source and target domain data
In this experiment, mislabeled data is added to the clean source domain as poison. The labels provided by the attacker to the poison data are chosen to fool UDA methods into learning a representation that incurs large errors on the target domain. We use two simple and effective labeling functions for this. For the Digits dataset, we use a labeling function that systematically labels the poison data to the class next to their true class (e.g. poison points with true class one are labeled as two, points with true class two are labeled as three, and so on). For the Office-31 dataset, we use a labeling function that assigns the poison data the label of the closest (in the representation space learned using the clean source domain data) incorrect source domain class. Here the attacker is limited to adding only 10% poisoned data with respect to the size of the available target domain data (for experiments with different poison percentages see Appendix D). In wrong-label correct-domain poisoning, mislabeled source domain data are used as poisons. The results in rows marked with Poisonsource in Tables 1 and 2, show that UDA methods suffer only a minor decrease in target domain accuracy with this approach. This happens because the presence of a large amount of correctly labeled source domain data prevents the small amount of poisoned data from affecting the performance of UDA methods. A similar effect is observed when a small amount of mislabeled data is used for poisoning in the traditional single domain setting [23, 26]. However, if the relative size of poisoned data is larger or comparable to the size of clean source domain data, poisoning can be effective. This is observed in Table 2 when Amazon is the target data. The Amazon dataset is roughly 5 times bigger than both DSLR and Webcam datasets. Due to this, the permissible amount of poisoned data (10% of the target domain) makes the size of clean and poisoned data comparable, leading to successful poisoning. Thus, this attack causes UDA methods to fail in presence of a large amount of poisoned data.
In the wrong-label wrong-domain approach, mislabeled target domain data is used for poisoning. The effect of poisoning on discriminator-based methods [9, 18] is shown in Fig. 4. The domain discriminator used in these methods is maximally confused when marginal distributions of the source and target domains are aligned. However, alignment of the marginal distributions does not ensure alignment of the conditional distributions [36, 18]. The objective of achieving
(a) MCD trained on clean data (b) MCD trained on poisoned data (c) SSL trained on clean data (d) SSL trained on poisoned data
Figure 3: (Best viewed in color.) t-SNE embedding of the data in the representation space (for MNIST → USPS task) learned using MCD and SSL on clean and poisoned source domain data. Without poisoning, correct classes (data from source class 2 is zoomed in) from two domains are aligned ((a) and (c)). The presence of poisoned data prevents the methods from aligning correct classes from the two domains ((b) and (d)).
low source domain error pushes the source domain classifier to correctly classify the poison data. Thus placing the poison and source data with the same labels close in the representation space.
Since the poison data is mislabeled target domain data, poisoning makes UDA methods align wrong source and target domain classes. This leads to a significant decline in the target domain accuracy.
This is also evident from the t-SNE embedding for DANN and CDAN methods in Fig. 2. In absence of poisoning, correct source and target classes are aligned (Fig. 2 (a) and (c)), whereas in presence of poisoning wrong classes from the two domains are closer (Fig. 2 (b) and (d)). For MCD [27], which uses use classifier discrepancy to detect and align source and target domains, our poisoned data prevents the method from detecting target examples. This happens because the term that minimizes the error on the poisoned source domain reduces the discrepancy of the classifiers on poison data, which are from the target domain. Thus, both the generator (common representation) and discriminator (in the form of two classifiers) become optimal and there is no signal for the generator to align the two domains. The t-SNE
embedding in Fig. 3 shows this effect. In presence of poisoned data Fig. 3 (b), we see twenty distinct clusters rather than just ten (we have ten classes in the Digits) as seen in the absence of poisoning in Fig. 3 (a). In SSL [35], the generator must work well on the main task, i.e., should correctly classify all data in the poisoned source domain data. The auxiliary task ensures that representations of the source and target domains become similar as seen on clean data (Fig. 3 (c)). But in presence of poisoned data, similar representations of correct source and target domain classes leads to a drop in the accuracy of the main task on the poisoned data. This creates a conflict between the main and auxiliary tasks due to which correct source and target domain classes cannot be aligned (Fig. 3 (d)). The objective of making the main task accurate on poisoned data leads to target domain data being assigned the labels of the poisoned points. Thereby leading to a significant drop in the target-domain accuracy. The results of wrong-label wrong-domain poisoning present in rows marked with Poisontarget in Tables 1 and 2 show a significant reduction in the target domain accuracy compared to the accuracy obtained on clean data. On Digits, poisoning makes the target domain accuracy close to 0% on most tasks. On Office-31, poisoning causes at least a 20% reduction in the target domain accuracy in most cases. The reason tasks in Office-31 are less hurt by poisoning is because of the use of a pre-trained representation. Similar to previous works for Office-31, we use all labeled source and unlabeled target domain data for training and fine-tune from a representation pre-trained on the ImageNet dataset. The slowly changing representation pre-trained on a massive dataset weakens the effect of poisoning but cannot eliminate it. For the two tasks D→ W and W→ D, the fine-tuned representation, trained just on the clean source dataset (Source Only in Table 2) achieves high accuracy indicating the domains are already well aligned in terms of conditional distributions as well. As a result, UDA methods can easily align correct classes and suffer a drop close to 10% which is the amount of poisoned data added. However, this is not an interesting case for the evaluation of UDA methods as domains can be aligned just by training on the source domain data without the need for target domain data.
4.2 Poisoning with mislabeled watermarked data
In this experiment, we evaluate the effect of using poisoned data that looks like the source domain data. The poisoned data is generated by superimposing an image from the source domain with an image from the target domain. This method of generating poison data is known as watermarking [28]. To generate watermarked poison data we select an image from the target domain (t) and a base image from the source domain (s) such that it has the same class as the target domain image and lies closest to the target image (in the input space). The poisoned image (p) is obtained by a convex combination of the base and target images i.e., p = αt + (1 − α)s where α ∈ [0, 1]. α is selected such that the target image is not visible in the poison image ensuring the poisoned image looks like the image from the source domain. We use the same labeling function as discussed in the previous section to label the poisoned image and add 10% poison data to the source.
The illustrative picture of the effect of poisoning in this scenario is presented in Fig. 5. Successful poisoning, in this case, works just like in the previous experiment i.e., by making the representations of the data from wrong classes in source and target domains similar for DANN/CDAN, by reducing the discrepancy between the classifiers on target data for MCD and inducing a conflict between the supervised and auxiliary task for SSL. The t-SNE embedding showing the effect of poisoning (Fig. 9) and watermarked poison data (Fig. 10) are shown in the Appendix. We evaluate the effectiveness of this method on the Digits dataset for different values of α. The results in Table 3 show a significant decrease in the target domain accuracy even with a small watermarking percentage for all methods except CDAN. This is because the success of CDAN is dependent on the correctness of the pseudolabels on the target domain data (output of the classifier), which are used in the discriminator. Correct pseudo-labels provide CDAN a positive reinforcement to align correct classes from the two domains, leading to a failure of poisoning. However, as we increase the amount of watermarking, the quality of pseudo labels deteriorates. Thus, providing a negative reinforcement to CDAN that causes the alignment of wrong classes from the two domains.

4.3 Poisoning using clean-label source and target domain data
In this experiment, correctly labeled data is used for poisoning. To generate clean-label poison data that can affect the performance of UDA methods we must affect the features of the poison data. This requires solving a bilevel optimization problem [11, 23, 24] which we present in the Appendix E. Due to the high computational complexity involved in solving the bilevel problem, we propose to use a simple alternating optimization to demonstrate the feasibility of a clean label poisoning attack against UDA. We use the setting of previous works [11, 28] and consider misclassification of a single target domain test point (xtargettest , y target test ) rather than affecting the accuracy of the entire target domain as done in the previous two experiments. Let u =
{u1, ..., un} denote the poisoned data. To ensure a clean label, each poison point ui must have a bounded perturbation from a base point xbasei i.e, ‖ui − xbasei ‖ = ‖δi‖ ≤ and has label of the base i.e., ybasei . Thus, D̂poison = {(ui, ybasei )} Npoison i=1 , D̂source = {(xsourcei , ysourcei )} Nsource i=1 and D̂target = {(xtargeti , y target i )} Ntarget i=1 . The clean-label poison data u is such that when the victim uses
D̂source ⋃ D̂poison and D̂target for UDA, the target domain test point (xtargettest , y target test ) is misclassified.
The optimization problem for the clean-label attack is as follows.
min u Npoison∑ i=1 ‖g(xtargettest ;θ)− g(ui; θ)‖22 s.t. ‖xbasei − ui‖ ≤ , for i = 1, ...,Npoison,
min θ LUDA(D̂source
⋃ D̂poison, D̂target; θ).
(4)
The first problem minimizes the distance between the representations of the poison and the target domain test data (first term) while ensuring the poison data is not too far from the base data (second term). The second problem optimizes the parameters of the representation using UDA methods.
Attack success is evaluated by solving the second problem in Eq. 4 from scratch and evaluating the classification of xtargettest . This is illustrated in Fig. 6. The left part shows the case before retraining using the poison data generated from Eq. 4 and the right part shows how poisoning induces misclassification. We use two approaches for poisoning. The first uses source domain data and the second uses target domain data as base data. We add 1% poisoned data and test the effect of poisoning on a two-class (3 vs 8) domain adaptation problem on MNIST→ MNIST_M (see Appendix H). The results in Fig. 7 show that using target domain data as base data is significantly more successful under small permissible perturbation ( ). Using base data from the source domain requires larger distortion to keep
the poison data close to the target point in the representation space and is hence less successful. This shows the feasibility of clean label attacks against UDA methods. We believe the attack success can be improved by solving the bilevel level problem (Eq. 9 in Appendix E) and is left as future work.

5 Conclusion
We studied the problem of UDA and highlighted the limitations of learning under this setting. We proposed a simple lower bound on the target domain error for UDA, dependent on the labeling function induced by the representation. The lower bound demonstrated that learning a domain invariant representation while minimizing error on the source domain cannot guarantee good generalization on the target domain. We analyzed a simple model and showed the existence of cases where UDA can naturally succeed or fail. The analysis also highlighted a case where, without access to any labeled target domain data, the success and the failure are equally likely. In such a case, the presence of even a small amount of poisoned data can make the data distribution unfavorable for UDA methods, making them fail dramatically in comparison to the case without poisoning. We proposed novel data poisoning attacks to demonstrate the failure of popular UDA methods with a small amount of poisoned data. Our results suggest that the performance of a UDA method in presence of poisoned data indicates how well the method aligns the conditional distributions across the two domains. Thus, we believe our attacks can be used for evaluating UDA methods, beyond simple benchmark datasets, to reveal their robustness to data distributions inherently unfavorable for UDA.

6 Acknowledgment
We thank the anonymous reviewers for their insightful comments and suggestions. This work was supported by the NSF EPSCoR-Louisiana Materials Design Alliance (LAMDA) program #OIA1946231 and by LLNL Laboratory Directed Research and Development project 20-ER-014 (LLNLCONF-824206). This work was performed under the auspices of the U.S. Department of Energy by the Lawrence Livermore National Laboratory under Contract No. DE-AC52-07NA27344, Lawrence Livermore National Security, LLC 2.
2This document was prepared as an account of the work sponsored by an agency of the United States Government. Neither the United States Government nor Lawrence Livermore National Security, LLC, nor any of their employees make any warranty, expressed or implied, or assumes any legal liability or responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately owned rights. Reference herein to any specific commercial product,

Reviewer Summary
Reviewer_4: The paper addresses the vulnerability of domain adaptation to adversarial attacks. First, a lower bound on the target risk of a given classifier is derived, which provides necessary conditions for the success of domain adaptation methods. Then, 3 examples on 2D data are given and analyzed to show the potential failure of DA methods aiming at aligning both domains while remaining good for classification on the source one. The last example describes an ambiguous situation for which success and failure of adaptation are equally likely, from which the authors have the inspiration that adding a few poison data can easily favor the bad solution. They concretize this idea by proposing 3 data poisoning strategies and they show their ability to deteriorate the performance of previously proposed DA methods.

Reviewer_5: In this work, the authors aim to study the limitations of Unsupervised Domain Adaptation through data poisoning. Specifically, the authors first show the failure of UDA by proving a simple lower bound, then present example distributions where UDA succeeds or fails. Finally, they propose several novel data poisoning attacks that use clean-label and mislabeled points to fail existing UDA methods and verify these attack methods in the experiments on two benchmark datasets.

Reviewer_6: This paper aims to understand the limitation of unsupervised domain adaptation (UDA) through data poisoning. The paper reveals that the modern UDA deep learning approach is vulnerable to noise data, with almost zero accuracies in the poisoning target samples.
-----------Updates after Discussion
I appreciate the efforts and additional experimental results by the authors. Thus I recommend a (weakly) positive score.

Reviewer Comments
Reviewer_4: The paper is clearly written and easy to follow, and the theoretical technical results are supported by proofs.
Strengths
The overall idea of showing the vulnerability of DA methods to adversarial attacks is interesting. It illustrates the intuition that the complete absence of target data is a considerable lack of information for DA approaches methods. Also, the provided empirical evidence for this vulnerability concerns even new DA approaches that aim to improve previous ones (IW-DAN and IW-CDAN).
Corollary 3 is very insightful as it offers more precision on which quantity is being minimized by DA methods that try to have a good performance on the source domain while embedding the source and target domain into a representation space in which they are aligned.
Weaknesses
About lower bounds on the ideal joint error: in line 39, it is argued that a large upper bound does not guarantee the failure of DA methods. However, any lower bound on the ideal joint hypothesis is equivalent to a lower bound on the target risk, due to the equivalence between
min
h
∈
H
e
S
(
h
)
+
e
T
(
h
)
≥
B
and
∀
h
∈
H
,
e
T
(
h
)
≥
B
−
e
S
(
h
)
, where
B
denotes some lower bound.
Lack of comparison to previous work on the drop of performance on the target domain due to the deep embedding of both domains
Comparison to the lower bound established in Theorem 4.3 of [1]: using the notation of the paper being reviewed, their result states that for any
h
∈
H
,
g
∈
G
,
e
T
(
h
)
+
e
S
(
h
)
≥
1
2
(
d
J
S
(
p
~
S
,
p
~
T
)
−
d
J
S
(
p
S
Y
,
p
T
Y
)
)
2
(where
p
S
Y
and
p
T
Y
denote the label marginal distributions), whereas Theorem 1 of the reviewed paper states that
e
T
(
h
)
+
e
S
(
h
)
≥
max
(
e
S
(
f
~
S
,
f
~
T
)
,
e
T
(
f
~
S
,
f
~
T
)
)
−
D
1
(
p
~
S
,
p
~
T
)
. The r.h.s. of the former inequality is always nonnegative, which is not the case for the latter. This comparison should be carried out in detail.
Another contribution concerned with the performance drop on the target domain after the embedding in the new representation space is [2]. In particular, their Example 1 illustrates a case with two global solutions to the classic DA minimization problem, which is the same spirit of Case 3 of Figure 1 of the paper under review.
Suggestions
Detailed comparison to [1,2].
Plotting 2D map of the effect of both the percentage of poisonous data and the divergence between domains.
Comments
In line 232, the "generator" term should be replaced with "source classifier" for example. Indeed, in spite of the similarity in mechanism between GANs and adversarial domain adaptation methods, they are not the same. This holds also for line 242
Having seen Figure 1, in which the data is represented in a 2D space, I was confused when looking at Figure 3 for a moment. Indeed, it seems like the target classes were flipped from left to right, whereas the arrow represents an alignment in a new representation. In this case, another schematic representation for the data in the new representation space should be used.
I do not see the interest of Corollary 2 concerning the KL-divergence, as it states a looser lower bound than the one of Theorem 1 (as
−
D
1
(
⋅
,
⋅
)
≥
−
1
2
D
K
L
(
⋅
‖
⋅
)
), and it does not have other interests shown in the paper, besides involving another divergence.
Minor comments
In lines 168 and 170, it should be
e
(
f
~
S
,
f
~
T
)
instead of
e
(
f
~
S
,
f
~
S
)
.
Line 252: "poisoned data, leads" --> "poisoned data leads"
Line 253: ". There by leading " --> ", thereby leading"
Line 288: "Fig. 9 ... Fig. 8" ---> "Fig. 10 ... Fig. 9"
Line 338: "showe" --> "show"
Line 340: "presence of even small" --> "the presence of even a small"
Questions
In the third case of examples in section 3.2, the classifier is initialized randomly, and this randomness causes the optimization to converge to either a good or bad solution. Do you have an idea about the initial conditions of the solution that lead to one solution or the other? (e.g. a partition of two regions in the solution space according to which the optimization results in one of the two minimizers).
References
[1] Zhao, H., Combes, R. T. D., Zhang, K., & Gordon, G. (2019). On Learning Invariant Representations for Domain Adaptation. International Conference on Machine Learning, 7523–7532.
[2] Johansson, F. D., Sontag, D., & Ranganath, R. (2019). Support and Invertibility in Domain-Invariant Representations. The 22nd International Conference on Artificial Intelligence and Statistics, 527–536.
Limitations And Societal Impact:
Although the societal impact was not directly addressed in the paper, the aspects of the adversarial attack it tackles are vital in real-world applications. For example, in autonomous driving, it can be very problematic for an algorithm to adapt to new environments if it is not robust to data poisoning.
Ethical Concerns:
.
Needs Ethics Review: No
Time Spent Reviewing: 4

Reviewer_5: Pros:
As far as I’m aware, this work is the first to explore the limitations of UDA methods from the lens of robustness. I believe this work could provide new insight for the community of UDA. The code is available with the submission which improves the reproducibility index of the paper. Overall, this paper is well-written and easy-to-follow.
Cons:
The relationship between data poisoning and the natural failure of UDA methods is not clear. From the view of robustness, supervised learning methods are still vulnerable under adversarial attacks like data poisoning, even though there are enough labeled examples from the target domain. It is not surprising that UDA methods are also vulnerable to data poisoning. Therefore, I am not sure that the fact that UDA methods are easy to be attacked can demonstrate the natural failure problem of UDA methods. If the authors could provide a convincing explanation during rebuttal, I will consider improving my score on this work.
The authors have addressed my concerns during rebuttal, however, the writing in the manuscript is still problematic. Thus I decide to improve my score to 6 here.
Limitations And Societal Impact:
The authors claim that they have addressed the limitations of this work in Section 3 but I cannot find it. Indeed, they introduce two potential future directions, like extending the simple lower bound to different divergence metrics and boosting the poisoning attack by solving the bilevel problem.
Needs Ethics Review: No
Time Spent Reviewing: 6

Reviewer_6: Pros:
This paper reveals an important and interesting fact that the modern UDA approach is quite vulnerable in the noisy datasets (particularly the poisoned target samples), which restricts its real practical utility.
Cons:
Despite the empirical discovery being interesting and promising, the current version seems preliminary and the in-depth analysis is lacking.
The proposed theoretical result is insufficient to understand the inherent limitation.
Overall it is a good paper in the proof of concept but the main discovery is not surprising. Moreover, the in-depth analysis/justification/proof is not sufficient. Thus, I recommend a weak rejection but encourage a major revision for future submission.
Comments:
This paper has several discoverings, the following are the comments for each contribution.
“a lower bound on the target domain error providing a necessary condition for the success of learning in a UDA setting. The bound shows the failure of learning a domain invariant representation that minimizes source domain error at guaranteeing target generalization”.
In fact, this discovery is not surprising and several recent works have a good discussion on this idea. e.g ref [1,2], which is missing from the author. Besides, the theoretical proof is only valid for the bijective or invertible representation function (see Line 83 the definition). In fact, the most present deep UDA approach ** is not satisfied** this assumption. That being said, theory and practice have several non-ignorable gaps.
As for the conclusion of Theorem 1, the results do not provide novel insights, the lower bound can be vacuous in the presence of noisy labels. Supposing we have a MLP network the noisy source error is
e
s
(
h
)
>
0
, then the lower bound can be still zero. From this perspective, the lower bound has some issues.
Line 40, the author states that “a large upper bound does not guarantee failure of UDA methods.“ I agree with this point. However, the upper bound is introduced as sufficient condition to ensure it works. Since the UDA only works in very specific scenarios, in the practical algorithm we need to justify the sufficient condition rather than the necessary assumption.
Another important concern of the proposed theoretical analysis is that the data generation procedure should be stochastic rather than a deterministic labeling function
f
. This is particularly important in this paper. The later experimental settings are exactly the stochastic setting: the same digit input
x
can have two different values with 0 and 1. In contrast, the deterministic setting has no random sampling procedure (i,e for the same X, f(x) is fixed), which clearly violates the empirical settings.
“ present example distributions where UDA succeeds, fails, and where success and failure are equally likely. The last case opens door to adversarial attacks such as data poisoning.”
In fact, the first two counterexamples are essentially the same as the ones in Ref [1]. Thus they are not surprising. As for the last counterexamples, the problem lies in the inherent property in UDA and is impossible to solve without additional assumptions/conditions. The authors are strongly encouraged to discover these conditions and justify them.
“several novel data poisoning attacks….The poisoning attacks can be used to obtain a better sense of the robustness of the future UDA methods beyond simple benchmark datasets”
In general, this part is quite interesting and deserves further attention in the community. And I have the following question:
If we added the target sample noise to the source, the UDA-based approach will tend to find a similar point to learn the target. Naturally, the performance will have a significant drop in the shallow network since it is almost the worst-case attack. My concern is in the worst case, no algorithm can probably work. Thus it is an impossible task?
As far as I understand, the target sample attack is believed to be the most interesting and promising direction. (other types of attacks are quite common and fairly easy to solve by adopting the existing approaches) Thus I do think an additional discussion about the assumption, the condition is important for the future version.
Other Comments
The writing and organization can be improved. Sec 3.2 has only one paragraph and is relatively difficult to read.
The performance drop in the digits and large dataset (e,g office) have quite different behavior. An in-depth analysis is expected.
Line 142-144, the overparameterized network does not necessarily guarantee an expected term as zero. A rigorous justification is expected.
Reference [1] Support and Invertibility in Domain-Invariant Representations.
[2] Representation bayesian risk decompositions and multi-source domain adaptation
Limitations And Societal Impact:
The authors have adequately addressed the limitations and potential negative societal impact.
Needs Ethics Review: No
Time Spent Reviewing: 6