Understanding the Limits of Unsupervised Domain Adaptation via Data Poisoning
Authors: Akshay Mehra
Bhavya Kailkhura
Pin-Yu Chen
Jihun Hamm
Venue: NeurIPS 2021 Poster (2021)
Keywords: data poisoning
unsupervised domain adaptation
domain adaptation

Abstract: Unsupervised domain adaptation (UDA) enables cross-domain learning without target domain labels by transferring knowledge from a labeled source domain whose distribution differs from that of the target. However, UDA is not always successful and several accounts of `negative transfer' have been reported in the literature. In this work, we prove a simple lower bound on the target domain error that complements the existing upper bound. Our bound shows the insufficiency of minimizing source domain error and marginal distribution mismatch for a guaranteed reduction in the target domain error, due to the possible increase of induced labeling function mismatch. This insufficiency is further illustrated through simple distributions for which the same UDA approach succeeds, fails, and may succeed or fail with an equal chance. Motivated from this, we propose novel data poisoning attacks to fool UDA methods into learning representations that produce large target domain errors. We evaluate the effect of these attacks on popular UDA methods using benchmark datasets where they have been previously shown to be successful. Our results show that poisoning can significantly decrease the target domain accuracy, dropping it to almost 0% in some cases, with the addition of only 10% poisoned data in the source domain. The failure of these UDA methods demonstrates their limitations at guaranteeing cross-domain generalization consistent with our lower bound. Thus, evaluating UDA methods in adversarial settings such as data poisoning provides a better sense of their robustness to data distributions unfavorable for UDA.

Understanding the Limits of Unsupervised Domain Adaptation via Data Poisoning
Akshay Mehra', Bhavya Kailkhura', Pin-Yu Chen and Jihun Hamm'
'Tulane University LLawrence Livermorc National Laboratory (amehra, jhamm3)@tulane edu, kailkhural@lnn..ge pin-yu.
'IBM Research
aan-yu.chen@a chen@ieeennccm com
Abstract
Unsupervised domain adaptation (UDA) enables cross-domain learning without target domain labels by transferring knowledge from labeled source domain whose distribution differs from that of the target. However, UDA not always successful and several accounts of 'negative transfer' have been reported in the literature. In this work, we prove a simple lower bound on the target domain error that complements the existing upper bound. Our bound shows the insufficiency of minimizing source domain error and marginal distribution mismatch for a guaranteed reduction in the target domain error, due the possible increase of induced labeling function mismatch. This insufficiency is further illustrated through simple distributions for which the same UDA approach succeeds, fails, and may succeed or fail with an equal chance. Motivated from this, we propose novel data poisoning attacks to fool UDA methods into learning representations that produce large target domain errors. We evaluate the effect of these attacks on popular UDA methods using benehmark datasets where they have been previously shown to be successful. Our results show that poisoning can significantly decrease the target domain accuracy, dropping it to almost 0% some cases, with the addition of only 10% poisoned data in the source domain. The failure of these UDA methods demonstrates their limitations at guarantecing cross-domain generalization consistent with our lower bound. Thus, evaluating UDA methods in adversarial settings such as data poisoning provides a better sense their robustness to data distributions unfavorable for UDA.
1 Introduction
The problem of domain adaptation (DA) arises when the training and the test data distributions are different, violating the common assumption of supervised learning. In this paper, we focus on unsupervised DA (UDA), which is special case of DA when labeled information from the target domain is available. This setting is useful for applications where obtaining large-scale well-curated datasets is both time-consuming and costly. The seminal works [ 2 proved an upper bound classifier's ' target domain error in the UDA setting leading to several algorithms for learning this setting. Many of these algorithms rely on learning domain invariant representation by minimizing the error on the source domain and a divergence measure between the marginal feature distributions of the source and target domains. Popular divergence measures include total variation distance, Jensen- and maximum mean discrepancy The success of these algorithms is argued in terms of minimization of the upper bound proposed in 1 along with an improvement in the target domain accuracy on benchmark UDA tasks.
Shannon divergence 9 32, 37 Wasserstein distance (29 15
19 17 20
Despite this success benchmark datasets, some works 19 the failure these methods in different scenarios. Recent works have explained this apparent failure of UDA methods by proposing new upper bounds [36]
]6
36 33
have presented evidence of
34 the target domain error while
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

others have demonstrated the cause of failure using experiments showing that learning a domain invariant representation that minimizes the source domain error can cause an increase in the error of the ideal joint hypothesis [16]. To provably explain this apparent failure of learning in the UDA setting we propose a lower bound on the target domain error. Our lower bound provides necessary condition for successful learning in the UDA setting, complementing the existing upper bound of []] and is dependent on the difference between the labeling functions of source and target domain data induced by the representation map. For cases where the induced labeling functions match on the source and the target domain data (i.e., favorable case), the success of UDA is explained using the upper bounds proposed by previous works [ 36 14] For a representation that aligns the source and the target domain data and minimizes the error on the source but induces labeling functions that don't agree the source and the target domain data (i.e., unfavorable case), our lower bound explains the failure of UDA. Our analysis brings to light yet another case (i.e., ambiguous case) of data distributions where success and failure of UDA are equally likely. This happens due the lack of label information from the target domain. This case opens doors for adversarial attacks against UDA methods since small amount of misinformation about the target domain labels can lead the UDA methods into producing a representation similar to the unfavorable case, incurring a significant increase in the target domain error.
Motivated from this analysis of UDA methods under different data distributions, we evaluate the extent to which the performance of current UDA methods can suffer in presence of small amount of adversarially crafted data. For this purpose, we propose novel data poisoning attacks, using mislabeled and clean-label points. We evaluate the effect of our poisoning attacks popular UDA methods using benchmark datasets, where they were previously shown to be very effective. We find that poisoning attacks cause UDA methods to either align incorrect classes from the two domains or prevent correct classes from being very close in the representation space. Both of these lead to the failure of UDA methods at reducing target domain error. With just 10% poison data in the source domain, target domain accuracy for current UDA methods is significantly reduced, dropping to almost 0% in some cases, This dramatic failure of UDA methods demonstrates their limits and suggests that the future UDA methods must be evaluated in adversarial settings along with evaluation on benchmark datasets to truly gauge their effectiveness learning under the UDA setting.
Our main contributions are summarized as follows:
We prove a lower bound on the target domain error that provides a necessary condition for suc- cessful learning in the UDA setting. Our bound shows the failure of learning domain invariant representation while minimizing the source domain error at guarantecing target generalization.
We present example data distributions where UDA succeeds, fails, and where success and failure are equally likely. This sensitivity of UDA methods to the data distribution brings to light new vulnerability of UDA methods to adversarial attacks such as data poisoning.
To concretely understand the extent of this vulnerability of UDA methods, we propose novel data poisoning attacks using lleen-label and mislabeled data. Our results show a dramatic failure of current UDA methods at target generalization in presence of poisoned data. Thus, our poisoning attacks can provide better insights into the robustness of UDA methods than those obtained from performance evaluation on benchmark datasets.
2 Background and related work
Analysis of unsupervised domain adaptation: Several previous works have studied the problem of UDA and have provided conditions under which UDA possible [ 22 g 34 2 proposed an upper bound on the target domain eтror which has inspired many UDA algorithms. Recent works [36] 14 have improved the upper bounds on larget domain error and have proposed a lower bound dependent on the labeling functions in the input space. Using this lower bound, the failure of learning in the UDA setting was explained in the case when marginal label distributions are different between the two domains. Our lower bound, on the other hand, is dependent on the labeling functions for the source and target domains, induced by the representation. This suggests that if a representation induces labeling functions that disagree on source and target domains then UDA provably fails even if the representation domain invariant and minimizes error on the source domain. Thus our lower bound directly explains the observations of failure of UDA in many previous

---

works 16 33 36 (4]. A detailed comparison of our work with other works analyzing the failure of learning the UDA setting present in Appendix G
for UDA learn a domain invariant representation while minimizing error the source domain. A popular adversarial approach to domain adaptation is DANN [-] which uses a discriminator to distinguish points from source and target domains based on their representations. Another popular method CDAN uses classifier output along with representations to identify the domains of the points. IW-DAN and IW-CDAN [ were recently proposed as extensions of the original DANN and CDAN with an importance weighting scheme to minimize the mismatch between the labeling distributions of the two domains. A different approach MCD [27]. makes use of two task task-spe classifiers as discriminators to align the two domains. This method ad adversarially trains the representation to minimize the disagreement between the two classifiers on the target domain data (classifier discrepancy) while training the classifiers to maximize this discrepancy. Another recent approach, SSL. 35 uses self-supervised learning tasks (e.g. rotation angle prediction) to better align the two domains. In this work, we study the effect of poisoning these methods they have been shown to be effective at various UDA tasks.
Algorithms for UDA: Many algorithms p
32
18
37
30
10
is training time attack where the attacker has access to the data which will be used by the victim for training. Most works [8] [26] 28 23 I have considered data poisoning in a fully supervised setting where train and test sets are drawn from the same underlying data distribution (single domain setting). These works either target the classification of a single test point by adding a large number of poisoned data or require modifying all points in a class to affect the model's performance on that class after retraining. Our work studies the effect of poisoning on the entire target domain in the UDA setting using popular UDA algorithms for training. The success of our poisoning attacks at significantly reducing the target domain accuracy with a small amount of poisoned data shows the ease of poisoning in the UDA setting. In contrast, poisoning leads to small decrease in the overall test accuracy in the single domain setting, especially when using state-of-the-ar classifiers such as deep neural networks.
Data poisoning:
Data poisoning 4
3 When does learning fail in the unsupervised domain adaption setting?
Notations and settings: Let X' be the data domain and D be the distribution over X' with the corresponding pdf p(x)). We assume there is a deterministic labeling function f X [0, 1] for the given binary classification task. The f(x can be interpreted as Priy = 1|æ].. Let g X' Z denote the representation map that maps an input instance æ to its features where Z is called feature or representation space. Let h : Z - [0, 1] be a hypothesis for binary classification on the representation space. Note that the representation map g induces distribution over Z denoted by Pr == Prolg 1 (B)] and the corresponding density function p(2) on Z (2] The g also induces the labeling function
f(z) = FF(s())lo(s)) = =| =].
for any B such that g (BB) is D- measurable. The misclassification error e(h) w.r.t. the induced labeling function is e(h) F. (l) -(=)]] where D is the induced distribution over 2. Similarly, we define e(f, f') = E, E.-bl(((=) ['(=)]] and e(h, R²) = E [|A(2) - [/(=)]]. The distributions ` and the labeling functions ff for the source and the target domains will be written as Ps. pr, fs and fr, respectively. The total variation distance is (², = Jz 10(2) - f (z)]d2.
3.1 Lower bound on the target domain error
Most adversarial DA methods learn a domain invariant representation g : X + Z by minimizing егтor on the source domain and penalizing the mismatch between the marginal source and target distributions since the conditional distribution (2ly) for target domain is unavailable in the UDA setting. Some works 33 36 [6] have shown this to be insufficient at guaranteeing target gener- alization. Recent works have proposed a new upper bound (36) or argued failure in terms of the upper bound in [ 2 (h) S min(er (Js, [T), (Js,
+ (s(h) + Di (ūs. being large. However, a large upper bound does not guarantee failure. Thus, we prove a simple lower bound on the target domain error which shows the necessary condition for the success of learning in the UDA setting and also explains the failure of current UDA methods at guarantceing target generalization.
3

---

Case
Case
Case 3
OR
18.01
S
LOL S
SOURCE TARGET
Figure 1: Illustrative cases for UDA. In UDA, can observe the source data (blue blobs). the source labels (class + and -), and the target data (green blobs) but not the target labels. The optimal decision boundary (red dotted line) and linear representation g(x) u² x from the input space in R to the feature space in R (dashed line) minimizing the source and the alignment losses in Eq. can be computed accurately. Depending
the data distribution, domain adaptation can be successful (Case 1), fail (Case 2); or be undetermined (Case 3). Case 3 has two global minima with drastically different target domain performance, even though representations in both cases minimize the the source distribution as well as align the marginal distributions of the two domains.
Theorem 1, Let H. be the hypothesis class and G be the class representation maps. Then, for all h H. and g E g.
eT(h) 2 fr)
er( - D -
The proof is in Appendix A
Even though our bound depends on the total variation distance which is a strict measure of distance and is difficult estimate, the bound still explains the limitations of UDA. Extending the bound to different divergence metrics is left as future work. The following corollary is immediate from Theorem Corollary 1.1. For all h H.
(r max
fr) eg(h) (o.5Dx
This is due the Pinsker's inequality: Di (p,p') <0.5Dx< The interpretation of Eq. 2 is as follows. Since fr not observable, the goal UDA to minimize the observable source classifi- cation error eh( and the domain mismatch Dj (ps, fT) (or other metrics such dHAH (p.,, pr) )):
min (h) Dr (T)) g.h
This leads to maximization of the second and the third term in the RHS of Eq. With overparameter ized models and large datasets, minimizing the empirical estimates of the quantities Eq. can drive (h) > and D₂ (ps, pr) ~ 0. Consequently, (h) 2 maxfes(fs. fr), (fs. fr)). fs and Ji disagree on the source and target domains the representation space, target domain error er will be provably large.
Corollary 1.2. For all h E H and g E G.
ets I es(h) + Di Pr), and (er(h) - ers, s es(h) + Di Pr).
This is obtained by combining the upper and the lower bounds (Appendix Ap. Thus UDA method can only guarantee the target error er-(h) to be close to the labeling function mismatch e(fs, fr). Whether e(fs: fr) becomes larger or smaller after solving Eq. is data/model dependent. For concreteness, analyze the sensitivity of the performance of UDA methods to different data distributions using the following illustrative examples.
Illustrative examples showing the sensitivity of UDA methods to data distributions
Assume mikture-of-GGaussan Gaussian distributions (x) and pr (xx for the source and the target domains the input space, shown as blue and green blobs in Fig. [ We consider linear representation map g(x) = w x from the input space X' C I22 to the feature space Z C R (dashed line) that
minimizes the source classification loss plus the marginal mismatch loss in Eq. 3 The optimal
solution g the minimization problem can be found accurately using mix of analytical and

---

numerical optimizations (detailed in Appendix Bj. We demonstrate the performance of UDA on three different data distributions. In Case (favorable case), the true labeling function for source and target is
= fr ((11, x22) T [r > 0]. that is, f is in the upper halfspace and O in the bottom halfspace. One can verify (Appendix [ that the optimal u is the vertical direction (u = [0. 1)²) and the best hypothesis is h(=) = 7|= 2 Oj. in which case (5(h) = 0 and Dj (fs: PT) 0. That is, perfect source classification and a perfect alignment of the marginals are achieved. Furthermore, the true labeling function in E is fs (=) - fr(=)= = M[ 2 0J (from Eq. [] and therefore e(ʃs, /T) - 0. In other words, the representation that minimizes Eq.
as well the target loss er (h) simultaneously minimizes e(fs, fr). achieving the goal of reducing er(h). However in Case
(unfavorable case), the true labeling function for target is upside-down frr((11, x2)) = //222 I 0]. Since UDA does not use the target label nor the true labeling function, the optimal g is exactly the
same as Case 1 (u = [o, IJ²). in which case we still have es (h) = 0 and Di (ās, pr) = 0, but the labeling function mismatch becomes e(fs: fr) = 1 as well as er (h) = 1 which is the worst case. In other words, the representation that minimizes Eq. 3 simultanoously maximizes e e(fs: totally failing at the goal of reducing (T()). Case exemplities an ambiguous case. The optimal projection minimizing& eg is still the vertical direction = (0, 1|a but the optimal projection minimizing the alignment loss Di (Ās.Pr) can be either of the +-45 directions (u 11/ 1/v2 with no preference of one over the other. Therefore the optimal solution u for Eq. 3 that trades off the source error and the mismatch loss has two equal- -valud global solutions [+ Ug
IT for some u1, U2. One solution (Fig. 1 Case 3, left) yields small e e(Ss, Ir) and the other (Fig. Case 3, right) yields
large e(fs, fr)) As can be intuitively seen, UDA is successful in the former but fails in the latter, and which equal-valued solution will be chosen is undetermined. A similar example of the failure of UDA methods due to the presence of two global optima was presented in [4]. Unlike their example, we use mixture-of-GGassin distributions and analytically compute the representation that will be obtained from the minimization of Eq. 3 Using the obtained representations, we evaluate the quantities appearing in our lower bound and show that it is predictive of the performance of UDA methods on different data distributions. Details of the analysis and the results are in Appendix B
Motivated from this extreme sensitivity of the performance of UDA methods on simple data distri- butions, we set out to explore the effects of small changes in real-world data distributions (via data poisoning) on offte-of-the-art UDA methods. The results of our poisoning attacks in the next section suggest that slight changes in the data distributions could have a drastic impact on the target domain generalization performance of state-of-the of using adversarial settings (such as evaluating the performance of UDA methods in presence of poisoned data) to gauge the effectiveness of future UDA methods at learning in the UDA setting.
UDA methods. Our findings highlight the importance
4 Breaking unsupervised domain adaptation methods with data poisoning
In this section, we present our novel data poisoning attacks to evaluate the ease with which current UDA methods can be fooled into producing a representation that leads to large error on the target domain") We propose three methods generate poisoned data which will be added to the clean source domain data. The first poisoning attack uses mislabeled data as poisons. Under this attack, we evaluate two approaches (a) adding mislabeled source domain data (wrong-labed correct-domann poisoning) and (b) adding mislabeled target domain data (wrong-labes incorrect-domaiie poisoning) as poisons. The second attack adds images from the source domain watermarked with images from the target domain with incorrect labels (watermarking attack) as poisons. The last poisoning attack uses poisoned data with clean labels. We evaluate two approaches for this attack (a) using source domain data (clean-label correct-domain poisoning) and (b) using target domain data (clean-label wrong-domain poisoning) to initialize the poison data. The intuitive pictures of how these poisoning attacks hurt UDA methods are shown in Figs. 4 5 and (details in Appendix (C. We evaluate the effect of poisoning on popular UDA methods namely, DANNO) CDAN[8], MCD[27], [SL (with rotation-angle prediction task), IW-DAN and IW-CDAN(6). We compare the difference in the target accuracy attainable by these methods when using clean versus poisoned data. Two benchmark datasets are used in our experiments, namely Digits and Of We evaluate four tasks using S SVHN, MNIST, MNIST M. and USPS datasets under Digits and six tasks under the Office-31 using Amazon (A), DSLR (D), and Webcam (W) datasets. Additional experiments evaluating the
'Our code can be found at https: //:gt:/bbbttt http:///ww.....caaoooo.//aaaaaaa
5

---

(a) DANN trained on clean data
(b) DANN trained on poi- soned data
(c) CDAN
trained on
(d) CDAN trained on poi- soned data
clean data
SOURCE
TARGET
POISON
Figure (Best viewed in color.) t-SNE embedding of the data in the representation space (for MNIST - USPS task) learned using DANN and CDAN clean and poisoned source domain data. Without poisoning, correct classes (data from source class zoomed in) from two domains are aligned ((a) and (c)). The presence of poisoned data fools the methods into aligning incorrect classes from the two domains ((b) and (d)). The mismatch between the source and target classes dependent on the labels of the poison data (due to which the target class is aligned the source class 2).
performance of our poisoning attacks against other popular UDA methods are present in Appendix F For all experiments, we train UDA algorithms using neural networks whose architectures are similar to those used by previous works (see Appendix H for details).
4.1 Poisoning using mislabeled source and target domain data
In this experiment, mislabeled data is added to the clean source domain poison. The labels provided by the attacker to the poison data are chosen to fool UDA methods into learning a representation that incurs large errors the target domain. We use two simple and effective labeling functions for this. For the Digits dataset, we use a labeling function that systematically labels the poison data to the class
their true class (e.g. poison points with true class one labeled as two, points with true class two are labeled as three, and so on). For the Office-31 dataset, we use a labeling function that assigns the poison data the label of the closest (in the representation space learned using the clean source domain data) incorrect source domain class. Here the attacker is limited to adding only 10% poisoned data with respect to the size of the available target domain data (for experiments with different poison percentages see Appendix DD. In wrong-label correct-domain poisoning, mislabeled source domain data are used as poisons. The results rows marked with Poisonsource in Tables [ and show that UDA methods suffer only a minor decrease in target domain accuracy with this approach. This happens because the presence data prevents the small amount of poisoned data from affecting the performance of UDA methods. similar effect is observed when small amount mislabeled data used for poisoning in traditional single domain setting (23 26]. However, if the relative size of poisoned data is larger or comparable to the size of clean source domain data, poisoning can be effective. This observed in Table 2 when Amazon is the target data. The Amazon dataset is roughly 5 times bigger than both DSLR and Webcam datasets. Due to this, the permissible amount of poisoned data (10% the target domain) makes the size of clean and poisoned data comparable, leading to successful poisoning. Thus, this attack causes UDA methods to fail presence of a large amount of poisoned data.
large amount of correctly labeled source domain
In the wrong-label wrong-domain approach, mislabeled target domain data is used for poison- ing. The effect of poisoning on discriminator-based methods [8] shown Fig. The domain discriminator used in these methods is maximally confused when marginal distributions of the source and target domains are aligned. However, alignment of the marginal distributions does not ensure alignment of the conditional distributions [36] [8] The objective of achicving

---

(a) MCD trained on clean data
(b) MCD trained on poi-
(c)
SSL trained on clean
(d) SSL trained on poi- soned data
soned data
data
SOURCE
TARGET
POISON
Figure 3: (Best viewed in color.) t-SNE embedding of the data in the representation space (for MNIST + USPS task) learned using MCD and SSL on clean and poisoned source domain data. Without poisoning, correct classes (data from source class 2 is zoomed in) from two domains aligned ((a) and (c)). The presence of poisoned data prevents the methods from aligning correct classes from the two domains ((b) and (d)).
Table Decrease in the target domain accuracy for UDA methods trained on poisoned source domain data (with poisons sampled from source/target domains) compared to accuracy attained with clean data on the Digits tasks ts.d of trials).
Method
Data
SVHN + MNIST
MNIST MNIST_M
MNIST + USPS
USPS + MNIST
Source only
Clean
72.4211.44
39.05:12.30
87.13:11.75
78.62:145
Clean
78.05=1.115
76.224:2.38
92.17:0.73
92.73=0.71
DANN
Puisunlaaree
70.26:1:2.84
69.98:13.449
93.44:10.84
92.08:10.68
Poisotlaagge
Clean
19:0000070
73.88
93.92-+0.97
05.94=20.71
CDAN
Poisonsourse
73.67:24.19
73.36:1
92.061:0.50 0.59
02:85:00.31
Poisotleagge
12.27
0.59.1..11
1.92.10.42
2.96.10.1
Clean
16.1831.553
93.95-+0.33
89,9 99.96-+-2.04
88. 2..34
MCD
Poisonapurcn
85.861:5 66
93.3 93.33:20.77 71
87 17.99+1.00
83.19922.00
PoisoBlarger
0.97.10.94
0.37=10.06
0.66.1..16
2.07
Clean
66.85+22.30
92.76::0.91
69.69:1:1
82. 22.h3:h11 1.59
SSL
Poisonlaures
61.97:1:..22
1.13
85.74
82. 02.56:10.844
Poisotlaagger
low source domain етгоr pushes the source domain classifier to correctly classify the poison data. Thus placing the poison and source data with the same labels close the representation space. Since the poison data is mislabeled target domain data, poisoning makes UDA methods align wrong source and target domain classes. This leads significant decline in the target domain accuracy.
This is also evident from the t-SNE embedding for DANN and CDAN methods in Fig. 2 In absence of poisoning, correct source and target classes are aligned (Fig. 2 (a) and (c)), whereas in presence of poi- soning wrong classes from the two domains are closer (Fig. (b) and (d)). For MCD (27). which uses use classifier discrepancy detect and align source and target domains, our poisoned data prevents the method from detecting target examples. This happens because the term that minimizes the error on the poisoned source domain reduces discrepancy of the classifiers on poison data, which are from target domain. Thus, both the generator (common representation) and discriminator (in the form of two classifiers) become optimal and there signal for the generator to align the two domains. The t-SNE
SOURCE
TARGET
Figure 4: Wrong-label incorrect-domain poisoning causes discriminator-bate UDA approaches to align wrong classes (+ to from the two domains, leading to significant decline in the target domain accuracy.
7

---

Table 2: Decrease in the target domain accuracy for UDA methods trained on poisoned source domain data (with poisons sampled from source/target domains) compared to accuracy attained with clean data on the Office tasks (mean s.d. of trials).
Method
Dataset
A
A W
D- A
D- W
Source Only
Clean
79.61
73.18
59.33
96.31
58.75
99.68
Clean
84.06
85,41
64.67
96.08
66.77
99.44
DANN
Poison.unee
79.11=0.33
83.98=1119
44,31422.94
05.22:00.22
43.35:+1.65
96.88:0.87
PoisoDaages
59.83:0.20
63.18:1996
17.5811.39
76.43=0.62
19.82:10.33
84.20=0.71
Clean
89.56
93.01
71.25
99.24
70.32
100
CDAN
Puison.eures
90.16:10.61
90.94:11.13 1.
53.68:10..7
98 08045
57.27:10.57
99.6610.23
PoisOBrarger
71.88=0.20
7..94=0.76
11.19::1.77
86.37=0.36
18.54:0.45
89.08=1.23
Clean
84.3
86.42
68.38
97.13
67.16
100
IW-DAN
Poison,enrse
81.25=0.91
83.27=0.45
50.76d:1.58
96.68:0.299
48,31+2.02
99.73=0.12
PoisoBraages
61.6410.53
63.43:1.14
15.69.11.76
8..2910.07
26.54:10.48
8..62:0.23
Clean
88,91
93.23
71.9
99.3
70.43
100
IW-CDAN
Poison.unnce
89.83=0.31
10.77#1.27
$7.51+00.0
98.41
61.16:1.21
99.66=0.12
PoisoD
72.62:10442
70.15=2.21
14.36110.66
88.26=0.115
22.36:10.96
07.75=0.55
embedding in Fig. 3 shows this effect. In presence of poisoned data Fig. 3 (b), we see twenty distinct clusters rather than just ten (we have ten classes in the Digits) as seen in the absence of poisoning in B In SSL (35], the generator must work well on the main task, i.e., should correctly classify all data in the poisoned source domain data. The auxiliary task ensures that representations of the source and target domains become similar as seen on clean data (Fig. 3 (c)). But in presence of poisoned data, similar representations of correct source and target domain classes leads a drop in the accuracy of the main task on the poisoned data. This creates a conflict between the main and auxiliary tasks due to which correct source and target domain classes cannot be aligned (Fig. (d)) The objective of making the main task accurate poisoned data leads to target domain data being assigned the labels of the poisoned points. Thereby leading to a significant drop in the target-domain accuracy. The results of wrong-labed wrong- doong-dominn poisoning present in rows marked with Poisontarget in Tables
T and show a significant reduction in the target domain accuracy compared to the accuracy obtained on clean data. On Digits, poisoning makes the target domain accuracy close to 0% on most tasks. On Office-31, poisoning causes at least a 20% reduction in the target domain accuracy in most cases. The reason tasks in Office-31 are less hurt by poisoning is because of the use of a pre- trained representation. Similar to previous works for Office-31, we use all labeled source and unlabeled target domain data for training and fine-tune from representation pre-trained on the ImageNet dataset. The slowly changing representation pre-trained on a massive dataset weakens the effect of poisoning but cannot eliminate it. For the two tasks D - W and W - D, the fine-tuned representation, trained just on the clean source dataset (Source Only in Table 2 achieves high accuracy indicating the domains are already well aligned in terms of conditional distributions as well. As result, UDA methods can casily align correct classes and suffer a drop close to 10% which is the amount of poisoned data added. However, this is not an interesting case for the evaluation of UDA methods domains can be aligned just by training on the source domain data without the need for target domain data.
4.2 Poisoning with mislabeled watermarked data
In this experiment, we evaluate the effect of using poisoned data that looks like the source domain data. The poisoned data is generated by superimposing an image from the source domain with an image from the target domain. This method of generating poison data is known as watermarking 288. To generate watermarked poison data we select an image from the target domain (t) and a base image from the source domain (s) such that it has the same class as the target domain image and lies closest to the target image (in the input space). The poisoned image (p) is obtained by a convex combination of the base and target images i.e.,
SOURCE
TARGET
Figure 5: Successful poi- soning with mislabeled wa- termarked data prevents discriminator-based UDA approaches from aligning correct classes from the source and target domains.
at + (1 a)s where a E [0. 1]. a is selected such that the target image is not visible in the poison image ensuring the poisoned image looks like the image from the source domain. We use the same labeling function as discussed the previous section to label the poisoned image and add 10% poison data to the source.
8

---

Table Decrease in target accuracy when training different domain adaptation methods on poisoned watermarked data in comparison to the target accuracy obtained with clean data on the Digits task t.d. of trials).
Method
Dataset
SVHN - MNIST
MNIST
MNIST M
MNIST + USPS
USPS - MNIST
Cleun
78.05:+1.15
76.2224:238
92.17:10.73
92.73:10.71
DANN
68.76::3.91c.05 57.9625.84p.
27.36:::
91 91.84=+..55.10
88.93-+4.360.10
Poisoned,
7.19:22.5pp.n
85.51=3.01o 20
78.29:18.22..m
33.33:+-..80.15
4.73=#0380..s
39.29=113Mo.
41.52:::.430.30
Cleun
79.19:10.70
73.88:11.10
93.92:10.77
95.94-10.71
CDAN
65.77:t:.826.o0
55,47:+3.88..n
92.05=+0.96, 10
86.53+-1.555m
Poisoned,
$7.57:3.11o..m
1.37:
96.54=2443p...
77.39:14.84p
44.83::4.000..s
6.68=1.640.11
88.67==0444.5r
79.54:2..02..m
Clean
16,18:tt:,53
93.95.40.33
89.96-12.04
88.34442.50
MCD
74.96:1:3220p.mm os
92.18:+0.78o.m
6.75:t-4.81p
30.35:t:.30pp
Poisoned,
35.85.13.23p.m
85.38:13.77..1
0.77::0.22p
11.34::0.77...
17.01=11520.1s
70.34=11.49..s
0.71::0,220.30
3,28:40.440.50
Clean
66.85-42.30
92.76::0.91
88.69:411.28
82.23-41.59
SSL
44.64:+2.06.00
13.33=13.488..c
12.38:::0.77,n
34.72:t:.7lo.10
Poisoned,
10.86=1122o..0
26.64=10..oo.n
6.12+t2. 13..00
1..86::1001l.00
3.411.1lo.1
12.14:14.66y.1
2.42::0.44a.30
11.90:10.81..a
The illustrative picture of the effect of poisoning in this scenario is presented in Fig. 5 Successful poisoning. in this case, works just like in the previous experiment i.e., by making the representations of the data from wrong classes in source and target domains similar for DANN/CDAN, by reducing the discrepancy between the classifiers on target data for MCD and inducing a conflict between the supervised and auxiliary task for SSL. The t-SNE embedding showing the effect of poisoning (Fig. 9 and watermarked poison data (Fig. T0p are shown in the Appendix. We evaluate the effectiveness of this method on the Digits dataset for different values a. The results in Table show significant decrease in the target domain accuracy even with small watermarking percentage for all methods except CDAN. This is because the success of CDAN is dependent on the correctness of the pseudo- labels on the target domain data (output of the classifier), which are used in the discriminator. Correct pseudo-labels provide CDAN positive reinforcement to align correct classes from the two domains, leading to a failure of poisoning. However, as we increase the amount of watermarking, the quality of pseudo labels deteriorates. Thus, providing a negative reinforcement to CDAN that causes the alignment of wrong classes from the two domains.
4.3 Poisoning using clean-label source and target domain data
In this experiment, correctly labeled data is used for poisoning. To generate clean-label poison data that can affect the performance of UDA methods we must affect the features of the poison data. This requires solving bilevel optimization problem IT]
23 24 which we
present in the Appendix E Due to the high computational complexity involved in solving the bilevel problem, we propose to use a simple alternating optimization to demonstrate the feasibility of a clean label poisoning attack against UDA. We use the setting of previous works and consider misclassification of single target domain test point (z target Jarget) rather than affecting the accuracy of the entire target domain as done in the previous two experiments. Let (u...., )n) denote the poisoned data. To ensure a clean label, each poison point u, must have bounded perturbation from a base point w.base i.c, ||u basee = ||o:|| S and has label of the N Npoison Damrec {( yourre ) Nuource and D Drangot = (x ,3/, (arget)) )) The clean-label poison data u is such that when the victim uses Doource Dpoolson and Dtarget for UDA, the target domain test point (I (xrageet
SOURCE TARGET
Figure 6: Successful poi- soning using clcan-label poisoned data aligns the tar- get point (purple) close to the wrong class (-).
11 28
base i.e., pase Thus, Dpoison {(u, (.asee)) (
target
Vtest stagget ) is misclassified.

---

The optimization problem for the clean-label attack is as follows.
Npoteon
min
l
;0) - 0)12 s.l. ( test
sarget,
|| hase
uill Se,
for iori= 1y... N poison:
min LUDA (RiDA(Damuaa (Dsource UDioison Drarges, 8).
0
The first problem minimizes the distance between the representations of the poison and the target domain test data (first term) while ensuring the poison data is not too far from the base data (second
term). The second problem optimizes the parameters of the representation using UDA methods.
Attack success evaluated by solving the second problem in Eq. 4
from scratch and evaluating the classification of Feest larget This
Success Ratm PNi
#DANN COAAN
MCD
#B8L
illustrated in Fig. 6 The left part shows the case before retraining
using the poison data generated from Eq. and the right part shows
how poisoning induces misclassification. We use two approaches for poisoning. The first uses source domain data and the second uses targer domain data as base data. We add 1 1% poisoned data and test the effect of poisoning on a two-class (3 vs 8) domain adaptation problem MNIST - MNIST M (see Appendix H The results in Fig. show that using target domain data as base data significantly more successful under small permissible perturbation (e). Using base data from the source domain requires larger distortion to keep the poison data close to the target point in the representation space and is hence less successful. This shows the feasibility of clean label attacks against UDA methods. We believe the attack success can be improved by solving the bilevel level problem (Eq. in Appendix E and is left future work.
from Base data
Figure 7: Attack success rate of clean-label poisoning using base data from source/target for a two-class problem in MNIST MNIST_M,
Conclusion
We studied the problem of UDA and highlighted the limitations of learning under this setting. We proposed simple lower bound the target domain eπror for UDA, dependent the labeling function induced by the representation. The lower bound demonstrated that lcarning a domain invariant representation while minimizing eπror the source domain cannot guarantee good generalization on the target domain. We analyzed a simple model and showed the existence of cases where UDA can naturally succeed or fail. The analysis also highlighted a case where, without access any labeled target domain data, the success and the failure are equally likely. In such a case, the presence of even a small amount of poisoned data can make the data distribution unfavorable for UDA methods, making them fail dramatically in comparison to the case without poisoning. We proposed novel data poisoning attacks demonstrate the failure of popular UDA methods with small amount of poisoned data. Our results suggest that the performance of a UDA method in presence of poisoned data indicates how well the method aligns the conditional distributions across the two domains. Thus, we believe our attacks can be used for evaluating UDA methods, beyond simple benchmark datasets, reveal their robustness data distributions inherently unfavorable for UDA.
6 Acknowledgment
We thank the anonymous reviewers for their insightful comments and suggestions. This work was supported by the NSF EPSCoR-Louisiana Materials Design Alliance (LAMDA) program #OIA- 1946231 and by LINI. Laboratory Directed Research and Development project 20-ER-014 (LLNI.- CONF-824206). This work was performed under the auspices of the U.S. Department of Energy by the Lawrence Livermore National I Laboratory under Contract No. DE-AC52-00NAA7334 Lawrence Livermore National Security, LLC
"This document was prepared Government. Neither the United States Government nor Lawrence Livermore National Security, LLC, nor any of their employees make any warranty, expressed or implied, or assumes any legal liability or responsibility for the accuracy. completeness, or usefulness of any information, apparatus. product, process disclosed, or represents that its use would not infringe privately owned rights. Reference herein any specific commercial product.
account of the work sponsored
agency of the United States
10

---

References
[1] Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jen- nifer Wortman Vaughan. A theory of learning from different domains. Machine learning. 79(1)1151-175, 2010.
[2] Shai Ben-David, John Blitzer, Koby Crammer, Fernando Percira, et al. Analysis of represen- tations for domain adaptation. Advances in neural information processing systems, 19:137, 2007.
[3] Shai Ben-David and Ruth Urner. On the hardness of domain adaptation and the utility of unlabeled target samples. In International Conference on Algorithmic Learning Theory, pages 139-153. Springer, 2012.
[4] Battista Biggio, Blaine Nelson, and Pavel Laskov. Poisoning attacks against support vector machines. arXiv preprint DrXiv::1200 6389, 2012.
[5] Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning arXiv preprint arXiv:1712.05522 2017.
[6] Remi Tachet des Combes, Han Zhao. Yu-Xiang Wang. and Geoff Gordon. Domain adap- tation with conditional distribution matching and generalized label shift. arXiv preprint MrXiv:2003.0447: 2020.
[7] Nicolas Courty, Rémi Flamary, Amaury Habrard, and Alain Rakotomamoniy. Joint distribution optimal transportation for domain adaptation. arXiv preprint arXiv: /705.08848 2017.
[8] Shai Ben David, Tyler Lu, Teresa Luu, and Dávid Pál. Impossibility theorems for domain adaptation. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pages 129-136. JMLR Workshop and Conference Proceedings, 2010.
[9] Yaroslav Ganin, Evgeniya Ustinova, Hama Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-aavversrral training of neural networks. The journal of machine learning research, 17(1):2096 2030, 2016.
[10] Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. In International conference on machine learning, pages 1999-1998. PMLR, 2018.
[11] W Ronny Huang, Jonas Geiping, Liam Fowl, Gavin Taylor, and Tom Goldstein. Metapoison: Practical general-purpose clean-label data poisoning. arXiv preprint (tt::2004400222,. 2020.
[12] Matthew Jagielski, Alina Oprea, Battista Biggio, Chang Liu, Cristina Nita-Rotaru, and Bo Li. Manipulating machine learning: Poisoning attacks and countermeasures for regression learning. In 2018 IEEE Symposium on Security and Privacy (SP). pages 19-35. IEEE, 2018
[13] Yujie Ji, Xinyang Zhang, and Ting Wang. Backdoor attacks against learning systems. In 2017 IEEE Conference on Communications and Network Security (CNS), pages 1-9. IEEE, 2017.
[14] Fredrik D Johansson, David Sontag, and Rajesh Ranganath. Support and invertibility in domain- invariant representations. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 527-536 PMLR, 2019.
[15] Jaeho Lee and Maxim Raginsky. Minimax statistical learning with wasserstein distances. arXiv preprint arXiv: 1exxvv:775...m 1705 077 2017.
[16] Hong Liu, Mingsheng Long, Jianmin Wang, and Michael Jordan. Transferable adversarial training: A general approach to adapting deep classifiers. In International Conference on Machine Learning, pages 4013-4022. PMLR, 2019.
process, service by trade name, trademark, manufacturer, otherwise does not necessarily constitute imply its endorsement, recommendation, or favoring by the United States Govermment or Lawrence Livermore National Security, LLC. The views and opinions of the authors expressed herein do not necessarily state or reflect those of the United States Government or Lawrence Livermore National Security. LLC, and shall not be used for advertising product endorsement purposes.

---

[17] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In International conference on machine learning, pages 97-105. PMLR, 2015.
[18] Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I Jordan. Conditional adversarial domain adaptation. arXiv preprint arXiv: 10667, 2017.
[19] Mingsheng Long, Jianmin Wang, Guiguang Ding, Jiaguang Sun, and Philip Yu. Transfer joint matching for unsupervised domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1410-1417, 2014.
[20] Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I Jordan. Unsupervised domain adaptation with residual transfer networks. arXiv preprint vxxiiv:602..444 2016.
[21] Yishay Mansour, Mehryar Mohri, and Afshin Rostamizadeh. Domain adaptation: Learning bounds and algorithms. arXiv preprint arXiv:0902.3430, 2009.
[22] Yishay Mansour and Mariano Schain. Robust domain adaptation. Annals of Mathematics and Artificial Intelligence, 71(4):365-38) 2014.
[23] Akshay Mehra and Jihun Hamm. Penalty method for inversion-free deep bilevel optimization. arXiv preprint arXiv: 1.03432. 2019.
[24] Akshay Mehra, Bhavya Kailkhura, Pin- Yu Chen, and Jihun Hamm. How robust are randomized smoothing based defenses to data poisoning? arXiv preprint arXiv:2012.0/274, 2020.
[25] Shike Mei and Xiaojin Zhu. Using machine teaching to identify optimal training-sct set attacks on machine learners. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.
(26) Luis Muñoz-González, Battista Biggio, Ambra Demontis, Andrea Paudice, Vasin Wongras- samee, Emil C Lupu, and Fabio Roli. Towards poisoning of deep learning algorithms with back-gradient optimization. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, pages 27-38, 2017.
[27] Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier discrepancy for unsupervised domain adaptation. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3723-3732, 2018
[28] Ali Shafahi, W Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Du- mitras, and Tom Goldstein. Poison frogs! targeted clean-label poisoning attacks on neural networks. arXiv preprint weXiv:884.000992 2018.
[29] Jian Shen, Yanru Qu, Weinan Zhang, and Yong Yu. Wasserstein distance guided representation learning for domain adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018
[30] Rui Shu, Hung Bui, Hirokazu Narui, and Stefano Ermon. dirt-t approach to unsupervised domain adaptation. arXiv preprint arXiv:1802.088 08735, 2018.
[31] Alexander Turner, Dimitris Tsipras, and Aleksander Madry. Clean-label backdoor attacks. 2018.
[32] Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 7167-7176, 2017.
[33] Zirui Wang, Zihang Dai, Barnabás Póczos, and Jaime Carbonell. Characterizing and avoiding negative transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11293-11302 2019.
[34] Xi Wu, Yang Guo, Jiefeng Chen, Yingyu Liang, Somesh Jha, and Prasad Chalasani. Repre- sentation bayesian risk decompositions and multi-source domain adaptation. arXiv preprint 2XXi:2004.10390 2020.
12

---

[35] Jiaolong Xu, Liang Xiao, and Antonio M López. Self-supervised domain adaptation for computer vision tasks. IEEE Access, 7:156694- 2019.
[36] Han Zhao, Remi Tachet Des Combes, Kun Zhang, and Geoffrey Gordon. On learning invariant representations for domain adaptation. In International Conference on Machine Learning, pages 7523-75322 PMLR, 2019.
[37] Han Zhao, Shanghang Zhang. Guanhang Wu, José MP Moura, Joao Costeira, and Geoffrey Gordon. Adversarial multiple source domain adaptation. Advances in neural information processing systems, 31:8559-8570, 2018.
[38] Chen Zhu, W Ronny Huang, Ali Shafahi, Hengduo Li, Gavin Taylor, Christoph Studer, and Tom Goldstein. Transferable clean-label poisoning attacks on deep neural nets. arXiv preprint 1xXiv:1905.05887, 2019.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See Sec. and 4
(b) Did you describe the limitations of your work? [Yes] See Sec. 3
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Sec. 3
and
Appendix A
(b) Did you include complete proofs of all theoretical results? [Yes] See the Appendix.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material URL)? [Yes] See Appendix
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] See Appendix H
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Appendix H
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "lI_SWTPH076", "reviewer": "Reviewer_vTUY", "summary": "The paper addresses the vulnerability of domain adaptation to adversarial attacks. First, a lower bound on the target risk of a given classifier is derived, which provides necessary conditions for the success of domain adaptation methods. Then, 3 examples on 2D data are given and analyzed to show the potential failure of DA methods aiming at aligning both domains while remaining good for classification on the source one. The last example describes an ambiguous situation for which success and failure of adaptation are equally likely, from which the authors have the inspiration that adding a few poison data can easily favor the bad solution. They concretize this idea by proposing 3 data poisoning strategies and they show their ability to deteriorate the performance of previously proposed DA methods.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is clearly written and easy to follow, and the theoretical technical results are supported by proofs.\n\n## Strengths\n* The overall idea of showing the vulnerability of DA methods to adversarial attacks is interesting. It illustrates the intuition that the complete absence of target data is a considerable lack of information for DA approaches methods. Also, the provided empirical evidence for this vulnerability concerns even new DA approaches that aim to improve previous ones (IW-DAN and IW-CDAN).\n* Corollary 3 is very insightful as it offers more precision on which quantity is being minimized by DA methods that try to have a good performance on the source domain while embedding the source and target domain into a representation space in which they are aligned.\n\n## Weaknesses\n* About lower bounds on the ideal joint error: in line 39, it is argued that a large upper bound does not guarantee the failure of DA methods. However, any lower bound on the ideal joint hypothesis is equivalent to a lower bound on the target risk, due to the equivalence between \n$\\min_{h \\in H}e_S(h) + e_T(h) \\geq B$ and $\\forall h \\in H, e_T(h) \\geq B - e_S(h)$, where $B$ denotes some lower bound.\n* Lack of comparison to previous work on the drop of performance on the target domain due to the deep embedding of both domains\n  * Comparison to the lower bound established in Theorem 4.3 of  [1]: using the notation of the paper being reviewed, their result states that for any $h \\in H, g \\in G$, $e_T(h) + e_S(h)\\geq\\frac{1}{2}(d_{JS}(\\tilde p_S, \\tilde p_T)-d_{JS}(p_S^Y, p_T^Y))^2$ (where $p_S^Y$ and $p_T^Y$ denote the label marginal distributions), whereas Theorem 1 of the reviewed paper states that $e_{T}(h) + e_{S}(h) \\geq \\max (e_S(\\tilde{f}_S, \\tilde{f}_T), e_T(\\tilde{f}_S, \\tilde{f}_T)) - D_1(\\tilde{p}_S, \\tilde{p}_T)$. The r.h.s. of the former inequality is always nonnegative, which is not the case for the latter. This comparison should be carried out in detail.\n  * Another contribution concerned with the performance drop on the target domain after the embedding in the new representation space is [2]. In particular, their Example 1 illustrates a case with two global solutions to the classic DA minimization problem, which is the same spirit of Case 3 of Figure 1 of the paper under review.\n\n\n## Suggestions\n* Detailed comparison to [1,2].\n* Plotting  2D map of the effect of both the percentage of poisonous data and the divergence between domains.\n\n## Comments\n* In line 232,  the \"generator\" term should be replaced with \"source classifier\" for example. Indeed, in spite of the similarity in mechanism between GANs and adversarial domain adaptation methods, they are not the same. This holds also for line 242\n* Having seen Figure 1, in which the data is represented in a 2D space, I was confused when looking at Figure 3 for a moment. Indeed, it seems like the target classes were flipped from left to right, whereas the arrow represents an alignment in a new representation. In this case, another schematic representation for the data in the new representation space should be used.\n* I do not see the interest of Corollary 2 concerning the KL-divergence, as it states a looser lower bound than the one of Theorem 1 (as $-D_1(\\cdot,\\cdot) \\geq -\\sqrt{\\frac{1}{2}D_{KL}(\\cdot\\Vert \\cdot)}$), and it does not have other interests shown in the paper, besides involving another divergence.\n### Minor comments\n\n* In lines 168 and 170, it should be $e(\\tilde f_S, \\tilde f_T)$ instead of $e(\\tilde f_S, \\tilde f_S)$.\n* Line 252: \"poisoned data, leads\" --> \"poisoned data leads\"\n* Line 253: \". There by leading \" --> \", thereby leading\"\n* Line 288: \"Fig. 9 ... Fig. 8\" ---> \"Fig. 10 ... Fig. 9\"\n* Line 338: \"showe\" --> \"show\"\n* Line 340: \"presence of even small\" --> \"the presence of even a small\"\n\n## Questions\n* In the third case of examples in section 3.2, the classifier is initialized randomly, and this randomness causes the optimization to converge to either a good or bad solution. Do you have an idea about the initial conditions of the solution that lead to one solution or the other? (e.g. a partition of two regions in the solution space according to which the optimization results in one of the two minimizers).\n\n#  References\n\n[1] Zhao, H., Combes, R. T. D., Zhang, K., & Gordon, G. (2019). On Learning Invariant Representations for Domain Adaptation. *International Conference on Machine Learning*, 7523–7532.\n\n[2] Johansson, F. D., Sontag, D., & Ranganath, R. (2019). Support and Invertibility in Domain-Invariant Representations. *The 22nd International Conference on Artificial Intelligence and Statistics*, 527–536.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Although the societal impact was not directly addressed in the paper, the aspects of the adversarial attack it tackles are vital in real-world applications. For example, in autonomous driving, it can be very problematic for an algorithm to adapt to new environments if it is not robust to data poisoning.", "review_text": "", "extra_scores": {"ethical_concerns": ".", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "ZTr2sPh0ghX", "reviewer": "Reviewer_Hs6Q", "summary": "This paper aims to understand the limitation of unsupervised domain adaptation (UDA) through data poisoning. The paper reveals that the modern UDA deep learning approach is vulnerable to noise data, with almost zero accuracies in the poisoning target samples.\n\n-----------Updates after Discussion\n\nI appreciate the efforts and additional experimental results by the authors. Thus I recommend a (weakly) positive score. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Pros:\n- This paper reveals an important and interesting fact that the modern UDA approach is quite vulnerable in the noisy datasets (particularly the poisoned target samples), which restricts its real practical utility.\n\n### Cons:\n- Despite the empirical discovery being interesting and promising, the current version seems preliminary and the in-depth analysis is lacking.\n- The proposed theoretical result is insufficient to understand the inherent limitation. \n\nOverall it is a good paper in the proof of concept but the main discovery is not surprising. Moreover, the in-depth analysis/justification/proof is not sufficient. Thus, I recommend a weak rejection but encourage a major revision for future submission.\n\n\n### Comments:\nThis paper has several discoverings, the following are the comments for each contribution.\n\n- “a lower bound on the target domain error providing a necessary condition for the success of learning in a UDA setting. The bound shows the failure of learning a domain invariant representation that minimizes source domain error at guaranteeing target generalization”.\n\nIn fact, this discovery is not surprising and several recent works have a good discussion on this idea. e.g ref [1,2], which is missing from the author. Besides, the theoretical proof is only valid for the bijective or invertible representation function (see Line 83 the definition). In fact, the most present deep UDA approach ** is not satisfied** this assumption. That being said, theory and practice have several non-ignorable gaps. \n\nAs for the conclusion of Theorem 1, the results do not provide novel insights, the lower bound can be vacuous in the presence of noisy labels. Supposing we have a MLP network the noisy source error is $e_s(h)>0$, then the lower bound can be still zero. From this perspective, the lower bound has some issues.\n\nLine 40, the author states that “a large upper bound does not guarantee failure of UDA methods.“ I agree with this point. However, the upper bound is introduced as **sufficient condition** to ensure it works. Since the UDA only works in very specific scenarios, in the practical algorithm we need to justify the sufficient condition rather than the necessary assumption.\n\nAnother important concern of the proposed theoretical analysis is that the data generation procedure should be stochastic rather than a deterministic labeling function $f$. This is particularly important in this paper. The later experimental settings are exactly the stochastic setting: the same digit input $x$ can have two different values with 0 and 1. In contrast, the deterministic setting has no random sampling procedure (i,e for the same X, f(x) is fixed), which clearly violates the empirical settings.\n\n- “ present example distributions where UDA succeeds, fails, and where success and failure are equally likely. The last case opens door to adversarial attacks such as data poisoning.”\n\nIn fact, the first two counterexamples are essentially the same as the ones in Ref [1]. Thus they are not surprising. As for the last counterexamples, the problem lies in the inherent property in UDA and is impossible to solve without additional assumptions/conditions. The authors are strongly encouraged to discover these conditions and justify them. \n\n\n- “several novel data poisoning attacks….The poisoning attacks can be used to obtain a better sense of the robustness of the future UDA methods beyond simple benchmark datasets”\n\nIn general, this part is quite interesting and deserves further attention in the community. And I have the following question:\n\nIf we added the target sample noise to the source, the UDA-based approach will tend to find a similar point to learn the target. Naturally, the performance will have a significant drop in the shallow network since it is almost the worst-case attack. My concern is in the worst case, no algorithm can probably work. Thus it is an impossible task? \n\nAs far as I understand, the target sample attack is believed to be the most interesting and promising direction. (other types of attacks are quite common and fairly easy to solve by adopting the existing approaches) Thus I do think an additional discussion about the assumption, the condition is important for the future version.\n\n### Other Comments \n\n- The writing and organization can be improved. Sec 3.2 has only one paragraph and is relatively difficult to read.\n- The performance drop in the digits and large dataset (e,g office) have quite different behavior. An in-depth analysis is expected.\n- Line 142-144, the overparameterized network does not necessarily guarantee an expected term as zero. A rigorous justification is expected.\n\nReference \n[1] Support and Invertibility in Domain-Invariant Representations.\n\n[2] Representation bayesian risk decompositions and multi-source domain adaptation\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- The authors have adequately addressed the limitations and potential negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "-SNKCHYRYPY", "reviewer": "Reviewer_ErCq", "summary": "In this work, the authors aim to study the limitations of Unsupervised Domain Adaptation through data poisoning. Specifically, the authors first show the failure of UDA by proving a simple lower bound, then present example distributions where UDA succeeds or fails. Finally, they propose several novel data poisoning attacks that use clean-label and mislabeled points to fail existing UDA methods and verify these attack methods in the experiments on two benchmark datasets.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:  \nAs far as I’m aware, this work is the first to explore the limitations of UDA methods from the lens of robustness. I believe this work could provide new insight for the community of UDA.\nThe code is available with the submission which improves the reproducibility index of the paper.\nOverall, this paper is well-written and easy-to-follow. \n\nCons:\n\nThe relationship between data poisoning and the natural failure of UDA methods is not clear. From the view of robustness, supervised learning methods are still vulnerable under adversarial attacks like data poisoning, even though there are enough labeled examples from the target domain. It is not surprising that UDA methods are also vulnerable to data poisoning. Therefore, I am not sure that the fact that UDA methods are easy to be attacked can demonstrate the natural failure problem of UDA methods. If the authors could provide a convincing explanation during rebuttal, I will consider improving my score on this work.\n\n\n---\n\nThe authors have addressed my concerns during rebuttal, however, the writing in the manuscript is still problematic. Thus I decide to improve my score to 6 here. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors claim that they have addressed the limitations of this work in Section 3 but I cannot find it. Indeed, they introduce two potential future directions, like extending the simple lower bound to different divergence metrics and boosting the poisoning attack by solving the bilevel problem.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the vulnerability of domain adaptation to adversarial attacks. The key idea is to show the failure of UDA by proving a simple lower bound, then present example distributions where UDA succeeds or fails. This paper proposes several novel data poisoning attacks that use clean-label and mislabeled points to fail existing UDA methods and verify these attack methods in the experiments on two benchmark datasets.

However, there exists some limitations as follows.

1) In-depth analysis: despite the empirical discovery being interesting and promising, the current version seems preliminary and the in-depth analysis is lacking.

2) Insufficient theory: the proposed theoretical result is insufficient to understand the inherent limitation.

3) Writing issues: the writing in the manuscript is still problematic.

This paper is a boardline case according to the average rating. While the reviewers had some concerns on the significance, the authors did a particularly good job in their rebuttal. Thus, all of us have agreed to marginally accept this paper for publication! Please include the additional experimental results in the next version.
Decision: Accept (Poster)