Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper addresses the vulnerability of domain adaptation to adversarial attacks. First, a lower bound on the target risk of a given classifier is derived, which provides necessary conditions for the success of domain adaptation methods. Then, 3 examples on 2D data are given and analyzed to show the potential failure of DA methods aiming at aligning both domains while remaining good for classification on the source one. The last example describes an ambiguous situation for which success and failure of adaptation are equally likely, from which the authors have the inspiration that adding a few poison data can easily favor the bad solution. They concretize this idea by proposing 3 data poisoning strategies and they show their ability to deteriorate the performance of previously proposed DA methods.
main_review: The paper is clearly written and easy to follow, and the theoretical technical results are supported by proofs.

## Strengths
* The overall idea of showing the vulnerability of DA methods to adversarial attacks is interesting. It illustrates the intuition that the complete absence of target data is a considerable lack of information for DA approaches methods. Also, the provided empirical evidence for this vulnerability concerns even new DA approaches that aim to improve previous ones (IW-DAN and IW-CDAN).
* Corollary 3 is very insightful as it offers more precision on which quantity is being minimized by DA methods that try to have a good performance on the source domain while embedding the source and target domain into a representation space in which they are aligned.

## Weaknesses
* About lower bounds on the ideal joint error: in line 39, it is argued that a large upper bound does not guarantee the failure of DA methods. However, any lower bound on the ideal joint hypothesis is equivalent to a lower bound on the target risk, due to the equivalence between 
$\min_{h \in H}e_S(h) + e_T(h) \geq B$ and $\forall h \in H, e_T(h) \geq B - e_S(h)$, where $B$ denotes some lower bound.
* Lack of comparison to previous work on the drop of performance on the target domain due to the deep embedding of both domains
  * Comparison to the lower bound established in Theorem 4.3 of  [1]: using the notation of the paper being reviewed, their result states that for any $h \in H, g \in G$, $e_T(h) + e_S(h)\geq\frac{1}{2}(d_{JS}(\tilde p_S, \tilde p_T)-d_{JS}(p_S^Y, p_T^Y))^2$ (where $p_S^Y$ and $p_T^Y$ denote the label marginal distributions), whereas Theorem 1 of the reviewed paper states that $e_{T}(h) + e_{S}(h) \geq \max (e_S(\tilde{f}_S, \tilde{f}_T), e_T(\tilde{f}_S, \tilde{f}_T)) - D_1(\tilde{p}_S, \tilde{p}_T)$. The r.h.s. of the former inequality is always nonnegative, which is not the case for the latter. This comparison should be carried out in detail.
  * Another contribution concerned with the performance drop on the target domain after the embedding in the new representation space is [2]. In particular, their Example 1 illustrates a case with two global solutions to the classic DA minimization problem, which is the same spirit of Case 3 of Figure 1 of the paper under review.


## Suggestions
* Detailed comparison to [1,2].
* Plotting  2D map of the effect of both the percentage of poisonous data and the divergence between domains.

## Comments
* In line 232,  the "generator" term should be replaced with "source classifier" for example. Indeed, in spite of the similarity in mechanism between GANs and adversarial domain adaptation methods, they are not the same. This holds also for line 242
* Having seen Figure 1, in which the data is represented in a 2D space, I was confused when looking at Figure 3 for a moment. Indeed, it seems like the target classes were flipped from left to right, whereas the arrow represents an alignment in a new representation. In this case, another schematic representation for the data in the new representation space should be used.
* I do not see the interest of Corollary 2 concerning the KL-divergence, as it states a looser lower bound than the one of Theorem 1 (as $-D_1(\cdot,\cdot) \geq -\sqrt{\frac{1}{2}D_{KL}(\cdot\Vert \cdot)}$), and it does not have other interests shown in the paper, besides involving another divergence.
### Minor comments

* In lines 168 and 170, it should be $e(\tilde f_S, \tilde f_T)$ instead of $e(\tilde f_S, \tilde f_S)$.
* Line 252: "poisoned data, leads" --> "poisoned data leads"
* Line 253: ". There by leading " --> ", thereby leading"
* Line 288: "Fig. 9 ... Fig. 8" ---> "Fig. 10 ... Fig. 9"
* Line 338: "showe" --> "show"
* Line 340: "presence of even small" --> "the presence of even a small"

## Questions
* In the third case of examples in section 3.2, the classifier is initialized randomly, and this randomness causes the optimization to converge to either a good or bad solution. Do you have an idea about the initial conditions of the solution that lead to one solution or the other? (e.g. a partition of two regions in the solution space according to which the optimization results in one of the two minimizers).

#  References

[1] Zhao, H., Combes, R. T. D., Zhang, K., & Gordon, G. (2019). On Learning Invariant Representations for Domain Adaptation. *International Conference on Machine Learning*, 7523–7532.

[2] Johansson, F. D., Sontag, D., & Ranganath, R. (2019). Support and Invertibility in Domain-Invariant Representations. *The 22nd International Conference on Artificial Intelligence and Statistics*, 527–536.
limitations_and_societal_impact: Although the societal impact was not directly addressed in the paper, the aspects of the adversarial attack it tackles are vital in real-world applications. For example, in autonomous driving, it can be very problematic for an algorithm to adapt to new environments if it is not robust to data poisoning.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In this work, the authors aim to study the limitations of Unsupervised Domain Adaptation through data poisoning. Specifically, the authors first show the failure of UDA by proving a simple lower bound, then present example distributions where UDA succeeds or fails. Finally, they propose several novel data poisoning attacks that use clean-label and mislabeled points to fail existing UDA methods and verify these attack methods in the experiments on two benchmark datasets.
main_review: Pros:  
As far as I’m aware, this work is the first to explore the limitations of UDA methods from the lens of robustness. I believe this work could provide new insight for the community of UDA.
The code is available with the submission which improves the reproducibility index of the paper.
Overall, this paper is well-written and easy-to-follow. 

Cons:

The relationship between data poisoning and the natural failure of UDA methods is not clear. From the view of robustness, supervised learning methods are still vulnerable under adversarial attacks like data poisoning, even though there are enough labeled examples from the target domain. It is not surprising that UDA methods are also vulnerable to data poisoning. Therefore, I am not sure that the fact that UDA methods are easy to be attacked can demonstrate the natural failure problem of UDA methods. If the authors could provide a convincing explanation during rebuttal, I will consider improving my score on this work.


---

The authors have addressed my concerns during rebuttal, however, the writing in the manuscript is still problematic. Thus I decide to improve my score to 6 here. 
limitations_and_societal_impact: The authors claim that they have addressed the limitations of this work in Section 3 but I cannot find it. Indeed, they introduce two potential future directions, like extending the simple lower bound to different divergence metrics and boosting the poisoning attack by solving the bilevel problem.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper aims to understand the limitation of unsupervised domain adaptation (UDA) through data poisoning. The paper reveals that the modern UDA deep learning approach is vulnerable to noise data, with almost zero accuracies in the poisoning target samples.

-----------Updates after Discussion

I appreciate the efforts and additional experimental results by the authors. Thus I recommend a (weakly) positive score. 
main_review: ### Pros:
- This paper reveals an important and interesting fact that the modern UDA approach is quite vulnerable in the noisy datasets (particularly the poisoned target samples), which restricts its real practical utility.

### Cons:
- Despite the empirical discovery being interesting and promising, the current version seems preliminary and the in-depth analysis is lacking.
- The proposed theoretical result is insufficient to understand the inherent limitation. 

Overall it is a good paper in the proof of concept but the main discovery is not surprising. Moreover, the in-depth analysis/justification/proof is not sufficient. Thus, I recommend a weak rejection but encourage a major revision for future submission.


### Comments:
This paper has several discoverings, the following are the comments for each contribution.

- “a lower bound on the target domain error providing a necessary condition for the success of learning in a UDA setting. The bound shows the failure of learning a domain invariant representation that minimizes source domain error at guaranteeing target generalization”.

In fact, this discovery is not surprising and several recent works have a good discussion on this idea. e.g ref [1,2], which is missing from the author. Besides, the theoretical proof is only valid for the bijective or invertible representation function (see Line 83 the definition). In fact, the most present deep UDA approach ** is not satisfied** this assumption. That being said, theory and practice have several non-ignorable gaps. 

As for the conclusion of Theorem 1, the results do not provide novel insights, the lower bound can be vacuous in the presence of noisy labels. Supposing we have a MLP network the noisy source error is $e_s(h)>0$, then the lower bound can be still zero. From this perspective, the lower bound has some issues.

Line 40, the author states that “a large upper bound does not guarantee failure of UDA methods.“ I agree with this point. However, the upper bound is introduced as **sufficient condition** to ensure it works. Since the UDA only works in very specific scenarios, in the practical algorithm we need to justify the sufficient condition rather than the necessary assumption.

Another important concern of the proposed theoretical analysis is that the data generation procedure should be stochastic rather than a deterministic labeling function $f$. This is particularly important in this paper. The later experimental settings are exactly the stochastic setting: the same digit input $x$ can have two different values with 0 and 1. In contrast, the deterministic setting has no random sampling procedure (i,e for the same X, f(x) is fixed), which clearly violates the empirical settings.

- “ present example distributions where UDA succeeds, fails, and where success and failure are equally likely. The last case opens door to adversarial attacks such as data poisoning.”

In fact, the first two counterexamples are essentially the same as the ones in Ref [1]. Thus they are not surprising. As for the last counterexamples, the problem lies in the inherent property in UDA and is impossible to solve without additional assumptions/conditions. The authors are strongly encouraged to discover these conditions and justify them. 


- “several novel data poisoning attacks….The poisoning attacks can be used to obtain a better sense of the robustness of the future UDA methods beyond simple benchmark datasets”

In general, this part is quite interesting and deserves further attention in the community. And I have the following question:

If we added the target sample noise to the source, the UDA-based approach will tend to find a similar point to learn the target. Naturally, the performance will have a significant drop in the shallow network since it is almost the worst-case attack. My concern is in the worst case, no algorithm can probably work. Thus it is an impossible task? 

As far as I understand, the target sample attack is believed to be the most interesting and promising direction. (other types of attacks are quite common and fairly easy to solve by adopting the existing approaches) Thus I do think an additional discussion about the assumption, the condition is important for the future version.

### Other Comments 

- The writing and organization can be improved. Sec 3.2 has only one paragraph and is relatively difficult to read.
- The performance drop in the digits and large dataset (e,g office) have quite different behavior. An in-depth analysis is expected.
- Line 142-144, the overparameterized network does not necessarily guarantee an expected term as zero. A rigorous justification is expected.

Reference 
[1] Support and Invertibility in Domain-Invariant Representations.

[2] Representation bayesian risk decompositions and multi-source domain adaptation


limitations_and_societal_impact: - The authors have adequately addressed the limitations and potential negative societal impact.
review_title: null

Meta-review: This paper studies the vulnerability of domain adaptation to adversarial attacks. The key idea is to show the failure of UDA by proving a simple lower bound, then present example distributions where UDA succeeds or fails. This paper proposes several novel data poisoning attacks that use clean-label and mislabeled points to fail existing UDA methods and verify these attack methods in the experiments on two benchmark datasets.

However, there exists some limitations as follows.

1) In-depth analysis: despite the empirical discovery being interesting and promising, the current version seems preliminary and the in-depth analysis is lacking.

2) Insufficient theory: the proposed theoretical result is insufficient to understand the inherent limitation.

3) Writing issues: the writing in the manuscript is still problematic.

This paper is a boardline case according to the average rating. While the reviewers had some concerns on the significance, the authors did a particularly good job in their rebuttal. Thus, all of us have agreed to marginally accept this paper for publication! Please include the additional experimental results in the next version.
Decision: Accept (Poster)