Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a type of data poison attack called adversarially inducing noise (ADVIN) that degrades the performance of adversarial training. It shows the necessity of considering the adversarial learning in the poison process and the experiments show that ADVIN can successfully make adversarial training ineffective.


main_review: ## Strengths
1. The paper studies the failure of the existing method, for instance, how the data poison attack proposed in Huang et al. (2021) fails in the adversarial setting. This motivates a stronger type of attack that can fool adversarially learned models.

2. The paper is clearly written. The logic and content are well organized.

3. The paper presents many experiments to show the effectiveness of fooling adversarial training.

## Weaknesses
1. The findings that existing data poison attacks do not work well in adversarial training are quite trivial since those attacks focus on the normal setting but might not consider the existence of adversarial training. For instance, the error-minimization-noise in Huang et al. (2021) generates noise based on normally trained models and therefore it is very natural that it doesn't fool adversarial training. Therefore, such findings are not surprising.

2. The novelty of the proposed poison algorithm is weak. The major idea in this paper follows the error-minimization-noise in Huang et al. (2021), but it considers updating the model based on the adversarial loss defined on the adversarial samples with poison noise instead of on the clean data with noise. Although this paper proposes other techniques for improvements such as the consistent label bias, overall, the novelty is quite weak.

3. The paper doesn't discuss whether existing defense methods against data poison attacks can defend the proposed attack. Although the poison data looks visually similar to the original data, it is still possible to detect or remove such attacks. More discussion on this will be helpful.

4. The data poison effect is not that strong as shown in the main paper. If I understand correctly, in most of the experiments in the paper, the poison ratio is 100%. In practice, it is unlikely that we can control and poison the whole dataset. However, Table 8 shows that when the poison ratio is lower, the poison effect is much weaker. For instance, when 20% / 40% / 60% of the data samples are poisoned, the robust accuracy only decreases by 0% / 3% / 7%. Therefore, this suggests the proposed attack might not be very strong in practice. 


summary_of_the_review: The paper proposes a data poison attack to fool the adversarial training algorithms. The novelty of the proposed method is weak and the attack seems to be weak when the poison ratio is lower. I would like to recommend rejection.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work tackles the problem of data poisoning adversarial training in the (image classification) setting where the adversary can add $\ell_p$ constrained perturbations to each training example. Others have previously studied this problem in the standard ERM setting by adding $\ell_p$ perturbations to examples that minimize loss on pretrained models: when the downstream learner tries to minimize risk on these perturbed examples, they can do well on the train set but not on a held-out set from the original distribution (this work is all performed in CIFAR). It turns out that adversarially trained models can largely circumvent this class of attacks, as they can modify the adversarial $\ell_p$ perturbations in the adversarial training process to make learning require appropriately generalizable features. 

The authors try a new attack that focusses on adding "robust" features to examples: in the most basic attack, for each (example, label) combination the authors perturb the example to maximize the probability of a different (label-consistent --- i.e. each image of a given label is perturbed towards the same class, one such mapping could be target_label = label + 1 % num_classes ) label. The authors specialize the attack further by performing an alternating optimization routine in which one repeatedly (starting from a pretrained adv-trained model $f$): (a) completes the perturbation above on the examples using $f$ and then (b) adversarially trains on $f$ using the new label scheme. Using this technique the authors can force the model to perform well on the train set, but not on a held-out set. Using only step (a) (i.e. no alternating optimization routine) the authors can fool the model in the same sense but to a lesser extent.
main_review: The work is well written. It presents an effective approach to performing such poisoning; however, the work does not sufficiently explore the reasons for the success of the technique (ADVIN) or compare it to appropriate baselines. The work could also improve clarity in some areas.

Fleshing out technique/baselines: ADVIN, the main presented technique of the work, is not explained or motivated sufficiently. It is difficult to make sense of why this technique works, or even why it should work to begin with---I would have thought that changing the model objective so much would make the network relatively useless when poisoning new adversarially trained models. It would be good to start out with simple extensions of the AT-Pretrained method first: what happens when you ensemble and do the AT-Pretrained method, or adversarially train for the standard (i.e. proper labels with their corresponding perturbed $x$ images) objective in the loop?

Clarity: 
- It would be good to expand on Huang et al 2021 around equation (3) for greater context on methods used in this area (especially considering that the techniques in the work heavily build upon Huang et al 2021).
- Minor point: the title of the forum page for this paper does not match the title of the pdf of this paper.
summary_of_the_review: The work presents a new technique that effectively poisons adversarially trained models at train time. However, the authors do not properly motivate the technique, the baselines are inadequate, and the paper can be sometimes unclear. For these reasons I give a score of weak reject.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to reduce the adversarial robustness of adversarial training by perturbing the training data with large perturbation budget. Various strategies for assigning fooling labels are explored, which eventually leads to a strong poisoning attack named IAT. Extensive experiments show the effectiveness of the proposed poisoning algorithm.
main_review: Pros:

- This paper is the first to study the problem of poisoning the performance of adversarial training.
- The proposed poisoning attack can successfully make adversarial training with small $\epsilon$ ineffective.

Cons:

- This paper lacks an adequate discussion/reference of a few closely related works, including [1], [2], [3], [4], and [5]. In short, they all studied the threat that aims to degrade test accuracy by perturbing training data.
- The main drawback of this paper is that using such a large perturbation range $\epsilon_p=32/255$ for generating poisons may be not so reasonable. Actually, The commonly used $\ell_{\infty}$ perturbation budget in the literature is 0.3 for MNIST and 8/255 for CIFAR-10, SVHN, and CIFAR-100. However, it has been shown in [6] that perturbations with a large budget for MNIST may modify an input’s true label. Though [6] only conduct experiments on MNIST, the same conclusion may be true for other commonly used datasets. The few examples in Figure 7 are not enough to support the claim "it will not affect the semantics of the raw images".
- In section 4.1, could you tell me the budget of training perturbations for adversarial training?
- It may not be suitable to claim the success of the poisons when the the budget of training perturbations $\epsilon_t=8/255$ is much small than the poison budget $\epsilon_p=32/255$, since adversarial training needs a large $\epsilon$ to guarantee its performance (cf. Theorem 1 in [2]).
- As a reminder, [6] was the first to point out that adversarial example noise can be used as poisons. Later, this idea was experimented in [2] and [7].
- As a reminder, on Page 10, the reference to MART appears twice.

[1] Feng et al., Learning to Confuse: Generating Training Time Adversarial Data with Auto-Encoder, NeurIPS 2019  
[2] Tao et al., Better Safe Than Sorry: Preventing Delusive Adversaries with Adversarial Training, NeurIPS 2021  
[3] Fowl et al., Preventing Unauthorized Use of Proprietary Data: Poisoning for Secure Dataset Release, arxiv preprint, 2021  
[4] Yuan et al., Neural Tangent Generalization Attacks, ICML 2021  
[5] Evtimov et al., Disrupting Model Training with Adversarial Shortcuts, ICML 2021 Workshop  
[6] Nakkiran, Adversarial Examples are Just Bugs, Too, Distill, 2019. https://distill.pub/2019/advex-bugs-discussion/response-5/  
[7] Fowl et al., Adversarial examples make strong poisons, NeurISP 2021  
summary_of_the_review: Overall, I thought the threat model under consideration is interesting and worth exploring, but the authors may need to explain more motivations for their choice of threat budget. I would increase the rating if an updated draft addresses the mentioned issues in the main review.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies a data poisoning method for adversarial training. I like this paper in general: The derivations are clear and empirical results are clearly presented. However, this paper may suffer from unclear formulations of threat model, as I will discuss below in more detail.
main_review: My concern for this paper is along the following lines: The threat model is really not clear -- and to the degree of incomparable.

For example, when the paper first discusses data poisoning for standard training by Huang et al. (3), that threat model is about to say
that we do not modify the labels, but very small perturbations on the features **suffices** to fool the model. The fact that labels are not allowed to modify is the whole point, and the actual interesting part, of Huang's approach.

But this paper then silently switches to allow labels to be changed -- see (4). This is already a significant change of threat model, and if labels are allowed to change, why would it be surprising that AT can be fooled? In fact, if labels are allowed to change, then intuitively AT can be in an even worse situation than standard training: This is because that AT will insist to learn w.r.t. a wrong label, because we want "robustness". This seems to really diminish the "interestingness" of this paper.

I understand that this paper investigated a somewhat deep issue of consistent label bias. This is interesting -- and as I said this paper has some interesting content. To this end, I feel this paper at least needs a significant revision to make clear how to compare with previous work, and why the studied threat model is interesting after all.
summary_of_the_review: As I mentioned in the detailed review, I think this paper silently changed the threat model and allow modification of labels -- this is a significant change, and actually puts AT to a worse situation compared to ST (at least there are intuitions that would support such an argument). More revisions are needed -- in fact I think a significant revision is needed.
review_title: null

Meta-review: The paper shows that adversarial training can be fooled to have robust test accuracy < 1% with a new type of poisoning attack ADVIN on the CIFAR-10 dataset, even though the robust training accuracy > 90%. This requires 100% poisoning rate, though the claim is that the poisoned data is 'semantically similar' to the original data. This is an interesting research direction. Questions were raised about novelty as well as whether these poisoned data could be detected. During the rebuttal phase the authors provide some evidence that with adaptive attacks detection could be evaded (as expected). The authors are encouraged to take all comments into account and update the paper as indicated in the rebuttal for any future revision.
Decision: Reject