Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors empirically investigate the underlying reasons behind the effectiveness of indiscriminate poisoning attacks. According to the authors' claims, their analysis shows that the poisoning samples generated adding imperceptible perturbations to the training data are well separable (accordingly to the class of the original sample) in input space with hyperplanes. This discovery enables the authors to propose a more efficient attack, and to understand that using a pre-trained neural network as a feature extractor can enable defending such poisoning attacks.
main_review: Better understanding the root causes behind the success of indiscriminate poisoning attacks would be a valuable contribution. Unfortunately, this work in its current form presents different issues.

Ambiguous usage of the nomenclature. The authors refer to the poisoning attacks that maximize the error on the test dataset as "indiscriminate poisoning attacks." However, this may be confusing, and it may be worth clarifying the terms. Poisoning attacks that aim to maximize the test error are usually categorized as availability attacks. The term "indiscriminate" in Barreno et al. (2010) has a different meaning, i.e., that the attacker is not targeting a specific user/test sample (but rather any of them – think, e.g., to phishing vs spearphishing). In more recent papers, however, multiclass classification has been considered, and targeted/indiscriminate has been used to refer to the class which the attacker aims the test sample to be assigned to. I think here it would be good to try to disambiguate the terms – see, e.g., Munoz-Gonzalez et al., 2017.

Some claims are not supported by experimental evidence. The authors have made different claims, but unfortunately, the following are not supported experimental evidence.

1) The authors claim the poisoning samples generated adding imperceptible perturbations to the training data are well separable (accordingly to the class of the original sample) in input space with hyperplanes. However, this might be due to the type of classifier considered. For example, does this hold when the attacker targets a Support Vector Machine with an RBF kernel? The authors should consider different types of classifiers and not only deep neural networks with ReLU activation functions (ReLU-DNNs indeed learn a piecewise linear function in input space).

2) The authors claim that linear separability is necessary for indiscriminate poisoning attacks to succeed. However, this claim is not supported by evidence. The authors have only shown that the existing poisoning attack algorithms produce linearly separable perturbations when run against a deep neural network. Nevertheless, this does not exclude that poisoning samples that are not linearly separable remain effective. It may be thus worth investigating the possibility of crafting “adaptive” poisoning attacks that remain undetected when considering separating hyperplanes. Another issue may be related to the high dimensionality of the input space. Is the fact that such attacks are linearly separable due to the high dimensionality of the data, other than the given model?

3) The authors claim the proposed attack is more efficient than the state-of-the-art ones. They should add a comparison of their computational complexity. 

4) The authors claim that using a pre-trained model is sufficient to defend against clean-label poisoning attacks. However, their experiments have considered a single level of perturbation that the attacker can inject. To support their claim, they should instead assess the attack effectiveness for an increasing level of perturbation the attacker can add. The same observation has also been done in this work (https://arxiv.org/abs/2106.07214) for backdoor attacks, so it may be worth clarifying that this finding is not completely novel.

Overall, the authors should present a more extensive experimental analysis to support their claims.

The experimental setting is not sufficiently described. For some experiments, the experimental details are not provided. For example, against which deep neural network have the authors crafted the poisoning samples to generate Table 1? 

Confusing manuscript organization. The description of the experimental setup is fragmented into different sections.


summary_of_the_review: The paper tries to clarify an issue, but more insights need to be developed. For instance, whether such linear separability is induced by the kind of model, input dimensionality, and if it generalizes beyond poisoning attacks – similar evidence are probably existing for backdoor poisoning and adversarial examples too.

I’m also quite concerned about the threat model, as poisoning large fractions of points (more than 20%) is not quite realistic in many practical scenarios (I don’t think it’s realistic to poison up to 50% of the dataset). This should be clarified in the paper, and more details on the practicality of the threat model should be discussed or acknowledged as specific limitations of this work (and other papers that use similar setups).
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors study data poisoning attacks that aim to degrade the overall (test) accuracy of a model by adding small perturbations to all training samples. They find that existing attacks introduce features into the data that make the training dataset easy to fit without learning the underlying concept (and thus not in a way that would generalize to natural test examples). They then show that this principle is sufficient to create poisoning attacks by proposing a simple method relying on random, class-specific pattern. Finally, the authors propose defending against these such poisoning attacks by relying on a pre-trained model and only fine-tuning the last (linear) layer.
main_review: The finding that existing attacks produce linearly-separable datasets is interesting and somewhat counters the standard explanation offered as to why these attacks are effective.

At the same time, the rest of the paper's contributions are rather minor:
- The finding that shortcuts are sufficient for poisoning is relatively well-known. There has already been work that shows that if the training set contains a class-indicative, easy-to-learn pattern the classifier will disproportionately rely on that (e.g., [DecoyMNIST](https://arxiv.org/abs/1703.03717) or [https://arxiv.org/abs/1811.00401](https://arxiv.org/abs/1811.00401) in the context of interpretability). In fact, Tao et al. 2021 (cited in the manuscript) propose a very similar yet simpler attack (P5 in their paper) with virtually the same justification.
- As the authors acknowledge, the proposed defense is only demonstrated to be effective when pre-training is performed on clean data and the adversary has no knowledge of the defense in place. However, these are relatively strong assumptions. An adversary could devise methods to circumvent the defense were this not the case.
summary_of_the_review: While the work does contain some interesting findings, the overall contribution is rather minor when taking into account prior work. Thus, I do not find it suitable for the general NeurIPS audience.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper demonstrated existing indiscriminate data poisoning attacks are a form of linear separable features. They exploit the shortcut learning of deep learning. Based on the understanding, this paper also shows a simple synthetic data can be used to achieve the same effect as existing indiscriminate data poisoning attacks. 
main_review: Overall, I lean towards recommending accepting this paper.

---
Strengths:

- The understanding of this paper is interesting. The understanding applies to a wide range of existing works in this area. The indiscriminate data poisoning attacks are a form of linear separable features. The experimental results (t-SNE and shallow models) supports this claim. 
- Based on this understanding, the proposed synthesized linear separable data is technically sound and shows a effective performance. 
- This paper's presentation is easy to follow and most related works have been discussed.

---

Question/weakness:
- Are other attacks evaluated in Table 1/2 also showing linear separable on t-SNE? 
- The experiments baseline is not consistent. For tables 2 and 4, why not include Alignment/TensorClog in table 4?
- Legends and captions in Figure 4 are not informative enough. What does test (poisoned) mean? Is it training on poisoned training data evaluated against clean test data? Or is it the poisoned test data? 
- Need clarifications in the shortcut features. At the beginning of section 2.4, you describe shortcut learning as spurious features that are useful for training but do not generalize to test data. I believe in Geirhos et al., 2020, it is described as do not generalize to the OOD test set, but these features do generalize to the IID test set. I understand the connection of indiscriminate poisoning attacks, but some evidence/figures would be very helpful to establish this connection. 


summary_of_the_review: This paper made an interesting observation on the linear separability of a wide range of indiscriminate data poisoning attacks. This reveals the reason why such attacks can succuss.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper unveils an intriguing "linear separable" property of various existing indiscriminate poisoning attacks, and then designs a new attack using linearly separable features. Experiments show that the proposed attack is fast and efficient. Additionally, pre-trained feature extractors are suggested as a powerful defense strategy.
main_review: Strengths:

1. This paper shows that linear separability is indeed an underlying principle across many delusive attacks. I thought this finding is very interesting and would help the community to better understand the mechanisms behind the attacks.
2. The proposed synthetic noise is very effective, which makes the finding about the linear separability more solid.
3. The connection between linear separability and shortcut learning is thought-provoking. This shortcoming about shortcuts may also be related to the simplicity bias of neural networks, as observed in [3].

Weaknesses:

1. Some citations in the fourth sentence in the introduction section are not appropriate. Specifically, Fawkes (Shan et al., 2020) and LowKey (Cherepanova et al., 2021) are actually targeted attacks (i.e., causing test-time error on some given samples), rather than indiscriminate attacks (i.e., causing test-time error on all unseen samples) in the context.
2. The criticism from Radiya-Dixit & Tramer (2021) may fail to challenge the effectiveness of the indiscriminate attacks considered in this paper, since their criticism was directed only at those targeted attacks such as Fawkes and LowKey.
3. The first sentence in the abstract describing the indiscriminate attacks is somewhat imprecise. Specifically, not all indiscriminate data poisoning attacks work by adding imperceptible perturbations. Actually, another type of indiscriminate attacks can increase test error by injecting a small amount of arbitrarily crafted examples into the training set (e.g., [1]). Both two types of poisoning attacks belong to indiscriminate attacks. Therefore, it would be better to call the former "clean-label indiscriminate attacks" or, simply, "delusive attacks" as in [2-3].
4. The last sentence on page 1 is imprecise. Specifically, it was [4] that pointed out for the first time that "adversarial examples can serve as an indiscriminate poisoning attack". Later, [3] and [5] both experimented with this idea. The difference is that [3] only verified targeted adversarial examples, while [5] further tested untargeted adversarial examples.
5. In Table 1, it is suggested to report the results of the linear separability for both targeted and untargeted adversarial examples [3-5]. *The content in Section 2.1.3 may also need to be modified to precisely reflect this subtle difference.*
6. In Table 4, it is suggested to further report the performance of adversarial training with small $\epsilon$ as baselines, since it has been found that adversarial training with smaller $\epsilon$ may have better defense performance [3].

[1] Biggio et al., Poisoning Attacks against Support Vector Machines, ICML 2012  
[2] Newsome et al., Paragraph: Thwarting Signature Learning by Training Maliciously, RAID 2006  
[3] Tao et al., Better Safe Than Sorry: Preventing Delusive Adversaries with Adversarial Training, NeurIPS 2021  
[4] Nakkiran, "A Discussion of 'Adversarial Examples Are Not Bugs, They Are Features': Adversarial Examples are Just Bugs, Too", Distill, 2019.  
[5] Fowl et al., Adversarial Examples Make Strong Poisons, NeurIPS 2021  
[6] Huang et al., Unlearnable Examples: Making Personal Data Unexploitable, ICLR 2021
summary_of_the_review: Overall, I feel that the finding of linear separability would be of great interest to the community, and I consider the proposed synthetic noise as a supporting evidence for the main claim about the underlying principle and as a direct application of the linear separability. Therefore, I am leaning to accept. I would like to increase my score if the concerns are well addressed.
review_title: null

Meta-review: The paper studies poisoning attacks of small perturbations to training samples. Existing attacks introduce features that allow for easily fitting the data, but do not lead to good generalization. They use this principle to generate attacks based on random class-specific patterns. They finally propose defense using a pre-trained model whose last layer is fine tuned. Several criticisms of novelty of the work in comparison to prior work were raised during the reviews. At the same time the validity and effectiveness of defenses remains unsubstantiated to a large extent. Although the authors put an effort in addressing these concerns, for the most part some reviewers and myself remain critical of the contributions and novelty of this work.
Decision: Reject