Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: Reject

This paper consists of two main parts, essentially inspired from the argumentation of the Baconian method. In particular, the first part criticizes current work in the area of data poisoning attacks (pars destruens), while the second part tries to overcome the limits described in the first half of the paper (pars construens).

In their pars destruens, the authors heavily criticize the lack of standardized evaluations across different papers, claiming that: "inconsistent and perfunctory experimentation has rendered performance evaluations and comparisons misleading". The authors then continue by highlighting the inconsistencies observed across different papers and summarize them in Table 1.

First, I think that this criticism is not fully justified. None of the considered papers had the goal of proposing a common benchmark evaluation. They were all proposing different backdoor attacks, under different scenarios. Hence, it is unfair to claim that their evaluations are inconsistent. Of course they are. Every paper considers a more or less different experimental setups according to their hypotheses, to validate or reject them. And this is true also for many other different research topics and areas. I believe that the authors should revise the presentation of their paper, acknowledging that a benchmark methodology is lacking and it is required, but without blaming the others because they did not develop it. Their goal was different.

Second, this work does not consider the whole family of poisoning attacks, but only "targeted" (or better, integrity) ones - this includes backdoor attacks or attacks aimed to misclassify only specific test samples, but not poisoning attacks that aim to indiscriminately increase the test error (i.e., availability / denial-of-service attacks). In addition, the whole paper only considers deep neural networks, and not other models. This should be thus clarified since the beginning in the paper, and better reflected also in the title. A clearer taxonomy of the whole family of data poisoning attacks should also be reported (e.g., in the form of a table to help understand the existing different types of threat in the area of data poisoning).
I recommend the authors to refer to these papers that may help categorizing data poisoning attacks (and systematize nomenclature):
- https://arxiv.org/abs/1910.03137
- https://arxiv.org/abs/1712.03141
- https://dl.acm.org/doi/10.1145/2046684.2046692


The main inconsistencies/issues identified by the authors in the evaluation of backdoor attacks are delineated in Sect. 4:
1. Training without SGD or data augmentation;
2. Victim architecture matters;
3. "Clean" attacks are sometimes dirty;
4. Proper transfer learning is less vulnerable;
5. Performance is not invariant to dataset size;
6. Black-box performance is low;
7. Small sample sizes and non-random targets;
8. Attacks are highly specific to the target image;
9. Backdoor success depends on patch size.

These 9 causes, according to the authors of this work, hinder the impact of the 4 backdoor attacks (FC, CP, CLBD, HTBD) considered in this paper.
I am quite convinced that in some specific cases, as the ones identified in Sect. 4, the attacks may fail, and I agree with the arguments posed by the authors in this section. 

I am only concerned by Issue no. 3 about the need of "clean-label" attacks in realistic settings. This is a common criticism/misconception also related to adversarial examples with imperceptible perturbations.
Why the perturbation should be required to be small? Are there practical scenarios where humans are going to observe the samples and be trained to detect that these samples are "dangerous"?
As the authors of this work seem to be quite concerned on the realism of these attacks, this point should be better discussed, as well as the need of considering the perturbation model to be l-inf with size 8/255 (or anyway fixed). In this respect, note that one more pertinent motivation for requiring small perturbations may be the detectability of the attack by automatic tools (rather than imperceptibility to the human eye); see, e.g., the discussion in https://arxiv.org/abs/1802.07295 and consider expanding the paper to discuss this issue. 

A final comment for the part destruens is that, eventually, it is not well systematized. Besides the 9 issues delineated above, a clear systematization/taxonomy of the potential issues is lacking. For example, issues can be related to the model (architecture), training algorithm (SGD/Adam, etc.), training hyperparameters, etc. Unfortunately, this step is lacking in the paper.
And, as we will see, this impacts the development of a proper evaluation framework.

After their part destruens, in Section 5, the authors move to the part construens of their argumentation, in which they propose a standardized benchmark for evaluation of clean-label and hidden trigger backdoor attacks. Again, I agree that providing a benchmark to assess poisoning attack effectiveness is a valuable contribution, and the authors have done a good job highlighting the factors which may impact the performance of these attacks.

However, I am also concerned about the proposed benchmarking framework.
In particular, as the authors have shown, factors such as the type of the target model, the training dataset size, and the size of the perturbation that the attacker can inject into the poisoning samples substantially impact the attack effectiveness. However, in the proposed framework, those factors assume a single value that may unreasonably favor an approach rather than another. 
When a factor has a substantial impact on the results, it is recommended to analyze the performance when that factor assumes different values, as it is usually done for the size of the perturbation that an attacker can add to evasion samples (see, e.g., http://arxiv.org/abs/1902.06705). 
More generally, a clear evaluation procedure or methodology is neither discussed nor provided, and this stems from the fact that a clear systematization of the causes of failure is lacking in the previous part of the paper.
If we identify, e.g., that the model architecture, the training algorithm and the perturbation size are all affecting the attack impact, a proper evaluation framework for attacks (and defenses) should then consider variants of all these factors, which means:
- testing the attack/defense on different models;
- (for each model) testing with different training algorithms;
- (for each model, training algorithm) testing with different perturbation sizes.
This would indeed give a much more detailed understanding of how an attack/defense performs w.r.t. previously-proposed or existing ones.

To conclude, I mostly liked the idea presented in the paper, but a much higher level of systematization is required to propose a comprehensive framework for evaluation of poisoning attacks and defenses, as well as clarifying that the scope of the framework is also restricted to backdoor/integrity attacks on DNNs. I would anyway encourage the authors to continue working on this benchmark to make it more systematic, fairer and inclusive.
review_title: Review for Just How Toxic is Data Poisoning? A Benchmark for Backdoor and Data Poisoning Attacks

final_score: rating: 5: Marginally below acceptance threshold
review_content: **Summary:** This paper studies 4 different previously proposed (clean label and targeted) poisoning attacks and compares them in a systematic way. The authors argue that the experimental setup in these papers are significantly different and comparing them with each other is hard. This incentivizes them to use a benchmark to test all in a unified way. They show that these 4 attacks are not robust to changing parameters of learning algorithm or the experimental setting. To fix this issue, the paper suggest a benchmark for testing poisoning attacks. Their benchmark is only for "clean label" and "targeted" poisoning attacks. They identify several issues with the experimental settings of previous poisoning attacks and suggest some guidelines on  for evaluating poisoning attacks. 

**Evaluation:** I like the paper's analysis of different attacks and showing when they fail and more importantly, when they succeed. However, In my opinion, I don't find the findings of the paper in lack of robustness for the attacks surprising. These attacks are all proof of concepts that show a serious security concerns for real world applications. It is not surprising that none of them cannot succeed in all scenarios and in a robust manner. Attackers can always make tweaks to their algorithm for individual settings (adaptive attacks). 

In general, I think designing adaptive benchmarks for attacks is not a very good idea. There might not exist a single attack that succeeds in all the scenarios in benchmark and this can give a false sense of security. Even if there is an attack that passes all the tests in the benchmark, one can try and find a scenario where the attack fails and add it to the benchmark. The authors might argue that the goal of the benchmark is not to show that the security threat is not real or that the attacks are weak, but to give a way to compare different attacks and provide insight for when attacks are (un)successful. I think this goal would be much more valuable as it poses a lot of open question for future research, but the current tone of the paper seems not to focus on that. 

Comment to authors:
-The paper starts great by talking about the relevance of poisoning attacks in real world applications, but then the benchmarks are only for image classification. I think the focus on image classification is disproportionate to real world applications of machine learning. 
- Any particular reason why you don't consider indiscriminative attacks?
-In the description of attacks it is not specified how the labels of the poison points are selected.
-In the blackbox setting, does the attacker know the clean training set?


review_title: Attacks do not need to work in all settings

final_score: rating: 7: Good paper, accept
review_content: The authors study a number of existing data poisoning attacks, ablating different design choices of these attacks and evaluating them on a common benchmark. They find that many of these attacks are quite brittle to changes in their original experimental evaluation and fail to generalize to more realistic/challenging scenarios. Then, they propose a unified benchmarks for evaluating these attacks.

Overall, I found the paper quite interesting. As far as I can tell, the experimental methodology is solid and clearly demonstrates the brittleness of many of these attacks. Similarly, the benchmark proposed presents a reasonable and realistic setting to evaluate attacks on. I believe that such unified benchmarks are important for making progress in such an actively growing field. While early attack approaches serve as proof of concept for different vulnerabilities, understanding when these attacks fail and where should one focus to improve them is necessary to make progress.

Other comments (not affecting score):
- S2, 1st paragraph: Early backdoor attacks were not clean-label because they also explicitly flip the label of the poisoned examples.
- S2, 3rd paragraph: Backdoor attacks cause the victim to misclassify everything _except_ the target class (adding the trigger causes the model to predict the target class).
- The CLBD attack does examine a modified trigger that works in the presence of data augmentation.
- Is there some intuition as to why the success rate of CP goes down when adding more training samples? If I understand correctly, the success of CLBD increases since the overall accuracy of the model improves with additional data (the model is more likely to rely on the trigger to perform well on the poisoned examples which, after all are inconsistent with the clean classifier).

====== POST-RESPONSE UPDATE ======
After reading the authors' responses and the rest of the reviews, I still stand by my original score.

That being said, I do agree with the other reviewers that the framing of certain discussion points could be improved to avoid misconceptions. Specifically:
- There are places where the narrative feels accusatory towards earlier attacks (e.g., words like "flaws").
- It would be good to explicitly state the scope of the paper relatively early (e.g., "clean-label attacks on deep learning for images") and explain this choice.
- Clarify that this is not meant to be the ultimate poisoning benchmark but rather a first step that allows us to make progress in terms of attack development and evaluation.

It would be great if these could be incorporated in the next revision of the manuscript.
review_title: Solid methodology

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper discusses many issues in the data poisoning literature. They call into question the real world applicability of data poisoning attacks and discuss the short coming in the literature. They present several issues in the comparison of such attacks. Notably they show that many attributes of a data poisoning attack assumed to be irrelevant to the success of the attack are in fact statistically significant, such as a dataset size. Finally they present a new benchmark which attempts to standardize the comparison between data poisoning attacks.  

I think this paper will benefit the data poisoning literature by illustrating its short comings and presenting a path forward. 

----------------------------
Suggestions: 

in the '“Clean” attacks are sometimes dirty ' section the authors present two examples, but a human study could be set up in order to establish how many 'clean' attacks are actually identifiable. A small Mechanical turk study would greatly increase the strength of this argument and should not be financially prohibitive. 

The section on 'Performance is not invariant to dataset size ' convinces the reader that this attribute thought to be irrelevant to the success of the attack is relevant, but proposes no mechanism by which the data set size could effect the attack. A discussion section attempting to provide an answer to this question would be interesting, but is perhaps out of scope.  

The section on data augmentation does not mention if the augmentations are also applied to the poisoned inputs. If the clean data is augmented but the poisoned data is not, it seems expected that the success of the attack would drop. 

review_title: A good paper with needed clarity in a convoluted sub field

Meta-review: This paper generated significant discussion and division amongst the reviewers. On the positive side, some reviewers enjoyed both contributions, feeling the further empirical investigation of existing attacks to be interesting, and the creation of a benchmark to be very useful. On the negative side, no new positive results were proposed, criticism of previous attacks were considered to be unjust, the focus was somewhat narrow, and a benchmark could plausibly be misleading and detrimental.

Given the highly competitive nature of ICLR and the many other excellent submissions, the committee was unable to accept the paper at this time. Below are some suggestions for future submissions.

The content of the paper is generally fine, as long as the caveats and the "tone" are appropriate: we would hope to not mislead potential readers.
Here are some (strong) recommendations:
- Previous works were proof of concept attacks, and the authors should be careful to not frame them as being "broken" -- they perhaps were not meant to be robust to these modifications.
- The scope is somewhat narrow. There should be some explicitly statement and justification of the scope, and what in particular is *not* covered by the investigation.
- Importantly, a single benchmark can't be a unique gold standard, for many reasons discussed by reviewers. Please state these caveats clearly and prominently in the paper and/or code release, as otherwise the presence of a benchmark could do more harm than good. In particular, Reviewer 4 brought up the following philosophical concern with a benchmark, which I believe is quite reasonable, and I reproduce verbatim. The authors should try to address this in the next version: "This kind of benchmark can push the research to a wrong direction. In my view, the point of attacks are to create an alarm for using machine learning in critical applications. Developing these benchmarks would push the competition in the direction of making existing attacks "better" (whatever "better" means in the benchmark) instead of focusing on designing defense techniques or showing the severity of attacks in other situations. This benchmark could also have a bad effect on future attacks (attacks that want to show a new threat, not the one that try to improve the performance of clean label targeted poisoning attacks on deep neural nets) to gain attention from community as they probably will not pass all the criteria of this benchmark."

As another comment (I believe mentioned by other reviewers), it would be nice if all the terms and settings were defined clearly and precisely. For a benchmarking paper, it is important that the reader can clearly understand the threat model, and what does and does not count as a valid attack.

Finally, many of the reviewers gave detailed comments and concerns. The authors should please note and discuss these concerns in future versions (or at least in a supplement or an arXiv version).
Decision: Reject