Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary: This paper introduces a novel targeted clean-label poisoning attack, expected to be more efficient and scalable than current ones. The attack is formulated as a bilevel problem which is then solved with a (fast) heuristic approach based on aligning the gradients of the inner and outer objective functions. A theoretical analysis is also reported to show that this strategy consistently finds a descent direction for the outer objective, asymptotically converging to (a local) minimum. 

First of all, I like how the authors have derived their attack and its heuristic solution, and I'm wondering if this generalizes to other applications where bilevel problems are at the core, like meta learning or hyperparameter optimization. However, I have several concerns on the presentation and soundness of the reported results.

First, I think that this paper makes confusion (at least in the reader's mind) when introducing the data poisoning problem. At the beginning, there is no clear distinction between the two main goals of data poisoning:
(1) poisoning availability attacks, which aim to increase the test error causing a denial of service, and
(2) poisoning integrity attacks (which are often referred to -in a misleading manner- as targeted attacks), which aim to allow specific intrusions/attacks at test time (backdoor attacks belong to this category). 
For a clearer nomenclature/definition see: M. Barreno, B. Nelson, A. Joseph, and J. Tygar. The security of machine learning. Machine Learning, 81:121–148, 2010. 
There, indeed, targeted/untargeted is referred to the victim user, not to the goal/security violation caused by the attack.

While this work seems to claim that, in general, poisoning attacks are computationally demanding, a distinction should be made. While poisoning availability attacks are typically much more computationally demanding (they do require solving the bilevel optimization problem to work well) and this heavily hinders their scalability to large datasets, poisoning integrity attacks can be quite efficient (there's no need to solve a bilevel problem for them to work well, as in the case of most of the considered competing attacks in this paper, or anyway the approach can be simplified - see, e.g., Koh et al., ICML 2017, where the whole network was frozen and the bilevel problem was solved only by assuming that the parameters of the last layer were updated).
I believe that this aspect should be clarified from the beginning. First, this paper focuses on *targeted* (or integrity) poisoning attacks, and this should also be clear from the title. Second, sentences related to the overwhelming complexity of targeted/integrity poisoning attacks should be revised (e.g., Koh et al., ICML 2017 also worked on the DogFish data which should be a subset of ImageNet, if I'm not mistaken). 


Another important issue which I do not completely understand is what the authors mean with the word "from scratch", from the viewpoint of previous attacks. I agree with them that previous attacks designed to work on pre-trained models with fine tuning may not work against models which are trained from scratch, but what prevents the attacker to train a model from scratch on the clean data and then design their attack samples with fine tuning? The attack samples can then be added to the initial training set to see if, when learning again from scratch on the poisoned data, the attack remains effective or not. Is this the setup that the authors have considered in their paper for such attacks, or they run them against an "untrained" (or not fully trained) model? If we are in the second scenario, I don't think the analysis reported should be considered fair enough.


There are parts in the paper where it is claimed that 'clean-label' attacks are in some sense better than label flips or attacks that do not preserve image semantics. Why? Are we expecting human labelers to check the quality of our data?
Or are we expecting that clean-label attacks are harder to spot?
Both questions are unaddressed in this paper.
First, I don't think that in many realistic scenarios humans are expected to cleanup the whole training set, especially when it contains a lot of samples. Second, it's also true that the level of noise used in this paper is not so small. By zooming in Figs 6-8, the perturbation becomes quite visible even to the human eye.
Hence, we cannot only instruct humans to detect these patterns, but we can probably train detectors to do that automatically. Accordingly, I don't see in which practical, relevant application scenarios "clean-label" can be retained useful as a requirement.
Finally, even though the authors have analyzed the robustness of their attack against some defense mechanisms, the defenses considered aim to detect mostly poisoning availability attacks and NOT backdoor attacks or targeted/integrity poisoning.
I am even skeptical that such methods can detect label flips or even other current attacks. Have the authors tested such defenses against the competing approaches (poisoning frogs, convex polytopes, etc.)? Do these attacks work or not against them?
How do detection methods for backdoor attacks work against the proposed attack? For a list of such detection methods, see, e.g., Table 1 in https://arxiv.org/abs/1910.03137 (note that some detection methods should work against clean-label attacks too, there's no need to put a trigger on the image).


Finally, the experimental section is missing key information for reproducing the experiments. The parameters \tau, R and M are given a value but not a definition. The figure with the average accuracy vs. time is missing a caption and a figure label. It is extremely unclear what this figure is showing as 1) the parameters are missing descriptions 2) the metric used for evaluating the figure is described nowhere in the paper. This problem also extends to tables 1, 2 and 3: a clear definition of the "evaluation metric" should be given. Are we interested more in preserving accuracy or in the attack success? How is poisoning success defined? (this might be explained in the supplementary material, but it is important for understanding the whole results). Why not including a plot with poisoning success vs. accuracy of the model?

To summarize, the paper is promising, but important details and clarifications are still needed. The experimental section and the way results are presented needs major improvement, as it is hard to tell if the attack is working and how efficient and effective it is from the data presented in this paper. 



** Minor comments: ** 

* The Poison Frogs attack is described in Section 2, marking as a drawback the fact that it only works with fine-tuning. It is not clear however why this is a limitation, as one could train the model with normal training and add the poisoned data in the last epochs. 

* In Algorithm 1, step 9: what is the update being performed? It seems to me that the pseudo-code does not capture the entire processing steps, hence making the whole work hard to reproduce.

* Figure 2 shows gradient alignment along epochs (please report the axis labels), however it does not seem "flat" in the end, it is slightly decreasing. What happens if we increase the number of epochs? Will the alignment disappear?

* Sometimes the reader's expertise is taken for granted (e.g. define "unrolled gradient"). This might make it difficult for the paper to reach a broader audience.

* Eq. 1 shows no constraints on the data points staying in the feature space after perturbations. Is it considered during the experiments?

* It is observed that VGG11 on CIFAR10 is less transferable, but it would be interesting to read a possible explanation for this phenomenon.

* Equations should distinguish vectors from scalars to improve readability.

* Figure 4 is unreadable as the text in labels and legends is too small.


** Comments after reading the authors' rebuttal **

I would like to thank the authors for their clarifications. The threat model is now clearer to me - and I think it deserves clarifications in the paper as well.

First of all, as far as I understand now, there's a net distinction between backdoors and clean-label attacks. Backdoor attacks assume that the attacker controls the design phase and the training process, and releases a backdoored model (which then someone else re-uses possibly with fine tuning). Hence, defenses against backdoors aim to detect whether models have been backdoored or not, and it is reasonable to expect that the defender doesn't know the training data as well as other design choices (as the attacker released the model). In this setting, clean-label attacks do not make sense (as the attacker controls the training labels too).

Clean-label attacks assume a different setup. Here the attacker only injects poisoning samples into the training set but does neither control the training process nor the training labels. Hence, clean-label attacks make sense in this setting. However, it also makes sense that the defender knows the training data (as the defender is the one that trains the algorithm, and the purpose is to either detect and remove the poisoning points or reduce their influence over training) - and hence I'm expecting the authors to do consider previous defenses that assume knowledge of the training set in their work.

To summarize, I think that:

(1) the authors should clarify in the title that they restrict themselves to clean-label integrity/targeted poisoning attacks.

(2) the authors should clarify the threat model, and clearly distinguish poisoning availability attacks (bilevel data poisoning) vs poisoning integrity attacks. Furthermore, in the poisoning integrity/targeted family, backdoor and clean-label attacks should be distinguished and the threat models clarified (in particular, w.r.t assumptions on what the attacker/defender know and have access to).

(3) the authors should revise their sentences on the complexity of data poisoning (previous clean-label targeted attacks like poison frogs are not as complex as bilevel data poisoning attacks). A fairer comparison in terms of complexity should also be considered - how faster is this new attack w.r.t. poison frogs and the other clean-label targeted attacks? (poisoning availability should not be considered here as the goal is different in that case).

(4) In general, there is need to disambiguate clean-label targeted poisoning attacks from the rest, and better position this work in context. Reading the paper in its current form, it seems that the authors are also able to improve scalability of poisoning availability attacks whereas this is not the goal of this work. 

I'm willing to revise my score if the authors agree on making these clarifications in the paper, better highlighting the net contributions of their work and the proper context of competing approaches (which do not include backdoors and poisoning availability attacks).






review_title: Review for Witches' Brew: Industrial Scale Data Poisoning via Gradient Matching

final_score: rating: 7: Good paper, accept
review_content: ## Summary
- The paper proposes a novel data poisoning attack i.e., to perturb a small fraction of images in the victim's training dataset so as to cause targeted misclassification on certain examples at inference time.
- The proposed approach works by perturbing the clean poison set to introduce a gradient direction which mimics the victim training their model on a targeted mislabeled set.
- Experiments on CIFAR10 and ImageNet demonstrate that the model outperforms competing approaches.

---

## Strengths

**1. Attack insight**
- I appreciate the insight used to craft the perturbations for the poisoned instances. It seems reasonable to me to exploit the fact that the poisoned instances are used for training using a gradient-descent approach; using the gradient information as a yardstick to craft the perturbations is a nice insight that the paper leverages.

**2. Thorough evaluation**
- I am impressed by the thoroughness in the evaluation. The authors extensively evaluate numerous factors influencing the performances (e.g., size of ensembles, no. of restarts), compare with recent baselines, etc.
- To add to it, the authors further evaluate on Imagenet and achieve strong results.

**3. Writing**
- I enjoyed the writing in the paper. I found the presentation clear and easy to follow.

---

## Concerns

### Major Concerns

**1. Attack assumes access to exact training data**
- If I understand the approach correctly, it assumes access to the exact dataset used by the victim to train the model? Isn't this a really strong assumption?
- Because if this is the case in the threat scenario, couldn't the attacker simply poison the entire dataset?
- As a result, I wonder whether the attack also extends to the more interesting and practical case where the adversary has limited access to the victim's training set.

**2. From scratch**
- At many times in the paper, the authors remark that the attack works in spite of the targeted model being trained from scratch from an unknown initialization.
- However, I would suspect that it is easier to tailor the poisoned instances with access to a strong gradient signal, such as early on during training. Are the authors aware whether the approach is robust to victim models that has been pretrained?

### Minor Concerns

**3. Test accuracies**
- Could the authors comment on the difference in victim's test-set accuracy training with the clean and poisoned training set? I found this largely missing, since the focus primarily seems to be on the accuracy on the target set.
- Because a minor concern I have is that the victim model might be overfitting to the poisoned instances by trading off test-set accuracy. It would be nice to know how severe this is.

**4. "single differentiation"**
- The authors claim that the attack requires only a "single differentiation". But doesn't the model have to be twice differentiable ($\nabla_x \nabla_\theta \mathcal{L}(\cdot)$) to perform the updates?


### Nitpicks

**5. Strong focus on MetaPoison**
- The paper makes many head-to-head comparisons with MetaPoison. I'm not sure why, since MetaPoison doesn't seem too closely-related. Especially in S5.2, it appears that it is singled out to demonstrate the computational overhead. This is understandable since it's relies on a meta-learning approach.

**6. Writing**
- Fig 4b is unreadable -- I recommend resizing the figure.
- It seems surprising that the related work section claims that poisoning attacks, unlike backdoor attacks, do not require access to test data. As I'm aware, both require the same access to test-set -- specifically that a particular test instance is presented at inference time to cause misclassification. In fact I would think backdoors are more generalizable here since any test instance can be watermarked to introduce misclassification, unlike pre-specified instances in the case of poisoning.

### Post-rebuttal update
I thank the authors for their response -- this helps. Having read the other reviews, I am still leaning towards acceptance.
review_title: Blind review

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposed a simple yet effective approach for data poisoning attack targeting a few "clean-label" victim images, using the idea of gradient matching (cosine similarity maximization) between the gradients of adversarial and clean losses. Although the attack model still requires knowing the network architecture (gray-box setting), the resulting poisoned datasets are more effective against different initializations, and some techniques (e.g. model ensemble, multiple restarts) are proposed to further boost the attack performance. The attack results are significantly better than the compared poisoning attacks, and the authors show effective attacks on the ImageNet dataset as well as Google Cloud AutoML with the poisoned data. The authors also discussed the proposed attack on some defenses, showing that the poison has limited change to feature distribution, and differential privacy can mitigate the attack but at the cost of reduced utility (clean accuracy).

Overall, this paper shows some new insights and sets new benchmarks for targeted data poisoning attacks, with practical threat assessment on ImageNet datasets and Google Cloud AutoML, which I deem as a significant contribution. The proposed gradient matching is simple, intuitive, yet very effective. One limitation from Appendix A.8 is that the proposal may not scale well to more than 1 target image, as indicated by the rapidly decreasing attack accuracy. It will be more meaningful to control the effective budget/target and check the resulting accuracy of different number of targets, in order to understand whether gradient matching is scalable to multiple-target setting.
review_title: New and practical targeted poisoning attack

final_score: rating: 7: Good paper, accept
review_content: This paper presents a scalable data poisoning attack algorithm focusing on targeted attacks. The technique is based on gradient matching, where the intuition is to design the poisoning patterns such that their effect on the gradient of the training loss mimics the gradient as if the targeted test image is included in the training data.

The paper presents both theoretical intuitions behind the algorithm, as well as empirical reduction and simplification to make the algorithm scalable to ImageNet and applicable to even a black-box attack against the Google Cloud AutoML toolkit.

The algorithm proposed in this paper is practical and general, making it a realistic poisoning threat to modern deep learning systems. The presentation is clear and the theoretical justification is intuitive and easy to understand.

Overall, I think this paper is a good contribution to the study of the large-scale poisoning attack.

Minor typo:
 In proof of Prop 1, you need the angle between the two gradients to be almost always smaller than 90 degrees, not 180 degrees.
review_title: Important empirical work demonstrating real threat of poisoning attack on large-scale CNNs.

Meta-review: The paper presents a scalable data poisoning algorithm for targeted attacks, using the idea of designing poisoning patterns which "align" the gradients of the real objective and the adversarial objective. This intuition is supported by theoretical results, and the paper presents convincing experimental results about the effectiveness of the model.

The reviewers overall liked the paper. However, they requested a number of clarifications and some additional work, which should be incorporated in the final version (however, the authors are not required to use the wording as poison integrity/ poison availability). In particular, it would be great to see the experiment the authors suggested in their response to Reviewer 2 about the effectiveness of their method for multiple targets (this is important to better understand the limitations of the proposed approach).
Decision: Accept (Poster)