Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper identifies an incompatibility property of the interaction of clean and poisoned data, i.e., involving poisoned data does not improve model performance on clean data and vice versa. The authors then leverage this property to develop a detection algorithm for finding the clean samples among the poisoned dataset. It is achieved by first separating the dataset into multiple subsets of either all clean data or all poisoned data, and then identifying the clean data among these subsets through a voting algorithm. The proposed approach ISPL+B demonstrates great performance against various types of backdoor attacks.
strength_and_weaknesses: **Strengths**

1. The authors identified an interesting universal property intrinsic to the backdoor poisoning attack, which inspires the design of defense algorithms. 
2. The authors conducted extensive evaluations and demonstrated effectiveness of their approach.

**Weaknesses**

I do not have major complaints about the proposed method. Below are a few comments which I hope can help improve the paper or questions to assist my understanding.

1. A highly-relevant related work [1] is missing. This submission shares a similar flavor with [1]. Both works are developed based on the intuition that clean samples will not help the performance on poisoned samples, and both works iteratively refine the subsets. I recommend the authors to add [1] in Sec 6 related work and discuss the similarities and differences.
2. The approach proposed in [1] requires access to a trained poisoned model as well as a misclassification event during testing time (their setup is forensics); your work does not assume access to these. It is pretty normal to assume the access to a trained poisoned model ([2] uses this assumption as well; see their Sec. 6). Do you think this knowledge can be leveraged to improve your approach in any principled way?
3. I understand the goal of learning a parameter $\tilde \theta$ such that it is as close as possible to $\theta^\star=\mathcal{A}(C)$ which is a model obtained on clean data only (Sec 2.2). But I have a few following thoughts: 1. How close is it to the model obtained on the original dataset $D=C \cup A$? This can be answered through experiments (and I'd appreciate it if the authors can show some results.) 2. If the gap is large in the first step (e.g., when the size of $A$ is almost the same as that of $C$, is it possible to rectify the poisoned set $P$ to its original form $A$ in some way? This could be an interesting direction for future work.
4. The definition of compatibility Def 3.2 is not symmetric. Could the authors elaborate more on this point?
5. For the Incompatibility property in Def 3.3, I'm curious how well is it satisfied in the practical attacks, and thus have the two following questions: a) Can we propose a proxy to evaluate this? b) If an attack doesn't satisfy it very well, what's the reason, and how significant it will degrade the defense?

Experiments wise:

6. Issues with Fig. 2 and the descriptions: a) left column, 2nd subfigure from the top: 0 is above 10 in the y-axis; this looks incorrect to me. b) right column, I do not understand the "source" and "non-source" for CLBD. There is no "source" but only "target" in CLBD. I find the interpretations of the subfigure difficult to understand. c) The last row shows absolute change in accuracy on poisoned data; I would love to see the absolute accuracy on poisoned data as well to understand how effective the attack is.
7. In Sec. 5.2, the authors compared to "training on the original clean samples". Does it refer to training on $C$ or $D$? (This is related to point 3.)
8. In Sec. 5.2 "Comparison with Existing Approaches", the authors said they compared with "3 existing defenses". I would recommend the authors to add the names or at least references for the three methods in this sentence.

**Minors**

* In Eq. (1), $A_{emp}$ should be $R_{emp}$.
* Typos
  * "as an absolute different in accuracy" --> "difference"
 


**References**

[1] Shan, Shawn, et al. "Poison Forensics: Traceback of Data Poisoning Attacks in Neural Networks." 31st USENIX Security Symposium (USENIX Security 22). 2022.

[2] Cui, Ganqu, et al. "A Unified Evaluation of Textual Backdoor Learning: Frameworks and Benchmarks." Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track.


clarity,_quality,_novelty_and_reproducibility: The clarify, quality, novelty and reproducibility are all good.
summary_of_the_review: The problem is important and the solution is novel. The paper quality is good. I only have a few comments and questions. I'm leaning to acceptance of the paper.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper addresses the need to detect backdoor poisoned data, in which a subset of training data is permuted with a watermark, resulting in some elements of a testing set being incorrectly classified.  The paper first identifies an incompatibility property, in which poisoned training data does not improve model accuracy on clean data, and vice-versa.  They show this property applies with high probability to maximum-margin classifiers whose data have been backdoor poisoned.  They then introduce an algorithm which leverages this property to identify clean data.  Finally, they introduce an algorithm which loosens many of the requirements for the above, and tractably trains a classifier on data which is likely to be clean.
strength_and_weaknesses: The paper addresses the important data poisoning problem.  The approach is innovative and interesting, and effective defenses to data poisoning are rare enough that even incremental improvement on this problem is important and worth reading.  Experimental results are promising, and incompatibility provides a useful way of thinking about data poisoning.

The experiments, however, are very poorly introduced.  I was unclear if these were still being run on the maximum margin classifiers from the theoretical results, or if they were being run on something more "realistic."  I believe from the appendix they were run on ResNet variants, but it is still somewhat unclear.  This is obviously a very important point to be unclear on: does the theory, understandably developed on linear classifiers, extend well to more modern, nonlinear classifiers, or not?

I am also interested in whether the incompatibility properties hold, in theory or in practice, for other types of data poisoning that allow for distinct perturbations on individual training images.  I suspect this will be every reader's first question, and so it should be addressed in the paper.  Incompatibility is much more significant if it does hold on these other types of poisoning.
clarity,_quality,_novelty_and_reproducibility: The work is novel, and potentially significant.  It can be difficult to reason about the training process in the presence of poisoned data, but the incompatibility property can help.

Unfortunately, as stated above, I don't find it that clear.  I have significant questions about the basic layout of the experiments.  I believe this is easily clarified by the authors, but it must be.  Of course, this damages reproducibility significantly, as well.
summary_of_the_review: This is potentially an important result.  The bones of the work is interesting, useful, and potentially provides one of the first real defenses against data poisoning I've seen.  However, there are significant unexplained aspects of the experimental setup that make it difficult to judge and impossible to reproduce.  With editing, this would be an acceptable paper; without, it is not.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The backdoor attack has recently received increased attention from the community. The incompatibility property is proposed in this paper in terms of the interaction of clean and poisoned data with the training algorithm, specifically that including poisoned data in the training dataset does not improve model accuracy on clean data and vice versa. The results of the experiments show that prior dirty-label and clean-label backdoor attacks in the literature produce poisoned datasets with behavior consistent with the incompatibility property. Based on this motivation, the authors propose to partition the original dataset into disjoint clean and poisoned components. Experiment results on the CIFAR-10 and GTSRB datasets show that the proposed defense method can reduce attack success rates to less than 1% in the majority of scenarios with negligible accuracy drop.

=========
Because the authors' response partially addressed my concern, I have raised my rating to 6.
strength_and_weaknesses: Strength:
- Addressing the backdoor defense is an important and timely topic right now.
- The paper is overall well written.
- The proposed incompatibility property is novel, and experimental results validate its effectiveness in defending against backdoor attacks.

Weaknesses:
- This work considers the case where the defenders have complete access to the poisoned training set. The CIFAR-10 and GTSRB datasets used in the experiments are both low-resolution, allowing the defenders to easily examine the entire dataset to find the poisoned samples. In other words, the two toy datasets used are far too simple. The successful implementation of the incompatibility property on these two datasets does not imply that the proposed incompatibility property will work well for more difficult high-resolution real-world datasets (e.g., 224 by 224 resolution images with 5 by 5 resolution trigger patterns). It would be great if the authors could demonstrate the applicability of the proposed incompatibility property to these difficult cases.
- When the defenders have no knowledge of the ratio of poisoning samples, will the proposed partition solution still be useful?

clarity,_quality,_novelty_and_reproducibility: The proposed idea is intriguing, and the paper is well written. It appears to be reproducible for the CIFAR-10 and GTSRB datasets.

summary_of_the_review: Although this work has some merit, I am concerned about its applicability to more difficult real-world datasets with high-resolution images.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper notices that poisoned and clean "domains" do not help classify one another.  Therefore, identifying the subset of the data that does not help classify the rest can defend against poisoning attacks.  The paper also proposes a computationally feasible approach to this defense strategy and tries it on several datasets and compares its effectiveness against three existing defenses.
strength_and_weaknesses: I really like the concept of this paper, the writing is clear, and the experimental evaluations are solid.  I especially appreciate that you consider an adaptive attacker.

I would make a suggestion though.  You might consider trying gradient-based backdoor attacks such as Sleeper Agent since (1) it may have very different properties than the types of poisons you already consider which contain patches or 8x8 watermarks in the poisons, and (2) you may be able to use it for a stronger adaptive attack.

clarity,_quality,_novelty_and_reproducibility: -Figures are in general much easier to read than tables, but Figure 2 was very difficult for me to read, since the y-axis labelings are very coarse, and I can’t tell if it is log-scale or linear if there are only two labeled tick marks per subfigure.
-The captions are not descriptive enough to understand the figures without trying to find where they are referenced in the text.  For example, in Table 1, I can’t even understand what any of the entries mean without looking at the text.  It just says they indicate “performance”.  The caption must indicate what “performance” means.  I do understand that this information is contained in Section 5.2, but it would be good to include it in the caption as well.
-Please add error bars.  I might be missing something but some of the experiments seem like they were not run very many times and therefore the statistical significance may be small.
-I appreciate the comparison to other defenses.  It should be pointed out that the defenses you compare to are all several years old and stronger defenses exist right now.
summary_of_the_review: In general, the concept and execution are strong.  Several problems exist in the writing and some additional experiments and error bars should be added.  Nonetheless, I would tend to accept this paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper formally defines "incompatibility property" of the interaction between clean and poisoned data and shows how to use this idea to detect poisoned data. Although all the reviewers agree that the paper is novel and interesting, there were several questions raised by the reviewers in the initial phase including some clarification questions (e.g., what kind of models are used in the experiments, whether the method can work for higher resolution, and what kind of knowledge about poisoned data is required by the algorithm). This paper initially had a low average score (5) and was not listed as borderline (so no zoom meeting was scheduled).  

In the rebuttal phase, most of the important concerns are addressed, and the paper ends up getting 3 weak accepts and 1 reject in the very late stage. The AC carefully checked the reviews and rebuttals and found that most of the concerns are already addressed, and all the reviewers (even the rejection one) agree that the paper is interesting and novel.  The AC also agree with the reviewers that the idea of incompatibility is interesting, especially how to formulate it mathematically. As the reviewer who gave score 3 (reject) did not respond to rebuttal and several reminders sent by the AC, and the comments from the original review have been addressed, the AC decide to recommend acceptance for this paper. 
justification_for_why_not_higher_score: The experimental part is ok but not very strong for this paper. Also, clarity can be improved. 
justification_for_why_not_lower_score: The proposed idea is novel and interesting.
Decision: Accept: poster