Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors target to detect the input masks that extract the minimal backdoor patterns that lead to the model output. They further leverage the learned masks to detect and remove backdoor samples from poisoned datasets. Experiments show that they can detect several backdoor attacks and help detect biases from face datasets.
strength_and_weaknesses: *************************
Strengths
*************************
+ Clear exposition: well-organized demos and clear writing.
+ Reasonable motivation.
+ Straightforward method.
+ Extensive experiments covering 3 datasets, 6 victim model architectures, 12 backdoor attacks, and 4 baseline detection methods. Results show their efficacy in terms of backdoor detection accuracy and face bias detection accuracy.
+ Easy to reproduce.

*************************
Weaknesses
*************************
- Technical novelty is incremental.
  - It borrows the idea form Neural Cleanse that embeds backdoor patterns in a minimal image subregion.

- The threat model is unreasonable.
  - The detection can only apply to local patterns. Simple global patterns, e.g., patterns in the frequency domain, can easily circumvent the detection.

- Experiments are unconvincing.
  - Backdoor pattern detection recall looks reasonable, but the precision is more measured. Please test on clean samples and report the false detection rate.
clarity,_quality,_novelty_and_reproducibility: Clear exposition: well-organized demos and clear writing.

Easy to reproduce.
summary_of_the_review: See the above.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a defensive mechanism, Cognitive Distillation (CD), that extracts potential trigger patterns (and a mask) responsible for the model's prediction. The paper starts with the observation that, even if the trigger pattern consists of multiple pixels and/or is complex, a few pixel values correspond to activating backdoors. The paper exploits this observation and proposes an optimization that reconstructs a potential trigger pattern and a mask from an input image. Using those reconstruction outputs, the paper identifies whether the input contains the backdoor or not (if the mask is sufficiently small, then it contains a backdoor). In evaluation, the paper compares the detection rate with four baseline defenses. The paper also discusses the possibility of using CD for bias detection.
strength_and_weaknesses: Strengths:

1. The paper empirically analyzes the commonality across the backdoor attacks.
2. The paper exploits the commonality to build a backdoor input detection method.
3. The paper discusses some other useful ways to harness CD.


Weaknesses:

1. It's not convincing (from the paper) that the backdoored model only uses a subset of the trigger.
2. The technical novelty of the proposed method (CD) is weak (CD is an adaptation of Neural Cleanse).
3. The evaluation is weak (varying poisoning rates won't reflect adaptive attacks, and Neural Cleanse may perform better under the same setting).


Detailed comments:

The paper studies a way to capture input samples containing potential triggers that may lead to backdoor behaviors. As backdoor attacks are a growing concern, it is important to study defense mechanisms to reduce the attack surface. This paper strives to find the commonality across the backdoor behaviors. 

However, the major problem of this approach is that the defense relies on an assumption (backdoored models will use a subset of trigger patterns for classification) that can be easily broken. Another concern is that the optimization process (which is the paper's technical contribution) is highly similar to the process proposed by Wang et al., Neural Cleanse. The paper (moreover) somewhat lacks the evaluation against adaptive attacks, and the discussion about bias detection comes out of the blue. Thus, I think this paper is not ready for ICLR 2023. 

My detailed comments can be found below:


[On the Common Observation]

The paper's claim depends on the observation that most backdoored models rely on a small subset of pixels over the entire trigger pattern. I don't think this is true, and the paper studies only a subset of backdoor attacks. I assume that this may come from the way we train our models. However, as the attacker controls the entire training process, they can add an objective that hinders backdoor behaviors in the (partial) presence of the trigger pattern the attacker uses. By doing so, this defense cannot detect any backdoor inputs at all.

Moreover, even if we just assume that the backdoor attacks "only" use a small subset of pixels in the trigger pattern. There is a problem that the proposed CD can "exactly" reconstruct the pattern used by the attacker. Prior work [1]---that proposed a similar reconstruction method---showed some negative results regarding this. A backdoored model can contain multiple trigger patterns that aren't injected by the attacker. If CD reconstructs one of those patterns, can we say that CD detects backdoored input? 

In the long run, we may forecast a diminishing return. Are we sure whether a clean classifier does not naturally learn such small patterns? As the resolution increases (224+), I assume that there are several minor details in the dataset that a model can learn and focus on. In this case, can we say that a model is backdoored, or does it learn the correct features? Just relying on the assumption that a mask will be small won't lead to a successful defense in the end.

[1] Sun et al., Poisoned classifiers are not only backdoored, they are fundamentally broken.


[On the Technical Novelty]

The optimization process proposed by this work has already been exploited in the backdoor literature. Neural Cleanse proposed a trigger reconstruction process that relies on the same assumption; the pattern will be composed of a smaller number of pixels. [1] also proposed a reconstruction process (that seems to be more advanced than this paper uses, as they use smoothing with a denoiser). Thus, the novelty is less strong.


[On the Adaptive Attacks and Bias Detection]

As pointed out above, the attacker can train a model with an objective that forces the model to use all the pixels in the trigger pattern to trigger backdoor behaviors. By combining this objective and a pattern that uses all the input pixels (such as the trojan trigger used in the literature), the attacker could evade this defense completely.

It's nice that this paper discusses other use cases of CD. However, it's a bit unclear what scientific advances CD makes over the prior approaches to detecting biases in the training data. Unfortunately, in the evaluation, the paper does not draw any comparison with the baseline bias detection. Thus, I can make a conjecture that CD can detect biases, but I am unsure whether CD will have a novelty over the prior work.


[Minors]

1. (In the intro) References to the work that formulates backdooring as a multi-task learning is missing. It has been studied and presented in the "Blind Backdoors in Deep Learning Models" paper.
clarity,_quality,_novelty_and_reproducibility: I included this evaluation in my comments above.
summary_of_the_review: This paper studies a method to capture input samples containing potential triggers that may lead to backdoor behaviors. The proposed method relies on the empirical observation that backdoor models only use a subset of pixels in the actual trigger pattern. However, the major problems of this approach are as follows:

(1) The assumption can be easily broken.
(2) The optimization process (which is the paper's technical contribution) is not novel.
(3) The paper lacks the evaluation against adaptive attacks
(4) It's unclear about the novelty of bias detection.

Thus, I believe the weaknesses outweigh the strengths of this paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a novel backdoor defense called Cognitive Distillation (CD) that defends against data-poisoning attacks. Given a suspect model, CD extracts the minimal essential pattern, called Cognitive Pattern (CP), responsible for the model's prediction on each image input. It observes that backdoored models often provide exceptionally small CPs on poisoned inputs, and this can be used to detect backdoor examples. Then, CD can employ Anti-backdoor learning on the purified dataset to mitigate the model's backdoor effects. CD provides high AUC scores when detecting backdoor samples generated by various backdoor attacks on CIFAR-10. In backdoor mitigation, CD reduces the ASRs better compared with the previous mitigation baselines. CD is also used to detect potential biases on a face attribute classification dataset.
strength_and_weaknesses: ### Strengths
- The paper proposes a new idea of extracting the minimal essential pattern for each input image. It points out that such a pattern is often exceptionally small when the model and the input data are poisoned.
- Based on that observation, the authors proposed a new method to detect backdoor samples. Such a detection scheme provides high AUC scores when detecting backdoor samples generated by various backdoor attacks on CIFAR-10.
- The paper then proposes to mitigate the model's backdoor effects via Anti-backdoor learning, which effectively reduces the models' ASRs.
- CD is also used to detect potential biases on a face attribute classification dataset.

### Weaknesses
- FAR should be normalized by the poisoning rates. For example, given a dataset with 5% data poisoned, a random guess would provide a FAR of 5%, which sounds low but actually not. 
- Given the data imbalance (poisoned vs. clean) and the issue mentioned above, PR curves are preferable over ROC curves, and AUC-PRs are preferable over AUC-ROCs. While TRR and FAR are helpful metrics, Recall is the most critical metric (the percentage of poisoned data that can be detected and removed).
- The mitigated clean accuracy values in Fig.4 are too low, given the original numbers are often around 88-92% (Table 7). Given that significant accuracy drop, these mitigation approaches are impractical. 
- Also, the authors should conduct experiments on clean datasets to see how many clean examples are falsely detected and how much it affects the mitigated clean accuracy.
- The paper provides a massive amount of experiments. However, most experiments are conducted on CIFAR-10.
- In their original papers, recent backdoor attacks such as Dynamic and WaNet can easily pass STRIP by providing similar entropy histograms as from the clean model counterparts. However, this paper's detection AUCs for STRIP on these attacks are unexpectedly high. Can the authors explain the reason?
- The defense baselines are pretty old. The authors should include some more recent backdoor detection approaches, such as [1].
- While interesting, the bias detection part is irrelevant to the rest of the paper.

[1]. Zeng Y, Park W, Mao ZM, Jia R. Rethinking the backdoor attacks' triggers: A frequency perspective. In ICCV 2021 (pp. 16473-16481).
clarity,_quality,_novelty_and_reproducibility: The writing is clear. The proposed idea is new and interesting. However, I have some concerns about the evaluation metrics, the mitigated clean accuracy, and some defense results, as mentioned in the Weaknesses part.

The results seem to be reproducible since the code is submitted in the supplementary.
summary_of_the_review: The proposed idea is new and interesting. However, I have some concerns about the evaluation metrics, the mitigated clean accuracy, and some defense results, as mentioned in the Weaknesses part.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The given work proposes their "Cognitive Distillation" (CD) approach for distilling, detecting backdoor patterns from pixels. The author's claim that this is able to extract the minimal pattern responsible for a model's prediction. Based on their studies, they argue that a sparse input pattern is responsible for backdoor predictions regardless of the trigger pattern. They evaluate their approach on various architectures, standard datasets and attack types. The authors argue that could help detect potential bias in real world datasets.
strength_and_weaknesses: Strengths
* The paper is clearly written, well organized and easy to follow.
* The related work is well motivated.
* The results and ablation studies directly answer the authors claims

Weakness
* Nit: The theoretical motivation could be expanded in Section 3.1. How were each of the terms generalized and weighted?
* The case study on based on relatively simple datasets, it would be great to see this expanded on diverse distributions. 
* It would be interesting to see this approach not only in image space, but including temporal dependencies such as in videos.
clarity,_quality,_novelty_and_reproducibility: As mentioned in the Strengths, the work is well organized, clearly states related work and contributions. This helps understand the novetly which seems reasonable. Based on the details in Section 3 and experimental setup in 4, I'm reasonably confident this is reproducible.
summary_of_the_review: To the best of my knowledge, the authors provide a reasonable contribution to mitigate adversarial backdoor patterns. The work is well written, with the empirical evaluation answering the questions asked. The work seems reasonably novel clearly stating the related work, would be interesting to see it expanded to challenging benchmark tasks. Overall, recommending an accept.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes an algorithm to detect back-doored images by finding the "cognitive pattern" (CP), which is a minimum set of pixels responsible for the model's prediction. Based on the experiments on a wide range of backdoor attacks, the authors found that the images containing backdoor triggers often rely on small CP, and thus they proposed to use CP to detect images with backdoor triggers. 

The review scores for the paper are pretty split, ranging from 3 to 8. After the zoom discussion, we identified the following strengths and weaknesses of the paper: 

Strengths: 
- The empirical results are pretty comprehensive in terms of evaluating many existing backdoor attacks, and the results suggest that the proposed algorithm is useful for defending against existing attacks. 
- The proposed algorithm is well-motivated and easy to understand. 

Weaknesses: 
- The paper relies on the main assumption that adversarial triggers are small, and the model predictions on the back-doored images will rely on a small set of pixels. Although this is true for all the existing attacks, the assumption may not hold in general. For instance, it's possible to intentionally construct a large trigger and enforce the model to make predictions based on the whole trigger. 
- Related to the previous point, we think an adaptive attack (by intentionally constructing such a trigger mentioned above) may break the proposed defense/detection. Although such an adaptive attack requires a very careful design. 

Based on these weaknesses, we think the paper is borderline but finally decide to accept the paper due to its novelty and effectiveness (against standard backdoor attacks). We also compile a list of suggestions for the authors to improve their paper in the camera ready, as listed below: 

- (Most Important!) The authors should state clearly their assumption that "the model predictions on the back-doored images will rely on a small set of pixels", so the readers may not expect the defense to be working against any kind of back-door attacks. 
- To further improve the paper, the authors could conduct adaptive attacks to evaluate whether the proposed method can work with intentionally-designed large triggers. To ensure that the model focuses on the whole trigger instead of just a subset, the attacker can inject randomly masked triggers into the training set and associate them with the correct label. This will ensure that the model's prediction relies on the whole trigger instead of a subset of it. 
- (minor) The proposed method is essentially trying to obtain a saliency map for the model's prediction. There are many other methods to obtain a saliency map in the interpretability community, and it's worthwhile to discuss or even compare the proposed method with other ways to generate the saliency map. 



justification_for_why_not_higher_score: See the weakness mentioned above. 
justification_for_why_not_lower_score: The paper conducts solid experiments, and the method is well-designed if the assumption holds.
Decision: Accept: poster