Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a backdoor defense based on the observation that trojan and benign neurons exhibit different behavior in the victim model. Their defense assumes an attacker who can poison a subset of the training data and its labels. They design a training algorithm that detects and suppresses neurons that form shortcuts between classes, which they describe as "linear decision boundaries". Empirical results are provided for image classification datasets, showing that their training algorithm NONE has considerable improvements over existing defenses against three poisoning-based backdoor attacks. Finally, the authors show an ablation study over different trigger sizes and poisoning rates which show that NONE is robust in most cases. 
strengths_and_weaknesses: ** Strengths

* Novelty. Defenses against poisoning-based backdoor attacks are highly relevant. Detecting and suppressing malicious neurons during training is a novel idea that shows promising results. 

* High effectiveness. It appears that NONE is a promising defense that clearly outperforms many existing defenses. The drop in clean accuracy is also often low, (<=3%) making it a practically relevant defense. 

* Ablation studies. The defense was evaluated on multiple datasets against three attacks showing promising results everywhere. The authors also show ablation studies over different trigger sizes and poisoning rates. 

* Open-source code. 

** Weaknesses

Overall, the paper was an interesting read and I found only minor weaknesses in the paper's presentation. The results and intuition are convincing and I believe they are of interest to the research community. 

* Formalization. Many symbols in Algorithm 1 are only defined informally in the text, but it would improve clarity if they were formalized (e.g., the separate, norm, and init function). A table explaining the symbols would help readability a lot. 

* Minor. Please break a single, long paragraph into multiple smaller paragraphs to enhance readability. Some citations are missing (e.g., for all datasets such as TrojAI [1]) 

[1] Karra, Kiran, Chace Ashcraft, and Neil Fendley. "The trojai software framework: An opensource tool for embedding trojans into deep learning models." arXiv preprint arXiv:2003.07233 (2020).
questions: What is the runtime of NONE? How does it scale to larger models or datasets such as ImageNet? 

How many inputs are removed by NONE on datasets such as CIFAR-10? What is the precision and recall (with regards to malicious and clean inputs) of this removal process? 

Have you looked into the performance of NONE against backdoor attacks that can poison the training code? 
limitations: -
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper demonstrated the backdoor-related neurons in the DNN models form a hyperplane surface as the decision region to the backdoor target label. Based on this understanding, the authors proposed a robust training algorithm,  NONE (NON-LinEarity), that identifies linear decision regions, filters out inputs that are potentially poisoned, and resets affected neurons to enforce non-linear decision regions during training. 

strengths_and_weaknesses: The paper is well written and easy to understand.
questions: I have a few concerns regarding the proposed method and experiments：

- The finding that shortcuts or linearly-separable characteristics for backdoor attacks are relatively well-known. The same observation has also been done in works (https://arxiv.org/abs/2106.07214) and (https://arxiv.org/pdf/2111.00898.pdf) for backdoor attacks, so it may be worth clarifying that this finding is not completely novel.

- The authors claim that the hyperplane surface formed in backdoored model is because of the piece-wise linear function (i.e. Relu) and there exists one and only one hyperplane in the input space that corresponds to all poisoning samples. What evidence could support your argument and why the reason there is only exist one hyperplane? Whether such linear separability can be induced by the kind of model architecture, input dimensionality, and other activation functions?

- The authors have shown that the traditional backdoor attack methods such as BadNets attack, produce linear separability against a deep neural network. Nevertheless, It may be worth investigating the possibility of crafting “adaptive” attacks [1,2,3] that confuse both clean and backdoor features together when considering separating hyperplanes. In other words, whether the hyperplane surface assumption still held under such cases?

- For the backdoor remove training, why reset the neuron during training? Instead of resetting after training?

- The authors should add a comparison of their computational complexity.

 
[1] Shafahi A, Huang W R, Najibi M, et al. Poison frogs! targeted clean-label poisoning attacks on neural networks. NeurIPS, 2018.  
[2] Nguyen T A, Tran A. Input-aware dynamic backdoor attack. NeurIPS 2021.  
[3] Doan K, et al. Backdoor attack with imperceptible input and latent modification. NeurIPS 2021.  

limitations: The linearly-separable assumption as an underlying explanation for the effectiveness of backdoor attacks is interesting. However, this work in its current form still lacks sufficient analysis to support this conclusion.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proves that a backdoored DNN learns a hyperplane as the decision region, under some assumptions. The authors argue that the backdoored DNNs will increase linearity by introducing a large percentage of neurons activating on one piece of the activation function. Based on these understandings, the authors propose a new training-stage defense (i.e., NONE) that detects and removes potential poisoned samples and repairs compromised neurons. The proposed defense is evaluated on MNIST, GTSRB, CIFAR-10, ImageNet-10, and TrojAI datasets.
strengths_and_weaknesses: Pros 
1. The topic is of sufficient significance and interest to NeurIPS audiences. 
2. The codes are provided, which should be encouraged.
3. The authors try to provide theoretical support for their method, which should be encouraged.
4. The authors discuss the resistance to potential adaptive attacks, which should be encouraged.
5. In general, the idea is easy to follow, although many important details are missing.



Cons
1. Some statements need more details and justifications. 
- 'An adaptive attack with slow poisoning can bypass such defenses' (Line 5-6, Page 1): What the ‘slow poisoning’ means? It seems that there is no support (references or experiments) for it.
- ‘These methods fail to defend against the natural Trojan. (Line 38-39, Page 2)’: No experiments about whether ABL cannot remove natural Trojan (I only see the results of DPSGD). 
2. Missing assumptions in Theorem 3.3. I think Theorem 3.3 holds if and only if the Trojan is complete and accurate. 
3. Please provide more details about two statements in the appendix (Line 591-593, Page 14).
4. Missing an important related work and baseline. It seems that [1] is the SOTA training-stage backdoor defense, which is not mentioned and discussed in this paper.
5. The organization of Section 4 is very poor. There is a very long paragraph in this section, which significantly increases the reading difficulty. Besides, many important details are missing. For example, how to do the separation (Line 13, Algorithm 1)? What are the technical details of Fisher’s linear discriminant analysis and Jenks natural breaks optimization algorithm?
6. It seems that hidden Trojan backdoor attack is a training-stage attack, which is not effective if training from the scratch (as suggested in [2]). How did the authors evaluate defenses under this attack?
7. Why did the authors compare their defense only with DPSGD under natural Trojans? I understand that DPSGD seems to be effective in defending against natural Trojans. However, this is not the reason for not comparing with other defenses.
8. I think it is necessary to go deeper into the internal mechanism of the proposed method. Specifically, according to Algorithm 1, the proposed method can detect poisoned or biased samples. Please provide the precision and recall of this stage so that the readers can understand how well this stage is.

I will increase my scores if the authors can (partly) address my concerns.
questions: Cons
1. Some statements need more details and justifications. 
- 'An adaptive attack with slow poisoning can bypass such defenses' (Line 5-6, Page 1): What the ‘slow poisoning’ means? It seems that there is no support (references or experiments) for it.
- ‘These methods fail to defend against the natural Trojan. (Line 38-39, Page 2)’: No experiments about whether ABL cannot remove natural Trojan (I only see the results of DPSGD). 
2. Missing assumptions in Theorem 3.3. I think Theorem 3.3 holds if and only if the Trojan is complete and accurate. 
3. Please provide more details about two statements in the appendix (Line 591-593, Page 14).
4. Missing an important related work and baseline. It seems that [1] is the SOTA training-stage backdoor defense, which is not mentioned and discussed in this paper.
5. The organization of Section 4 is very poor. There is a very long paragraph in this section, which significantly increases the reading difficulty. Besides, many important details are missing. For example, how to do the separation (Line 13, Algorithm 1)? What are the technical details of Fisher’s linear discriminant analysis and Jenks natural breaks optimization algorithm?
6. It seems that hidden Trojan backdoor attack is a training-stage attack, which is not effective if training from the scratch (as suggested in [2]). How did the authors evaluate defenses under this attack?
7. Why did the authors compare their defense only with DPSGD under natural Trojans? I understand that DPSGD seems to be effective in defending against natural Trojans. However, this is not the reason for not comparing with other defenses.
8. I think it is necessary to go deeper into the internal mechanism of the proposed method. Specifically, according to Algorithm 1, the proposed method can detect poisoned or biased samples. Please provide the precision and recall of this stage so that the readers can understand how well this stage is.

Minor Comments
1. One typo in Line 91 (Page 2): ‘We’ should be ‘we’.
2. The ‘R’ should be ‘R^l’ in Line 143 (Page 4)?
3. Please recheck and cite the official version of all references (e.g., [3-6]).


References
1. Backdoor Defense via Decoupling the Training Process. ICLR, 2022.
2. Sleeper Agent: Scalable Hidden Trigger Backdoors for Neural Networks Trained from Scratch. arXiv, 2021.
3. Poisoning and backdooring contrastive learning. ICLR, 2022.
4. BadNL: Backdoor Attacks against NLP Models with Semantic-preserving Improvements. ACSAC, 2021.
5. BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain. IEEE Access, 2019.
6. BACKDOORL: Backdoor Attack against Competitive Reinforcement Learning. IJCAI, 2021.
limitations: Limited but sufficient.
review_title: null

Meta-review: metareview: This paper introduces a new algorithm to mitigate backdoors in neural network models.
The reviewers agreed this paper proposes an interesting defense that is well motivated,
carefully evaluated with many ablation studies, and highly effective. The weaknesses
raised have also been mitigated in the rebuttal period and the paper is generlly strong.
confidence: Certain
Decision: Accept