Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: Based on the observation that clean and backdoor training samples have different feature statistics, the paper proposes to detect and remove the neurons that are mostly affected by backdoor training samples. Empirical results show the effectiveness of the proposed method under certain scenarios. 
strengths_and_weaknesses: Strength:
1. The proposed method is intuitively correct: Clean samples and backdoor samples should have somehow different feature statistics. This is because the model overfits superficial correlation on backdoor samples while it learns semantic information on clean samples. I don't doubt the proposed method is effective in improving backdoor robustness. 
2. The paper is well-written and easy to follow. All notations and terms are defined clearly without ambiguous. 
3. The proposed method can defend some backdoor attacks when few clean samples are available. 

Weakness:
1. My main concern is how such good performance reported in this paper can be generalized to different settings. 

1.1 In practice, we don't know what type of backdoor attacks are used. So, it is very important for the defense method to be generally robust against multiple different attacks using the **same** hyper-parameters in the defense method. However, in this paper, the authors only mentioned the only hyperparameter $\mu_k$ "is **usually** set to 3". The use of the word "usually" makes me feel skeptical about the results. The readers need to know what value $\mu_k$ is used under all cases (on all datasets and against all attacks). The ideal case is we have a value that works well for all attacks, even under the mixture attack setting (i.e. when multiple different attack simultaneous exists in the training set). But that should be challenging in my point of view since the distributions of backdoor samples from different attacks can be very different. So what are the "unusual" cases in this paper? If different hyper-parameter values are used against different attacks in the proposed method, then it is unfair comparison with the baselines. As far as I know, NAD can achieve better performance if the hyper-parameters are tuned for different attacks. 
Again, I don't doubt for a second that the proposed method can help defense backdoor attacks. I just concern whether it can achieve such good performance across all reported different attacks using the same hyper-parameter setting. 
It won't harm the contribution of this paper if the performance drops a little bit when the same $\mu_k$ value is used for different attacks. 

1.2 Please consider more different attack settings. For example, when a different poisoning ratio (eg 5%, 20%, 30%...) is used? Does the proposed method need to use a different uk value under different poisoning ratios to achieve good performance?

1.3 Please provide ablation study results on different uk values. 


2. In Figure 5 (c) in appendix, the visualization of Blend attack looks a bit weird to me. I can't see any effect of the blend image (which is usually a hello kitty image or strong random noise). Please see Figure 5 in [1] or the original Blend attack paper for example. May I ask what blending ratio and blend image/pattern you are using? 

I will improve my score if the concerns are addressed. 
questions: The title in the submitted PDF is different from that in Openreview. 
limitations: Please see above.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this paper, the authors studies how to better defend models from backdoor attacks. Specifically, they identified that there are only a subset of neurons, which are responsible for the poisoned behaviors. The found that if one prunes these poisoned neurons, then the backdoor behavior can be effectively stopped. The authors found a simple way of identifying such neurons through the two measurements: discrepancy of differential entropy and mismatched batchnorm statistics. By using these two measurements, they are able to more effectively stop the backdoor behavior while requiring very few additional compute. The also can identified such backdoor neurons with as few as 10 benign examples.




strengths_and_weaknesses: **Strength**
- The biggest strength of the paper is the method's effective as well as simplicity
- The presentation of the paper is very clear and easy to follow 
- The experiments are thorough and convincingly shows the benefit of the proposed approach.

**Weakness**
- I do not find any obvious weakness in the paper. The wording may be improved slightly, but it doesn't impact the message of the paper very much.
- Given its superior effectiveness in defending against the standard backdoor attacks in standard settings, I am curious whether it can also help stop the backdoor attacks in self-supervised or unsupervised settings. I think the contribution of the paper is already solid, but am simply interested in learning more about its implication for other backdoor settings.

Overall, I really enjoy reading the paper, and recommend the paper being accepted.


questions: I don't have any questions about the paper as of now.
limitations: The authors have adequately address the limitation of their approach and societal impact of their work.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: In this paper, the authors propose a defense approach to mitigate the backdoor attacks on convolution neural networks (CNNs). Specifically, the authors first observe that the neuron activation distributions within the backdoor-infected CNN are different for benign and backdoored samples. Based on such an observation, the authors further propose to calculate the differential entropy to distinguish the compromised neuron. In the experiment section, the authors evaluate the proposed approach to CIFAR-10 and TinyImageNet benchmarks under different settings(e.g., training phase, post-training phase, etc). Moreover, the authors also test the robustness of the proposed approach on two adaptive attacks. The results demonstrate that the proposed method can significantly outperform comparison work in the dimensions of robustness, efficacy, and efficiency.  

strengths_and_weaknesses: +++++++Strengths+++++++

1. Well-structured presentation and easy to follow.
2. Comprehensive evaluation.  
3. Considering two adaptive attacks.

-------------Weakness------------

1. The novelty is somehow limited. Since inspecting compromised neurons for defending against backdoor attacks has been widely adopted by previous work, this work may not seem that novel. 

2.  I think more previous unlearning-based work or training-phase stage defense should also be considered for comparison. For example, DBD[1] and the most recent approach[2]. 

3. Missing evaluation on several practical scenarios. I think the proposed approach should also be evaluated under some rather practical scenarios, for example,  multiple-label scenarios, and multiple backdoors within the same infected label which previous work (neural cleanse) has discussed that.

[1]https://openreview.net/forum?id=TySnJ-0RdKI
[2]ADVERSARIAL UNLEARNING OF BACKDOORS VIA IMPLICIT HYPERGRADIENT

questions: 1. Is this work can still perform efficacy under multiple-label and other practical scenarios?


2. What device for you to test the running time in the experiment section?
limitations: Requiring clean validation data makes the approach somehow impractical.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper reveals that poisoned neurons and benign neurons have different behaviors, measured by the entropy of neuron activations and the BN statistics. Motivated by these understandings, the authors propose DDE and MBNS targeting the defense of poison-only backdoor attacks and attacked pre-trained models, respectively. The proposed methods are evaluated on both CIFAR-10 and Tiny-ImageNet datasets under seven attacks. 
strengths_and_weaknesses: Pros 
1. The topic is of sufficient significance and interest to NeurIPS audiences. 
2. The authors discuss the resistance to potential adaptive attacks, which should be encouraged.
3. In general, the idea is easy to follow.
4. The authors discuss the efficiency of different defenses, which should be encouraged.


In general, I enjoy reading this paper. However, I still have some concerns about this paper. I will increase my scores if the authors can (partly) address my concerns. The detailed comments are as follows:


Cons
1. Some statements need further support or modifications to avoid over-claim.
- ‘we claim that both the proposed indices can perfectly separate...’ (Line 60-62, Page 2): This statement should be modified. This claim can be made only if your methods have solid theoretical foundations. 
- ‘there was no such quantity defined in the literature to measure...’ (Line 139-140, Page 4): This is an over-claim. Specifically, all existing pruning-based defenses provided a quantity (e.g., activation values on benign samples) to measure it.  
2. Missing some important related works. 
- No discussion about pre-processing-based backdoor defenses (e.g., [1-3])
- Missing two poison-suppression-based backdoor defenses [4, 5].
3. My biggest concern is about the experiment parts.
- How to select the hyper-parameter u? Do you use the same u under different attacks on all datasets?
- To ensure a fair comparison among all pruning-based defenses, I would like to see the results (i.e., ACC and ASR) with respect to the pruning ratio. It is necessary to verify whether the proposed methods are truly better in finding malicious neurons.
- I am interested in whether the proposed defenses are truly resistant to adaptive methods since only poison-only backdoor attacks are used in the evaluations. The author should also test their methods on some training-controlled attacks (e.g., [6, 7]), which aim to reduce the difference between poisoned samples and benign samples in the hidden feature space.


Minor Comments
1. The ‘we will’ should be ‘we’ in Line 36 (Page 1).
2. There are two typos in Table 2: ‘0.1’-->‘0.10’, ‘64.2’-->‘64.20’.
3. Please recheck and cite the official version of all references (e.g., [8-9]).


References
1. DeepSweep: An Evaluation Framework for Mitigating DNN Backdoor Attacks using Data Augmentation. AsiaCCS, 2021.
2. Backdoor Attack in the Physical World. ICLR Workshop, 2021.
3. Neural Trojans. ICCD, 2017.
4. Backdoor Defense via Decoupling the Training Process. ICLR, 2022.
5. Anti-Backdoor Learning: Training Clean Models on Poisoned Data. NeurIPS, 2021.
6. Backdoor Attack with Imperceptible Input and Latent Modification. NeurIPS, 2021.
7. Bypassing Backdoor Detection Algorithms in Deep Learning. EuroS&P, 2020.
8. Backdoor Learning: A Survey. IEEE TNNLS, 2022.
9. Backdoor Scanning for Deep Neural Networks through K-Arm Optimization. ICML, 2021.
questions: 
Cons
1. Some statements need further support or modifications to avoid over-claim.
- ‘we claim that both the proposed indices can perfectly separate...’ (Line 60-62, Page 2): This statement should be modified. This claim can be made only if your methods have solid theoretical foundations. 
- ‘there was no such quantity defined in the literature to measure...’ (Line 139-140, Page 4): This is an over-claim. Specifically, all existing pruning-based defenses provided a quantity (e.g., activation values on benign samples) to measure it.  
2. Missing some important related works. 
- No discussion about pre-processing-based backdoor defenses (e.g., [1-3])
- Missing two poison-suppression-based backdoor defenses [4, 5].
3. My biggest concern is about the experiment parts.
- How to select the hyper-parameter u? Do you use the same u under different attacks on all datasets?
- To ensure a fair comparison among all pruning-based defenses, I would like to see the results (i.e., ACC and ASR) with respect to the pruning ratio. It is necessary to verify whether the proposed methods are truly better in finding malicious neurons.
- I am interested in whether the proposed defenses are truly resistant to adaptive methods since only poison-only backdoor attacks are used in the evaluations. The author should also test their methods on some training-controlled attacks (e.g., [6, 7]), which aim to reduce the difference between poisoned samples and benign samples in the hidden feature space.


Minor Comments
1. The ‘we will’ should be ‘we’ in Line 36 (Page 1).
2. There are two typos in Table 2: ‘0.1’-->‘0.10’, ‘64.2’-->‘64.20’.
3. Please recheck and cite the official version of all references (e.g., [8-9]).

limitations: The authors fail to adequately mention the limitations and potential negative societal impact of their work. Please discuss them explictly.
review_title: null

Meta-review: metareview: The authors propose a hypothesis that backdoor neurons in an infected neural network have a mixture of two distributions with significantly different moments, formed by benign samples and poisoned samples, respectively. They then propose two mathematically informed and intuitive ways to defend against the attack. The method also seems general against most types of backdoor attacks, and the evaluations give more confidence in that direction. The evaluation is extensive and includes most of the state of the art as comparison. An additional advantage is that this method has better runtime than most.

The most critical reviewer, WDQn, was concerned with the lack of evaluations against multiple attack models (multiple label attack, etc) and  the lack of comparisons against SOTA. The authors have responded with detailed and appropriate results in their last response. 

Concerns from other reviewers, such as the concern about the robustness of hyperparameter u, were alleviated through ablations/results posted in the authors' rebuttal.

I therefore recommend accept.
confidence: Certain
Decision: Accept