Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper propose to distinguish and prune neurons which are sensitive to adversarial perturbations so that backdoor attacks can be better defended. Different from traditional methods which focuses on the perturbations on the input images, this paper explores the correlations between the sensitive neurons to injected backdoor and the multiplicative perturbation on neurons, and proposes Adversarial Neuron Pruning (ANP) algorithm to prune the these neurons to improve the robustness against backdoor attacks.
main_review: Overall, I think this paper is clear and well-organized. Proposed algorithm seems natural. Extensive experiments are conducted to demonstrate the effectiveness of proposed algorithm. The comparison with other baselines show superiority. My only concern lies in the discussion in Section 3.2.

The authors argue that there exist a strong connection between the neurons which are sensitive to adversarial neuron perturbations and those to injected backdoor, which is kind of counter intuitive since adversarial attacks, such as PGD, always generate perturbations with different patterns for different samples, and could activate different neurons. There is no sufficient analysis or empirical observations to support this assumption.

limitations_and_societal_impact: This work could benefit AI safety in terms of outsourcing training and open-sourcing models.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper focuses on the problem of backdoor defense without any knowledge of triggers, which is an emerging topic recently (models are trained at the third party platform). The authors proposed to use adversarially neuron perturbation to find the most sensitive neuron to backdoor trigger, then used Adversarial Neuron Pruning (ANP) to purify the model. The experiments are complete and comprehensive to validate the effectiveness of ANP. 

main_review: Strength:
- It is interesting to know how to trigger the backdoor effect without knowing any knowledge of the trigger. The authors found that through adversarially perturbing neurons, backdoored DNNs can present backdoor behaviors and is much easier to output misclassification than normal DNNs. The methods are novel and the findings are inspiring. 
- The proposed Adversarial Neuron Pruning (ANP) is simple but effective. ANP prunes the most sensitive neurons under adversarial neuron perturbations without fine-tuning. This work also convinces the community that pruning (without fine-tuning) is still a promising defense against backdoor attacks.
- The experiments are comprehensive. Even given a limited amount of clean data and computation resources, ANP consistently removes the injected backdoor and provides the highest robustness against several state-of-the-art backdoor attacks. 

Weakness:
- How to choose the hyperparameters like alpha? 
- What is the overhead of ANP?
- Can ANP be extended to the defense during training? E.g., the sensitive neurons will be pruned along with the training process.
limitations_and_societal_impact: No flaw found
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a simple yet effective method to purify the injected backdoor based on a small portion of clean data without access to training phase.
The proposed method dubbed Adversarial Neuron Pruning (ANP) prunes some sensitive neurons to defend backdoor attacks. Empirical results validate ANP can significantly improve robustness against various backdoor attacks.
main_review: Pros:
1. The proposed method seems to be well motivated. Studying the relation between the backdoor trigger patten and the sensitivity of neurons under adversarial neuron perturbations is novel. 
2. The experimental results comprehensively show that ANP seems to achieve the state-of-the-art robustness against several strong backdoor attacks. In addition, ANP can be conveniently implemented to repair the model without fine-tuning. 

Cons:
1. It seems hard to intuitively understand why the sensitivity of neurons under adversarial neuron perturbation becomes closely related to the injected backdoor although this phenomenon is empirically validated.
2. Does the neuron pruning incur the degradation of generalization? It is better to compare and report the utility of purified model as well.
limitations_and_societal_impact: Limitations: Please carefully refer to "Main Review".
This paper targets to defend backdoor attacks. Therefore, it has no negative societal impacts.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposed Adversarial Neuron Pruning (ANP) to detect the backdoored DNNs. APN prunes the most sensitive neurons under adversarial neuron perturbations. Extensive experiments show that ANP provides state-of-the-art defense performance.

main_review: This work is inspired by adversarial training (AT), which tries to find the most adversarial data around the original data.
Different from adversarial training, this work finds the most adversarial training parameters. 
I like the idea of utilizing AT techniques in backdoor attacks, which can better enhance the defense in backdoor attacks. 

limitations_and_societal_impact: My main concerns:
1. On page 3, you take a fully-connected network as an example to show the training procedure. However, in my opinion, applying ANP on fully-connected layer and convolutional layer are different. FC layer has more neurons while conv layer has much fewer neurons. Thus, compared to conv layer, I think the FC layer can have a better performance but hard to train. Can you conduct a small experiment to show the results of ANP on FC layers?
2. AT is very show, can you report the training time of your methods and other baselines?
3. In each round, you apply PGD only once (PGD-1), how is the performance if you use PGD-10[1] (i.e., apply PGD 10 times each round) instead?
[1] Madry A, Makelov A, Schmidt L, et al. Towards deep learning models resistant to adversarial attacks[J]. arXiv preprint arXiv:1706.06083, 2017.
review_title: null

Meta-review: This paper focuses on the problem of backdoor defense without any knowledge of triggers. The proposal is an adversarially neuron perturbation to find the most sensitive neuron to backdoor trigger, and then use Adversarial Neuron Pruning (ANP) to purify the model. The philosophy behind sounds quite interesting to me, namely, exploring the correlations between the sensitive neurons to injected backdoor and the multiplicative perturbation on neurons. This philosophy leads to a novel algorithm design I have never seen.

The clarity and novelty are above the bar of NeurIPS. While the reviewers had some concerns on the significance, the authors did a particularly good job in their rebuttal. Thus, all of us have agreed to accept this paper for publication! Please include the additional experimental results in the next version.
Decision: Accept (Poster)