Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper presents an empirical study on the backdoor erasing in CNN via teacher-student alignment of the attention maps.

S1: An unexpected way to erase backdoors without the need for additional information (provided that the experiments are done correctly). 

W1: The experimental settings and results are highly doubtable.
W2: The findings lack theoretical support.

Detailed comments:

The training progress and the degree of overfitting of the teacher model are important but not clarified in the paper. In figure 4, it shows the four possible combinations of teachers and students. But I can’t understand why "4) C teacher and B student)" can work well too. Isn’t it prone to overfit? Maybe you can draw a learning curve for these 4 methods as well.

Please explain the possible source of information gain. For example: do the teacher and the student models use the same set of augmentation data? How does your approach compare with a baseline where only the teacher model is used and trained with both the data seen by the student and teacher models in your approach?

Please visualize the attention maps when the model is overfitting and when it is not. This helps people better understand the behavior of the NAD.

Please describe your data augmentation strategies in detail. What are their settings? Is the data augmentation methods used in each epoch or just at the beginning of the training process?

Overall, I am not convinced by the claims and results. The current results may be simply due to different degrees of overfitting or different data augmentation strategies. The author is encouraged to conduct more experiments to clarify this.



review_title: Unconvincing results

final_score: rating: 7: Good paper, accept
review_content: ## Overview 

The paper proposes a simple yet effective approach for purifying a neural network poisoned with backdoor attacks, AKA backdoor erasing. In short, the authors propose a two-step process: 1) fine-tuning the poisoned model on a small portion of clean data, which is a commonly used defense, and 2) treating the poisoned model as the student and the fine-tuned model as the teacher and performing attention distillation. The authors show the proposed approach's effectiveness by comparing to three commonly used techniques leveraging six state-of-the-art backdoor attacks on two datasets, namely GTSRB and CIFAR10.  

## Contributions & Strengths

The paper is well-written, the approach is simple yet effective, the ideas are clearly communicated, and the article has a tremendous experimental section.

The contributions of the paper are as follows:

1. Introducing attention-distillation as a simple yet practical approach for erasing backdoors in a poisoned neural network. 
2. Performing extensive studies to compare and validate the existing approach against state-of-the-art backdoor erasing methods
3. Providing substantive ablation studies that further clarify the contribution of each step of the proposed approach and its variations (for instance, in the iterative NAD).
4. The attacks used in the paper cover both patch-like ($3\times 3$ pixels) attacks as well as full image perturbation attacks. 

Given the simplicity of the approach and its practical performance, I would expect the paper to be impactful and of interest to the community.

## Weaknesses

The main weakness of the paper, in my opinion, is the sensitivity of the approach to the regularization parameter $\beta$. Specifically, the optimal parameter choice relies on 1) knowing the type of attack and 2) having access to attacked images. It would be great if the authors can provide insights about choosing a good $\beta$ while being oblivious to the type of attack. 

## Questions and comments for the authors

1. At first glance, the method is counter-intuitive. If I understand correctly, the teacher model is fine-tuned with the first term of the total loss in Eq (3), i.e., starting from the poisoned model, you fine-tune with cross-entropy loss on the clean data. Hence, the optimal solution for the first term of the loss in (3) is the teacher model.  The teacher model is also the minimizer for the second term in the loss. Therefore, I would expect the teacher model to be the optimal solution for minimizing (3). However, the paper shows that this is not the case, and optimizing (3) leads to a better model than the teacher model (i.e., the fine-tuned model). What am I missing?
2. Have the authors considered top-down attention mechanisms, like CAM, GradCAM, and GradCAM++, to calculate the attention maps in place of the norm-based attentions used in the paper? Could you provide any insights on this?
3. In "[Fooling Network Interpretation in Image Classification](https://openaccess.thecvf.com/content_ICCV_2019/html/Subramanya_Fooling_Network_Interpretation_in_Image_Classification_ICCV_2019_paper.html)" Subramanya et al. ICCV2019 propose adversarial patches that have a negligible effect on networks' attention (not in a backdoor setting). Given that the proposed method relies on attention, it would be interesting to see how it fairs against such attacks. 
4. In all your ``clean accuracy' plots (e.g., Fig 2 right panel), could you please provide a zoomed-in version of the curves as well?

## Evaluation logic

Overall, I have a high opinion of this paper and appreciate the work the authors have put into writing a comprehensive article. I am scoring the paper as a 7. I would be happy to increase my score conditioned upon clarification of my questions and addressing the concerns. 

## Post Rebuttal 

I thank the authors for their response. I have two responses: 1) I still don't find the answer to my Q1 convincing; in particular, the 'filtering effect of distillation' mechanism requires more rigorous discussion, and 2) with regards to the ICCV2019 paper, I think the authors may have misinterpreted my point; my point here is that one could design backdoor attacks that do not affect the attention maps substantially and was wondering if the logic would hold for such attacks. However, I agree with the authors that the points go beyond the scope of the current paper and would be interesting for potential future work. In any case, I think the paper is a good contribution to the field and would still vote for accepting the paper. 
review_title: Simple approach, great performance, and a comprehensive experimental setting

final_score: rating: 7: Good paper, accept
review_content: This paper proposed neural attention distillation (NAD), a defense aiming to erase trigger effects from backdoored models with limited clean data. The idea is to leverage knowledge distillation by using a fine-tuned model as a teacher model and use NAD loss as a regularizer to train a student model by matching their attention maps. 

Comparing to three existing defenses (finetuning, fine-pruning, and mode connectivity repairing (MCR)) over 6 different backdoor attacks, NAD shows significantly improved (lower) overall attack success rate while retaining similar accuracy on clean data. The authors also demonstrated that NAD training is more efficient than MCR. Some discussion on the teacher-student combination and the role of attention map is provided.

Overall, I find the defense results descent and the performance improvement significant. The authors also have done a thorough comparison across datasets, attacks, and defenses.

There are a few things I hope the authors can further clarify, and I believe the new results will add more value to this work. I am happy to increase my score if they are addressed.

1. Why is attention map regularization working, especially the fact that both the teacher and student models still carry backdoor effects (backdoored models w/ or w/o finetuning)? In the appendix the authors had some discussion on why using attention map over feature map for distillation loss. I am wondering does this imply feature maps are more vulnerable to trigger compared to attention maps? If so, since the tested attacks do not have the objective of deceiving the attention map (but will affect the feature maps), will the defense still be effective against an advanced backdoor attack that also aiming to deceive/bypass attention maps? In other words, knowing NAD will be used as a defense, it is possible that the attacker can devise an adaptive attack and design a backdoor model to make the distillation loss similar and mitigate the defense. I suggest the authors discuss and compare adaptive attacks to the proposed defense.

2. In the MCR experiment, it is not clear what two end models the authors chose to perform path connection. If the authros use backdoor finetuned model (teacher) and backdoor model (student) as two end models in MCR, will the defense results of MCR be different?
 
review_title: Nice work. More clarification is appreciated.

final_score: rating: 7: Good paper, accept
review_content: Summary:
This paper proposes a novel approach to erase backdoor triggers from neural networks through distillation. The defense method, called neural attention distillation (NAD), first finetunes the backdoored model on a set of clean data to get a teacher model. The second part of NAD then finetunes another copy of the original backdoored model (student) on the same clean data while minimizing the difference between the activation maps of the student model and teacher model. The teacher model’s weights are frozen in this distillation phase. The authors show that NAD outperforms or matches previous backdoor defenses over a range of attacks on the CIFAR-10 and GTSRB datasets. Further experiments are also conducted to show NAD’s performance under different conditions such as the percentage of clean data and configuration of teacher-student models.

Pros:
+Stronger empirical performance in defending against backdoor attacks versus previous methods.
+Solid experiments studying performance under different conditions such as clean data percentage and types of attacks.

Cons:
-The reason behind why NAD’s student model outperforms its teacher (Finetuning) model is unclear, leaving the intuition and basis behind NAD’s performance still not completely understood. 
-NAD’s premise assumes the presence of clean and validated data, having limitations especially for training datasets with a large size.

Recommendation:
NAD’s better empirical performance over prior art and the comprehensiveness of experiments in this paper would be valuable in the effort to tackle the threat of backdoor poisoning. My key concern about this paper is the lack of discussion about why NAD’s student model could outperform its teacher (Finetuning) model even though both are finetuned on clean data and the student model is essentially mimicking the teacher’s activation maps and has the same model architecture. More discussion or theoretical analysis would make the intuition behind NAD more convincing. Moreover, the proposed method did not include a component to detect the presence of a backdoor poison or consider an adaptive attack scenario. Overall, I am still inclined towards accepting due to its good empirical results. 


Comments & Questions:
Evaluating NAD on other model architectures might further show that NAD can generalize well.

Have you consider backdoor attacks targeting multiple classes? How would NAD perform?

What would happen to the required clean data size for the NAD defense to work if the dataset is of larger scale, such as imagenet? Would NAD still need 5% clean data, same, less or more?

--Update after rebuttal--
The reviewer thank the authors for the response. Most of the core issues have been addressed and scores have been updated accordingly.

review_title: Review #4

Meta-review: This paper introduces neural attention-distillation; a new scheme for erasing backdoors in a poisoned neural network. The paper performs an empirical evaluation of their proposed method against  6 state-of-the-art backdoor attacks. The authors show that attention-distillation succeeds by using only a small fraction of clean training data without any performance degradation. In addition, the authors have provided ablation studies to clarify the contribution of each component in their proposed approach. Reviewers find the simplicity and effectiveness of the approach an important attribute that may lead this work to have a high impact in the field. The paper is well-written, and all reviewers rate it on the accept side. I concur with their opinions and comments and I recommend accept.
Decision: Accept (Poster)