Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a self-supervised method to defend against backdoor attacks. Specifically, motivated by the adversarial example and backdoor example are closed in the feature space, the paper proposes to pull adversarial examples toward its benign parts by constructing a normalized temperature-scales cross entropy loss. Further, it uses the knowledge distillation to boost the clean example performance. The experiments have been conducted on CIFAR10 dataset in both supervised and self-supervised setting. The results show the proposed method could achieve similar performance with other baselines in supervised setting and be able to defend against backdoor attack in the self-supervised setting.
strength_and_weaknesses: Pros:
1.	The idea of using self-supervised method to defend against backdoor attack is interesting.
2.	The proposed method shows it could both defend against backdoor attack in the self-supervised setting and achieve similar performance in the supervised setting.

Cons:
1.	The paper is not well-organized and hard to follow. The motivation in Figure 1 is not clear and not illustrated. It only mentions the Figure 1a and 1c shows the adversarial example and backdoor examples are close in this projection. However, I am not sure what the difference between different scenarios in a,b,c,d. Also, I find myself couldn’t follow the section 3.3 as well. I don’t see why the contrastive loss would be better than the supervised loss and why the knowledge distillation is necessary. 
2.	The overall setting of the proposed method is vague. The paper aims to provide a method to purify the backdoored model without providing the label. However, it is not clear to me model is a self-supervised model or could be a supervised model as well. It then becomes very confused to interpolate the Table 2 as the pseudo-label data. What’s the pseudo-label’s definition? Why do we need to do the pseudo-label experiments?
3.	The performance improvement is also limited. In Table 1 and 2, the performance is similar with the previous baselines. Therefore, is the main contribution of the paper to improve the efficiency or to provide a way to do it in a self-supervised manner?

clarity,_quality,_novelty_and_reproducibility: Clarity: I find myself struggle to follow the paper. The motivation is unclear and the methodology part in Section 3.3 is also very focused. Several terms are not well-defined and the overall setting is vague.

Quality: The paper organization could be improved. The motivation should include why the self-supervised loss would be used and knowledge distillation is needed. Also, the writing could be improved as well.

Novelty: The proposed setting is interesting and could be a useful direction to explored. However, I am not sure what particular model they focus in the paper so it affects the understanding on the paper’s novelty.

Reproducibility: All hyperparameters are listed and detailed. I think the proposed work doesn’t have reproducibility problem.

summary_of_the_review: Overall, the proposed method proposes an interesting setting to defend against backdoor attacks. However, due to the paper's written quality and poor-organization, it is hard to understand the main message from the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes Contrastive Backdoor Defense (CBD), a novel backdoor defense that can exploit unlabeled data to remove potential backdoors in a provided model. It first generates untargeted adversarial examples via a contrastive loss. It then proposes to finetune the model using a Backdoor-to-Standard Pulling loss that pulls adversarial images toward its benign parts, hoping it can mitigate backdoor effects as the active neurons of the backdoor are changed. That loss is used along with the standard finetuning loss and a knowledge distillation loss. CBD was tested only on CIFAR-10. It shows quite competitive mitigation performance compared to the baseline backdoor mitigation methods when using labels, but more stable in the case of noisy labels. CBD also works better than its counterparts when defending self-supervised models.
strength_and_weaknesses: ### Strengths
- The paper defines an interesting problem of mitigating backdoor effects using unlabelled data. It proposes a novel method to handle this problem that smartly employs contrastive learning.
- From the experiments, CBD shows quite competitive mitigation performance compared to the baseline backdoor mitigation methods when using labels, but more stable in the case of noisy labels.
- This paper also discusses defending contrastive-learning-based models, which is new. CBD works better than its defined counterparts when defending such models.

### Weaknesses
- The connection between adversarial and backdoor examples is not discussed. I cannot find a clear explanation on why we can mitigate backdoor by pulling adversarial images toward its benign parts.
- The experiments are conducted only on CIFAR-10. More datasets should be used.
- CBD often introduces high clean accuracy drops (3-5%). It makes the defense impractical in real life. Though, I agree that the baseline methods face the same issue.
- By the way, the results look much better using L2 norm with \epsilon = 1024. I would recommend using that configuration instead.
- I guess the ImageNet-100 experiment is about image classification, but the writing is really confusing by combining it with the detection experiment description.
- Near the end of page 2: the phrase "defense for detection or detection" is hard to understand. Is it "defense for detection or segmentation" instead?
- Fig. 1: The text font is too small. It is not printing-friendly.
- Page 4: A typo "3nd".
clarity,_quality,_novelty_and_reproducibility: ### Clarity
- The connection between adversarial and backdoor examples is not discussed. I cannot find a clear explanation on why we can mitigate backdoors by pulling adversarial images toward their benign parts.
- I guess the ImageNet-100 experiment is about image classification, but the writing is really confusing by combining it with the detection experiment description.
- Near the end of page 2: the phrase "defense for detection or detection" is hard to understand. Is it "defense for detection or segmentation" instead?
- Fig. 1: The text font is too small. It is not printing-friendly.

### Quality
The writing should be improved more, as mentioned above.

### Novelty
The proposed method seems novel and interesting. This paper also discusses defending contrastive-learning-based models, which is new.

### Reproducibility
No code is submitted, and there is no statement about code release. However, there are some descriptions of how to implement the method and run the experiments.
summary_of_the_review: The paper defines an interesting problem of mitigating backdoor effects using unlabelled data, and proposes a novel method to handle it. CBD shows quite competitive mitigation performance compared to the baseline backdoor mitigation methods when using labels, but more stable in the case of noisy labels. It also works better than its defined counterparts when defending contrastive-learning-based models.

However, the intuition behind the adversarial-clean pulling loss is unclear and unconvincing. More datasets should be used in experiments. The results look much better using L2 norm with \epsilon = 1024, and I would recommend using that configuration instead.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Downloading pre-trained backbones from third-party platforms and deploying them in various downstream tasks is now standard practice. This poses security risks such as backdoor attacks. The authors investigate an interesting question: Do we need labels for backdoor defense? To that end, the authors propose using unsupervised contrastive loss to look for backdoors in feature space. Backdoored examples are created by mimicking adversarial examples generated by contrastive loss. The adversarial finetuning is then used to remove the backdoors. Experiments are carried out on the Cifar-10 dataset, with the authors considering both the supervised and the CL models. The results show that the proposed defense method is effective.

strength_and_weaknesses: Advantages:
- The problem is worth studying. It could happen that the defenders have no access to the labeled data.
- The proposed framework is interesting and has proved to be effective for the Cifar-10 dataset.

Drawbacks:
- My main concern is the generation of adversarial examples. The authors propose simulating backdoored examples with adversarial examples generated by contrastive loss. This goal is unlikely to be met, in my opinion. The generated adversarial examples will typically be found outside the manifold of the attackers' training data set (either clean samples or poisoned samples via adding trigger patterns). This is especially true for real-world applications that use high-resolution images. This could be why the authors only run experiments on the Cifar-10 dataset. Working on low-resolution toy datasets does not imply that the proposed defense method can be easily applied to more difficult real-world datasets.
- The pre-trained backbone is a general backbone in a self-supervised setting (not necessarily using Cifar-10 dataset in this case). In this scenario, how did the attackers insert the backdoor for the Cifar-10 task, and what motivated them? 
- In addition, the author mentioned that “To create backdoor on self-supervised model, we poison 60% of target label 6, which is 6% of all data.” Do you use the labeled data for the self-supervised training (contrastive learning) task? 

clarity,_quality,_novelty_and_reproducibility: The proposed idea is interesting, the paper is well written. It seems to be reproducible for the Cifar-10 dataset.
summary_of_the_review: Although this work has some merits, I am concerned about whether the proposed method can be applied to more difficult real-world datasets.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This work proposed a backdoor mitigation method using unlabeled data, based on contrastive learning. 

There are several important concerns from reviewers:
1. Very vague setting. For example, the input is a supervised or self-supervised backbone. According to the rebuttal, it should be supervised. But the authors emphasized pre-trained backbone multiple times, giving an impression of self-supervised backbone. 
2. Motivation. The connection between backdoored samples and adversarial examples are not well explained, thus the motivation of the proposed method is not clear. 
3. The illustration of the method part is not clear. (I read it, with the same feeling)
4. Unsatisfied results compared to other baselines.

Besides, I  have some additional concerns:
1. It is clearly that this work got the main inspiration from DBD (ICLR 2022), including the clustering and separation of backdoored samples (see fig 1), as well as the defense strategy using contrastive learning. But the authors didn't mention these aspects, instead emphasizing that DBD required labeled data. It is misleading, as DBD is a secure training defense, where the input is a poisoned dataset, rather than a backdoored model. The setting between DBD and this work is different. Besides, even the different setting, DBD could be still compared, if the input is a poisoned dataset, a backdoor model could be firstly trained, then utilizing the proposed CBD method. 
2. Since given a supervised trained model, there are backbone and classifier. The CBD only fine-tune the backbone, how about the classifier, unchanged? I doubt the performance if the combined model with a fine-tuned backbone and a fixed classifier. 
3. The authors emphasized the unlabeled data is the main difference with existing works. However, what is the relationship between this unlabeled data and the training/testing data? Besides, one ECCV 2022 work "Data-free Backdoor Removal based on Channel Lipschitzness" doesn't require any data. 



justification_for_why_not_higher_score: see above
Decision: Reject