Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper introduces an approach regarding how a backdoored pre-trained model can behave maliciously in various downstream tasks without foreknowing task information. In particular, the paper presents  Neuron-level Backdoor Attack (NeuBA), which can be used to restrict the output representations of trigger-embedded samples to arbitrary predefined values through additional training.  
main_review: Strengths.
+Great writing quality.

+The idea of introducing backdoors in PTMs is alarming and important. 

Weaknesses.
- The paper has missed important existing works.

- Important baseline comparisons are missing.

- Detection capability not evaluated.

Detailed Comments.
- I think that the paper has missed important existing works that are closely related to this paper. For example:

Yao, Yuanshun, Huiying Li, Haitao Zheng, and Ben Y. Zhao. "Latent backdoor attacks on deep neural networks." ACM SIGSAC Conference on Computer and Communications Security. 2019.

Jia, Jinyuan, Yupei Liu, and Neil Zhenqiang Gong. "Badencoder: Backdoor attacks to pre-trained encoders in self-supervised learning." IEEE Symposium on Security and Privacy. 2022.

- I would like the authors to state their contributions very clearly, keeping in mind the mentioned existing work. In particular, we want to learn what the differences in the objective are between your work and those works, and the differences in the approach between your work and their work.

- If those works are relevant, the authors should add comparative studies comparing their approach with the approach in those works.
I think that it is important to evaluate the approach for one PTM against multiple tasks, as this is a very common scenario for PTMs. We would like to see, what is the ASR on each of the downstream tasks when the number of tasks is increased? 

- Could the authors include how effective their approach is against backdoor detection approaches like NeuralCleanse and MNTD?

- “From the inference equation” - please refer to this equation.

- A minor aside: I didn’t understand the significance of “neuron-level” in the title. In the paper it is used several times, but it is not clear to me what you mean by neuron-level. Are you referring to the representation vector as “neuron-level”?

summary_of_the_review: Since this paper misses important existing works and baseline comparisons, I don't think the current version can be accepted to ICLR.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a framework to inject backdoor into pre-trained models so
that the backdoor can be inherited by different downstream student models. The
key part of the attack is to restrict the output representations of backdoor
samples via a proposed loss function. Experiment results show that the proposed
method successfully injects backdoors to NLP and CV tasks.
main_review: I appreciate the paper evaluating on both CV and NLP tasks. My main concern
about this paper is its novelty. I am not convinced that it makes significant
contribution or overcomes unique challenges. Moreover, similar works have been
proposed, yet not compared by this paper [0,1,2].

* What is the contribution of this paper? There are existing attacks that
  injects backdoors to pretrained models, such as [0,2]. What is the difference
  between the proposed attack and existing work? Also, what is the unique
  challenge that this paper tries to address? Modifying the loss so that the
  model can memorize the backdoor trigger without affecting the benign accuracy
  seems to be a standard solution, e.g., [2] also uses this. What is the key
  contribution of this work?

* Countermeasures. There has been three types of defense against backdoors:
  training time defenses, e.g., NAD; post-training model examination, e.g.,
  Neural Cleanse; and online detection, e.g., STRIP. This paper evaluates on the
  first type of defense only.

* Could you explain why Fine-pruning significantly outperforms the other two
  methods (i.e., Re-initialization and Neural Attention Distillation)?

* Are there theoretical guarantee of the attack, e.g., control all labels? If
  not, what are the failed cases? And why? 

* In Section 4.3.3, it shows that a large number of injected trigger pairs is
  helpful. Do they have a quantitative relationship?

* Threat models. The threat model of the attack is unclear. This paper mentions
  it needs to feed some backdoor samples to the victim models and collect the
  predicted results to identify target labels, which can be impractical because
  it is hard for attackers to get the output of the fine-tuned models. Are there
  more justifications, e.g., real world scenarios where this happens?

* Scalability. This method relies on injecting a large number of triggers to
  achieve the control over all labels. Evaluations are on downstream tasks with
  2, 4, and 6 classes. Can it scale to models with more labels?

* For the blending ratio of blending backdoor attack, why the blending ratio for
  VGGNet is 1:4, while the ratio for ViT is 3:7?

[0] Shen et al., "Backdoor Pre-trained Models Can Transfer to All", CCS 2021.

[1] Yao, Yuanshun, et al. "Latent backdoor attacks on deep neural networks",
CCS 2019.

[2] Jia, Jinyuan, Yupei Liu, and Neil Zhenqiang Gong. "Badencoder: Backdoor
attacks to pre-trained encoders in self-supervised learning" SP 2022.
summary_of_the_review: I am not convinced that it makes significant contribution or overcomes unique
challenges. Moreover, similar works have been proposed, yet not compared by this
paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed backdoor attacks to pre-trained models (PTM), in which an attacker can train a PTM such that it outputs pre-defined representations for triggered inputs.
main_review: Strengths:

1. Backdoor attacks to pre-trained models is an important security problem.

2. The paper studied the attacks for both NLP and CV domains. 

Weaknesses:

1. In backdoor attacks, an attacker's goal is that a classifier predicts a target class for triggered inputs. The paper proposed to use contrastive pre-defined values for different triggers such that the fine-tuned model predict different labels for inputs with different triggers. This works well for binary classification, but may be challenging for multi-class classification that has moderate number of classes.
The major challenge is that the dimension of the output of PTM can be very high. In this case, the number of pre-defined values is very large (e.g., same as the dimension of the output of PTM). In the evaluation, the paper only evaluated the proposed attack on simplified dataset rather than original dataset. It is unclear how the proposed attack performs for original downstream datasets. 

2. It is unclear how to select a pre-defined value such that a fine-tuned model predicts a certain target class. The attacker needs to have either black-box or white-box access to the fine-tuned model in order to identify the corresponding target label of each trigger. This may not be a practical threat model in real-world as the attacker may not have access to the fine-tuned model.  

3. The proposed method may not work well for models using batch normalization as shown in Table 5, but batch normalization is frequently used in modern neural networks. 

4. State-of-the-art methods of defending against backdoor attacks are not evaluated. Moreover, the proposed defense can be largely defended by Fine-Pruning. 

5. Comparison with existing methods (Zhang et al., Jia et al.) are insufficient. Those two papers also study traojan/backdoor attacks to PTM. Are their methods applicable?

Zhang et al. "Trojaning Language Models for Fun and Profit". In IEEE European Symposium on Security and Privacy, 2021.

Jia et al. "BadEncoder: Backdoor Attacks to Pre-trained Encoders in Self-Supervised Learning". In IEEE Symposium on Security and Privacy, 2022.

In summary, the major challenge of the proposed attack is that it may only work for downstream dataset with a small number of classes (e.g., several classes). 

Minor:

1. In Table 4, the C-Acc without attacks is not reported. 
summary_of_the_review: 1) The attack may be impractical in realistic settings and 2) comparison with existing work is not sufficient. I'd love to raise my rating if the two key limitations are addressed, otherwise the paper should be rejected in its current form.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper shows that a backdoored pre-trained model can behave maliciously in various downstream tasks without foreknowing task information. Instead of building up connections between triggers and target labels, this paper explores to assign predefined output representations to triggers. Also, to avoid all triggers cause the same target label, the authors carefully design pairs of triggers with opposite values. Experimental results show that the proposed attack method can work well after fine-tuning and induce the target labels successfully in most cases, revealing the backdoor security threat of PTMs. Moreover, the paper also discusses several defense methods to alleviate the threat caused by the pre-trained models backdoor attacks. 
main_review: *Pros*:

1. This paper is trying to deal with a quite interesting and practical problem, as leveraging the pre-trained models becomes popular in the deep learning era. 

2. Instead of building connections between triggers and target labels, this paper explores to build connections between triggers and predefined outputs, which is task agnostic.

3. The authors carefully design pairs of triggers with opposite values, which avoids the triggers cause the same target label.

4. Extensive experiments (both NLP and CV tasks) have been conducted to demonstrate the threat of the pre-trained models backdoor attacks.

5. Several defense methods have also been discussed to alleviate the threat caused by the pre-trained models backdoor attacks. 

*Cons*:

1. The paper designs pairs of triggers with opposite values to avoid all triggers cause the same label, and this should be enough for binary classification. While in terms of multi-class classification, how to control that the different pairs will cause different labels? Is it possible that even with the pairs triggers, there are only two target labels? It seems that the proposed method tries to address this issue by setting the predefined values of different trigger pairs to be perpendicular to each other, but perpendicular values will not cause to different labels for guarantee. Looking forward to further explanation for this.

2. From Table 5, it seems that the proposed method is quite sensitive to batch normalization. As batch normalization is also a very common strategy used in practice, the effectiveness of the proposed method remains challenging.

**Post Rebuttal** 

My concerns are somehow addressed. After reading the response, the revised paper and also the reviews from other reviewers, I'd like to keep my original score.
summary_of_the_review: This paper is trying to deal with a quite interesting and practical problem, and extensive experiments (both NLP and CV tasks) have been conducted to demonstrate the threat of the pre-trained models backdoor attacks.
review_title: null

Meta-review: This work proposed to insert backdoor into pre-trained models, such that down-streaming tasks can be attacked. 

One of the main issue indicated by most reviewers is that some important and closely related works are missed and not compared, which also studied the backdoor attack to pre-trained models. The authors argued in the rebuttal that these missed works require some instances of down-streaming tasks, while the proposed method in this work doesn't. However, this difference could not be the reason to miss and not compare with them. 
Besides, most reviewers also indicated the insufficient experiments, such as limited defense methods, and some experimental results are not well explained. 

After reading the manuscript, reviews and discussions between reviewers and authors, I think this work is not ready for publication. The reviewers' comments are supposed to be helpful to improve this work.
Decision: Reject