Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a task-agnostic backdoor attack against the pre-trained NLP models. The attack goal of this work is to implant the backdoor to the pre-trained model so that it can transfer to any downstream tasks and inherit the backdoor. To achieve the goal, the authors propose to make the foundation model produce wrong representations when it detects triggers in the input tokens, so the corresponding downstream tasks have a high probability to give wrong output as well. Experiments evaluate the attack effectiveness of BadPre from the perspective of functionality-preserving, effectiveness, and stealthiness, and demonstrate that BadPre can attack a wide range of downstream NLP tasks in an effective and stealthy way.
main_review: ## Strengths:
1. This paper is overall well written and the method is simple and easy to follow
2. The task-agnostic backdoor attack is an interesting and less explored problem, which can be a real practical threat.

## Weaknesses:
1. The main weakness of this paper is the lack of a comprehensive comparison with existing backdoor attacks on NLP models. The problem could have been addressed by answering the following questions:

a) How is BadPre related to the previous work? What makes BadPre have high transferability to downstream tasks, while others fail to do so? What is the design intuition behind BadPre? 

b) What is the effectiveness, stealthiness, functionality-preserving of baseline attacks? Can you benchmark the previous methods following the same setup? Otherwise, it is clear if BadPre can indeed improve upon prior work.

2. More experimental details should be included (e.g., in Appendix). For example,

a) What are the training details of injecting backdoors? How would it impact the training loss/MLM acc compared to clean training? What are the hyper-parameters to inject the backdoors? What is the computational cost?

b) What doe “clean DM” mean?

Answering the above questions can give us a better understanding how and why backdoors can achieve such a high attack success rate.

3. Some experimental results are a bit concerning and unconvincing:

a) The performance drop on RTE and SQuAD v2 is 7.69% and 3.9%, which is a significant drop. The authors explain that it is due to the “the conﬂict of trigger words with the clean samples”. Could the authors provide some qualitative examples? Can you change the trigger words and see if it can make the attack more stealthy?

b) Why in some cases the backdoored models achieve even higher acc than the clean models (e.g., SST-2).

c) In Figure 2, the authors show the same attack success rate when injecting one or two trigger words, which is a bit counterintuitive, as it is expected to see a much higher attack success rate when injecting more trigger words.

I am willing to raise my scores if the problems above can be well addressed.

summary_of_the_review: Strengths:
1. This paper is overall well written and the method is simple and easy to follow
2. The task-agnostic backdoor attack is an interesting and less explored problem, which can be a real practical threat.

Weakness:
1. Lack of a comprehensive comparison with related work
2. More experimental details are missing.
3. Some experimental results are a bit concerning and unconvincing.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes BadPre, a task-agnostic backdoor attack approach that injects backdoors into neural language models during the pre-training stage and can hence be used against downstream tasks that fine-tune on the pre-trained model. The authors state that a backdoor attack should satisfy the following properties: effectiveness and generalization, functionality-preserving, and stealthiness. The backdoor attack consists of a simple data poisoning method using trigger words that is used during pre-training.

The authors evaluate their method by pre-training BERT models and fine-tuning them on 10 different downstream tasks. The results show that the backdoor attack has little influence on the models’ performances against clean data. However, the backdoor triggers show to be effective against all fine-tuned models as performances drop drastically for malicious inputs. The authors furthermore investigate the attack’s effectiveness against backdoor trigger detection methods and show that (a slightly modified version of) BadPre is effective even if a state-of-the-art defense is employed. 
main_review: The paper addresses a relevant topic and proposes a novel method that has not been covered in the literature. The proposed method is simple yet effective. The paper is well-written and the experiments are extensive and technically sound. I only have a few remarks below, but I find that this work represents a solid contribution, and I am hence in favor of acceptance.

Remarks:
* It would be helpful to briefly elaborate on the practical applicability and relevance of backdoors attacks in NLP. Why do they pose a threat? How can they be used maliciously in practice?
* It would have been interesting to see whether BadPre would be as effective against other variants of BERT (e.g., robustly pre-trained models). Furthermore, additional experiments on whether different parameter configurations and the dataset size during pre-training affect the attack’s performance would be insightful.

Questions:
* Just to clarify: were poisoned samples generated for each clean sample in the pre-training data? If yes, did you conduct experiments on only using a fraction thereof? It would be interesting to provide some insights into whether the entire pre-training dataset needs to be poisoned for the attack to be effective, or if poisoning only a fraction of the data is sufficient.



summary_of_the_review: Overall an interesting paper and a relevant contribution. I recommend acceptance to the conference.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a task-agnostic backdoor injection paradigm BadPre for pretrained NLP models. Towards this, an attacker first poisons the pretraining corpus by introducing unnatural static trigger tokens, and then pre-training the public model to inject the backdoor. This model can then be finetuned on any downstream task and data regularly and can be attacked by injecting the trigger in the input. The approach is evaluated considering the BERT-Base model and over 10 downstream NLP datasets.
main_review: **Strengths:**

The problem considered by the paper of introducing task-agnostic backdoors to pretrained NLP models so that an attacker can attack without knowledge of the downstream task/data is important. However, I am not fully convinced about the empirical evaluation and why the proposed approach should work in practice.


**Weaknesses:**

1) The idea proposed in the paper has very limited technical novelty. There has been previous work that explores transferring of backdoors from pretrained NLP models for downstream fine-tuning (Kurita et al, 2020). The proposed BadPre approach is a straightforward injection of backdoor in the pretraining stage instead of the downstream fine-tuning, without any theoretical explanations/empirical justification/insights as to why the backdoors in the pretraining tasks should be retained over fine-tuning. This makes the technical contribution of this work minimal for presentation as a paper at ICLR 2022.

2) The paper misses providing crucial technical details, which makes understanding the underlying technical aspects difficult. Specifically, in section 4.1 "Pretraining the foundation model", the paper does not provide any details of how the pretraining task is modified for the poisoned data, which pretraining task is chosen, etc. Rather, the paper uses a vague terminology to say that "training procedure mainly follows the training process indicated in BERT" which makes the presented approach not reproducible. Modifying masked-LM and Next sentence prediction tasks to account for backdoors is non-trivial and the crux of the proposed approach, which has been omitted from presentation. 

3) There is no reasoning or explanation provided as to why the pre-training tasks (MLM, NSP) which are very different from the downstream tasks should transfer the backdoor triggers to the downstream task. This is a very important aspect of the paper, and there is no empirical evidence to this claim as well. 


4) The empirical evaluation of this paper is weak and unconvincing:
	- No baseline attacks (Kurita et al, 2020; Zhang et al, 2020; Qi et al, 2021) have been considered when presenting the attack performance of BadPre. While these attacks may require information of the downstream task, it is still important to compare how the attack strength and functionality compares with previously suggested attacks.

	- The drop in normal performance on certain tasks like RTE and SQuAD is significant, and more than what should be tolerable for a stealthy attack that maintains the original functionality.

	- No details have been provided about how poisoning pretraining dataset is made using the BooksCorpus and Wikipedia texts. This is crucial to achieve reproducibility of the work.

	- I disagree with the claim made in the paper about the stealthiness of the BadPre attack. The attack uses a fixed set of trigger words and randomly inserts them into the input as the trigger. This destroys the naturalness and grammatical fluency of the input text, and can easily be detected by a human and/or a simple low-frequency token identifier baseline. Furthermore, the stealthiness evaluation of BadPre is performed without comparisons with any baseline.


**Questions:**

In Section 4.1 "Pretraining the foundation model", is there an error in the phrase "the attacker starts to finetune the clean foundation model F"? Shouldn't it be "continuous pretraining" instead of "finetuning"?


**Missing References:**

Can Adversarial Weight Perturbations Inject Neural Backdoors?, Garg et al, 2020
summary_of_the_review: While the problem considered by the paper: injecting task-agnostic backdoors in pretrained models that sustain after finetuning is very interesting and important, this approach proposed in the paper has minimal technical novelty with missing crucial technical details which makes the paper tough to understand and reason. The empirical evaluation of this paper has several weakness on the fronts of missing baselines, missing details about backdoor poisoning data, and stealthiness evaluation.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose a backdoor technique for NLP models that is agnostic to the downstream task or dataset when the poisoned model is used with transfer learning. Empirical evaluation on a variety of downstream tasks and datasets demonstrates the effectiveness of the proposed approach. The proposed technique is quite simple and seemingly performs quite well when paired with relevant tricks to optimize final attack success rates.
main_review: # Positive feedback

1. I am particularly impressed by Trojan's ability to transfer to downstream datasets and tasks easily. The threat model is quite realistic (an adversary can indeed upload models to collections like those of HuggingFace), and empirical results suggest the backdoor behavior might indeed translate to models trained in the wild.
2. The fact there is a < 1% drop in performance for normal downstream tasks (on data without triggers) is again very useful.
3. Section 5.2: "...by just checking the training performance of the downstream tasks." Since the user does not know beforehand whether the model has a backdoor, it cannot really know whether the performance is "higher" or "lower" than a clean model. Absolute metrics are more relevant here, which the proposed method manages to maintain. The authors might want to emphasize this to make an even stronger case for their work.

---
# Criticism

1. A lot of content until Section 4.1 feels repetitive, getting to the core contribution only nearly halfway through the paper. The authors can probably restructure the paper to focus more on experimental findings and less on re-iterating the same points.
2. If the proposed method can be modified to add multiple triggers to bypass defenses, why can the defenses not be used multiple times? A fairly straightforward defense is to use ONION twice or use it as long as the threshold lies above a specific range. It feels as though this defense would have the potential to work well, and it would be worthwhile for the authors to include this. Maybe it doesn't perform quite as well, further showing how strong the attack is.
3. Section 5.2: "For simplicity, in our experiments, all the values are...". Not sure how scaling final results will help with simplicity in experiments. Please report actual/true values for all metrics.
4. Although the authors test adding triggers in both sentences for models that take pairs as inputs, it may not be possible in cases like question-answering, where the user asks questions and answers are selected from a database. It might work better to report the minimum performance for pair-wise inputs (especially for Q/A) to show how the model can withhold its effectiveness even in the worst case.
5. Evaluation on stealthiness seems to focus on detecting inputs that have triggers. It is equally important to ensure that the end-user cannot detect the presence of a Trojan in the model. I would urge the authors to talk a bit about this scenario (and preferably include results with state-of-the-art Trojan-detection models).

---
# Minor comments

1. Section 2.1: "(normally two fully connected layers)" please provide a source/reference for this.
2. Section 4.2, "However, they are not very useful for specific practical NLP tasks." Please provide a reference for this, or elaborate.
3. Section 4.2, "...will not erase our backdoors..." - this may not be true. If the victim can finetune for a sufficiently high number of iterations, it may bypass the Trojan behavior. Slight modification to this statement (to reflect this) would be appreciated.
4. Algorithm 2 represents a fundamental and standard information flow: not sure if the algorithmic notation justifies the space it consumes. 
5. Table 1: Please find a better way to present these results.
6. Section 4.2: "In another word,..." > Omit "In another word"

summary_of_the_review: Contributions in terms of achieving a successful backdoor technique are well explored. Apart from some cosmetic changes mentioned above, along with the issue of a more vigorous evaluation of potential defenses, I feel the paper is in good shape and would make an valuable contribution to the conference proceedings.
review_title: null

Meta-review: The paper presents a backdoor attack approach against pre-trained models that may affect different downstream languages tasks with the same trigger. The paper shows that the downstream models can inherit security holes from upstream pre-trained models. 

The paper is on the borderline and disagreement remains after discussion and author responses. In general, the new setting introduced in the paper is interesting and well-motivated. However, the options split in how realistic the setting is (e.g., use of uncommon trigger), the evaluation of stealthiness, and the novelty of the idea. After checking the paper, I believe the ideas and insights are justifiable for an ICLR paper and they differ significantly enough from the prior work. I do agree with reviewers that they are some 
limitations of the proposed techniques (mostly inherited from the prior work it based on). However, as backdoor attack in NLP is a relative new area, I would be more lenient on these weaknesses. 

The reviewers also provide constructive suggestions on how to improve the evaluation and writing. I hope the authors can address all the comments in the next revision.
Decision: Accept (Poster)