Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: Backdoor attack is becoming increasingly relevant in ML applications. This paper introduces a benchmark for backdoor attacks and defenses in the context of NLP models. To foster research progress in this area, the authors unify different aspects of backdoor attacks under a common framework by introducing evaluation protocols and metrics for improving the way defense and attack algorithms are benchmarked. The OpenBackdoor framework can be used by future attackers or defenders coming up with new algorithms.
strengths: - Attack scenarios considered in this paper make benchmarking move towards practical evaluation.
- Several defense techniques are evaluated and compared based on their training/inference categorization.

weaknesses: - Documentation is missing at this link https://openbackdoor.readthedocs.io/en/latest/
- There is no discussion on the config file structure and specification in the github directory
- Usage of Universal Sentence Encoder to distinguish between semantic shifts and backdoor triggers may not be a good strategy if the sentences change significantly while being semantically similar. 
- Shouldn't stealthiness be a part of effectiveness since rejecting poisoned samples can be a part of the defense strategy?
correctness: Most of the claims are correct in my understanding although more extensive discussion and documentation would improve their framework.
clarity: The paper is well and clearly written.
relation to prior work: I did not see any comparison with existing benchmarking effort in backdoor attacks. Such as [1].

1. Schwarzschild, A., Goldblum, M., Gupta, A., Dickerson, J.P. and Goldstein, T., 2021, July. Just how toxic is data poisoning? a unified benchmark for backdoor and data poisoning attacks. In International Conference on Machine Learning (pp. 9389-9398). PMLR.
additional feedback: I am positive about this work even though it doesn't seem completely ready end to end, I think it would be a good starting point for the community to build together.
review_title: A new promising benchmark to accelerate progress in backdoor attacks and defenses for NLP

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This manuscript proposes a framework for evaluating backdoor learning in textual models. The authors provide a critique of evaluation in prior work and present "OpenBackdoor" which comprises implementation of several attacks and tasks. 
strengths: The manuscript addresses an important practical issue in the security of deep learning systems, focusing on pre-trained language models. The manuscript makes a good observation on the lack of standard evaluation approaches for different backdooring attacks and defences for deep learning of text. The framework appears to bring together a collection of attacks and defences from prior work in an easy-to-access repository. 
weaknesses: - In section 2.1.2, three "attack scenarios" are presented, classified along the axes of how much data is provided by an adversary (Scenario 1: the dataset is poisoned, Scenario 2: the backdoored model is released with the expectation that it is fine-tuned by the end user, Scenario 3: the backdoored model is released as-is for direct use). Are these scenarios the only meaningful attack scenarios in this domain? Consider the scenario where the defender wants to fine-tune a "clean" model with poisoned data. Is this encompassed under Scenario 1? For instance, it could be the case that there is a malicious contributor of data. 

Perhaps it would be more useful to specify the various dimensions from which we could draw any combination to specify the threat model. For example, the ability of the attacker to train/retrain, the ability of the defender to train/retrain, the ability of the attacker to poison the initial training data, the ability of the attacker to poison any fine-tuning data, the ability of the defender to acquire clean data, etc...a scenario could be constructed by any collection of these elements (so there could possibly be more than just three scenarios). It does seem that the authors are maybe thinking on these lines also, given that the paper states that "one attack model is not limited to a single scenario...attack models which release datasets can also be used for training and releasing poisoned models." Then, it could be clarified and discussed why the three proposed scenarios are the only ones that we should be concerned with (if this is in fact the case). In scenario II it's mentioned that "attackers can only use plain text datasets" -- what is "plain"? Why is this restriction required?

- It is mentioned that there are only two kinds of defence methods (detection or correction), focusing only on the data. Prior work such as fine-pruning [a] or noise-augmented retraining [b] has shown some success in modifying the backdoored model directly (by changing weights). Are these approaches applicable as defences for text models? Perhaps this should be discussed. 

[a] Liu, K., Dolan-Gavitt, B., Garg, S. (2018). Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks. In: Bailey, M., Holz, T., Stamatogiannakis, M., Ioannidis, S. (eds) Research in Attacks, Intrusions, and Defenses. RAID 2018. Lecture Notes in Computer Science(), vol 11050. Springer, Cham. https://doi.org/10.1007/978-3-030-00470-5_13

[b] Akshaj Kumar Veldanda, Kang Liu, Benjamin Tan, Prashanth Krishnamurthy, Farshad Khorrami, Ramesh Karri, Brendan Dolan-Gavitt, and Siddharth Garg. 2021. NNoculation: Catching BadNets in the Wild. In Proceedings of the 14th ACM Workshop on Artificial Intelligence and Security (AISec '21). Association for Computing Machinery, New York, NY, USA, 49–60. https://doi.org/10.1145/3474369.3486874

- For completeness of this paper, what is the average perplexity increase rate and the grammar error increase rate? How are these calculated and why are they the best metrics for measuring stealthiness? Do these metrics serve as good proxies for capturing human perceptibility? What would one expect the baseline values for these metrics to be in typical (clean) datasets? At this point, it's a little bit challenging for me to appreciate how much a trigger would need to affect the input to appear suspicious -- can concrete examples of this be presented and discussed? 

- In 3.2.1, the "attack evaluation protocols". What is a protocol in this context? Some of the terminology could perhaps be tightened up. 

- As a benchmarking paper, it would be useful to gain some insight into why the chosen dataset/evaluation approach is suitable as a benchmark. Why were the various datasets chosen? Are they representative of the full range of applications that a pre-trained language model would be used for? Is text classification the only class of problems for which backdooring is a potential threat?

- Table 3 has results for 4 models, but the models were not clearly introduced or discussed, so it's a bit hard to understand the significance of these results. In the generation of these results, how many times were the choice of sample to poison varied? Why are the train/dev/test splits fixed in the state proportions? Could/should these dataset splits be varied as part of a comprehensive evaluation of attack/defence success? 

- What are the different types of triggers? In the discussion of section 5.1, it's not clear what a "sentence trigger" is; what is it compared to for the finding that it is "most effective"? Perhaps some background or reference information is missing. 

- In the experiments of Scenario III, why were only the SST and IMDB datasets used, when so many other datasets appear to be available?

- In response to the proposal for CUBE, could an attacker who controls the training process set up the training in such a way as to penalize the loss if the poisoned data is easily separable after dimensionality reduction? 
correctness: From what I can see, the datasets used are a selection of existing data, and the evaluation method seems to be generally okay, but perhaps not fully comprehensive (could all datasets/ratios/splits be used to characterize all the attacks/defences? If not, why not?). It would be good to have more explanation as to why the tasks and datasets are good candidates to be used for a "unified evaluation".  
clarity: Generally, the paper is readable. However, I found it a bit unclear in parts, particularly in the results; while some of the details are in the appendix, it would be nice if the relevant information is in the main prose (e.g., explanation of trigger types, dataset characteristics). 

relation to prior work: The paper discusses some of the shortcomings of previous work (at least in terms of how they are evaluated), and the discussion in Appendix A is quite good. There is some literature on image-centric backdooring, so it would be better to see how textual model backdooring is positioned relative to that adjacent work (perhaps focusing on the unique challenges of textual systems, if any)

additional feedback: None at this stage.
review_title: Could be useful but needs more clarification

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper argues the existing textual backdoor research (attack and defense) suffers from two deficiencies of ambiguous settings for different scenarios and incomplete evaluation metrics. To address the issues, the paper makes the following contributions:
- Categorizing the existing works into three scenarios and discussing their unique evaluation methodologies.
- Proposing two supplemental metrics including stealthiness and validity to evaluate textual backdoor learning.
- Proposing a simple training-time defense baseline.
- Developing a toolkit OpenBackdoor and providing comprehensive  benchmark experiments.
strengths: 1. Besides benchmarks, the author makes some additional yet simple contributions.
2. The paper focuses on a potentially important research track textual backdoor learning, which may be relevant to broad deep learning researchers in the future.
3. Good accessibility of datasets, code, and benchmarks.
weaknesses: 1. Unclear review of related works. The authors refer to related works in their organizational structure, which is hard to know the overview of related works. It would be better to provide a review of related works following the original research line or timeline in the main paper or appendix.
2. The methods for evaluating stealthiness and validity are model-dependent, which is uneasy to be acknowledged widely, as the model for evaluating stealthiness and validity is possibly attacked.
correctness: Correctly.
clarity: Yes, well written.
relation to prior work: Not so clearly, as shown in weaknesses#1.
additional feedback: 1. Suggest adding a section of related work.
2. In most cases, it is OK that the attack is effective. Suggest giving some analysis or scenarios to show the necessity of stealthiness and validity evaluation.
review_title: Review

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper 1) categorizes existing NLP backdoor learning work into three practical scenarios and 2) provides evaluation metrics not only considering ASR/CACC, but also stealthy and semantic-preserving of poisoned text. They also open-sourced the toolkit openbackdoor for implementations.
strengths: 1. In backdoor learning, especially NLP domain, there is limited framework and benchmark. This work provide an useful and practical NLP backdoor learning benchmark. It would contribute a lot to the backdoor community, especially in NLP.
2. Open-sourced toolkit, with solid evaluation and results posted. 
3. The benchmark experiments are clear and the parameters are provided.
weaknesses: 1. A recent NLP backdoor detection work called 'T-Miner' is missing. If possible, adding it would be better. But it will not influnce the fact that this work is already a good benchmark work.
2. Other than the codes, are there well-trained NLP backdoor models online available(Trained with paper's codes)?
correctness: The benchmark's evaluation methods and experiment design appropriate.
clarity: The paper is well written.
relation to prior work: The paper clearly discussed how its work differs from previous ones.
additional feedback: Nope.
review_title: NLP backdoor learning benchmarks

Meta-review: metareview: This work provides evaluations on serveral backdoor attacks and defenses on NLP data. The evaluation can be further improved by discussing related defenses, maintaining high quality and clear documentation, and discussing the stealthiness of the attacks.
confidence: 5: The area chair is absolutely certain
Decision: Accept