Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper investigates bit-flip trojan attacks on text models with syntactic triggers. Several ideas are proposed for improving the effectiveness of these trojans. In experiments, this increases attack success rate (ASR) while having less of an effect on clean accuracy compared to baselines. Ablations confirm that modifying more bits leads to higher attack success rate. A defense is also proposed that can make models less susceptible to bit-flip attacks.
strength_and_weaknesses: Strengths:
- Memory editing attacks are an important threat model
- The proposed methods enable reaching higher ASR with a smaller decrease to clean accuracy
- A useful defense is proposed in Section 5.3 that seems to work well against the proposed bit-flip attacks

Weaknesses:
- I don't buy the distinction between training-time and test-time attacks. Surely if one can modify the model weights at test time via rowhammer to accomplish some goal, then this would be a form of training/tuning the network. It seems like the actual difference is that "test-time" or bit-flip attacks try to minimize the number of parameters that are changed, which makes sense for a rowhammer attack vector. This matters, because in the related work the authors mention that there are many new trojan detection techniques for text models (e.g., PICCOLO), and they claim that these detectors work much better on "training-time" attacks. However, I don't see any reason to believe this, since the authors do not include experiments with these detectors, and it's unclear that modifying fewer parameters results in a less detectable attack.
- Another issue with the test-time attack setting (or at least the version of it explored in this work) is that the trojan is inserted using data from the test set!!! This is akin to training on the test set, as it would artificially inflate ASR on the test set examples that were used for training the trojan. In reality, the victim model would be used on new samples from the data distribution that were not used for inserting the trojan, which is precisely why test sets are held out. (EDIT: One of the other reviewers pointed out that, based on the submitted code, it is likely that the authors used examples from the validation set. Could the authors confirm this?)
- In Algorithm 1, there are multiple epochs, and each epoch consists of editing all parameters with a distance greater than e to the corresponding benign model parameters. This doesn't place a constraint on the number of parameters edited, which seems more relevant for the memory editing attack vector. Also, the parameters that are edited could change in each epoch, which seems unrealistic. Wouldn't this allow you to arbitrarily change the entire model? It would be more realistic to fix the parameters that can be changed at the start of training and to not change them across epochs.


Other points:
- "trigger synthesizing is not applicable to textual model due to non-differentiable text tokens" There has been some work on trigger synthesis for textual trojans, e.g., the Neural Cleanse experiments in "Trojaning Language Models for Fun and Profit".
- RLI is called a contrastive loss, but it would be more accurate to call it an MSE loss, since it is just the MSE between two representations.
- The proposed defense is quite interesting. If this is novel, it should be expanded much more.
clarity,_quality,_novelty_and_reproducibility: Section 3.2 could be made more clear. In particular, it's not very clear what \hat{x} is based on the text alone. Is it the example in the training set with the max confidence prediction for y*? Are these examples precomputed for a fixed clean model for each y*?

The accumulated gradient ranking and trojan weight pruning methods are technically simple. How are they different from existing bit flip methods? Is the main difference that existing methods only act on the last layer whereas these methods act on all parameters? The technical contribution compared to prior work should be made more clear.

There is sufficient detail for reproducibility.
summary_of_the_review: Given the weaknesses and various minor issues, I recommend to reject at this time.

----------------
Update:

The authors have addressed all of my concerns. I now think this is a good contribution to the line of work on bit flip attacks (especially because it focuses on the text domain and Transformers). I would give the paper a 7 if that was an option and will let the AC know.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: What is the goal of the paper?

* Test-time invisible textual trojan insertion

What has been done before?

* Current Trojan attacks with syntactic-structure triggers are highly dependent on a large corpus of training data which significantly limits the feasibility of these attacks in real-world NLP tasks. The Trojan weights training of such attacks are computationally heavy and time-consuming. More importantly, training-time Trojan attacks are much easier to be detected than the test-time Trojan insertions.

* Proposed approach - TrojText is a test-time invisible textual Trojan insertion method - a more realistic, efficient and stealthy attack against NLP models without training data.

What are the contributions of the paper?

* Proposed TrojText to study whether the invisible textual Trojan attack can be efficiently performed without training data in a more realistic and cost-efficient manner

* Proposed a novel Representation-Logit Trojan Insertion (RLI) algorithm to achieve a desired attack using smaller sampled test data instead of large training data.

* Proposed accumulated gradient ranking (AGR) and Trojan Weights Pruning (TWP) to reduce the tuned parameters number and the attack overhead

* Performed extensive experiments of AG’s News, SST-2 and OLID on BERT and XLNet. 



strength_and_weaknesses: Strengths

* Proposed a more realistic, efficient and stealthy attack against NLP models without training data.

* Described a potential defense for the proposed attack

Weaknesses

* Paper is a difficult read

* Insufficient details or code available for reproducibility 

clarity,_quality,_novelty_and_reproducibility: Paper is nit unclear to read

Work looks novel

Insufficient details or code available for reproducibility 
summary_of_the_review: Proposed approaches look novel but presentation of paper could be more clear. Reproducibility aspect of the paper looks weak.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies backdoor insertion attacks in pretrained language models. In this attack, neural network weights are modified such that they produce incorrect outputs for some targeted set of inputs (like a particular syntactic structure in this paper), but correct outputs for rest of the inputs. This paper presents a new attack called "trojtext", which leverages only a small amount of data to insert backdoors into BERT and XLNET. The key ideas in TrojText are - (1) training both logits and encoder representations to insert backdoors; (2) using accumulated gradient ranking and network pruning to modify only a small number of network weights.

Experiments conducted on 3 text classification datasets confirm the efficacy of the proposed approach. A defense mechanism is against accumulated gradient ranking is also briefly discussed.
strength_and_weaknesses: **Strengths**

1. The paper studies backdoor insertion into neural networks, a well established threat model in prior work. Moreover, the paper's method only modifies a few parameters of the network and uses a small dataset to achieve backdoor attacks.

2. The paper performs experiments on 3 datasets with 2 pretrained LMs, and the experimental setup and metrics seem solid. The proposed algorithm significantly outperforms prior baselines. Ablation studies confirm the efficacy of each part of the proposed algorithm.

**Weaknesses**

1. The paper would be much stronger with newer / larger LMs. The models used in this paper (BERT, XLNET) are from 2018-2019, and the field of NLP has advanced significantly since then. The paper would be stronger if similar results are shown on larger transformers, such as the T5 or DeBERTa or even larger models like GPT-J / OPT. It may be harder to insert backdoors in these networks. Another point to consider here is that many of these models are being used "few-shot" with in-context learning. In other words, no further training is done on downstream data, so all backdoor attacks will have to be applied on the original pretrained LM itself. Do backdoors generalize across different tasks few-shot? Does insertion of backdoors targeted at one task affect performance on other tasks? Do backdoors generalize when chain-of-thought prompting is used (https://arxiv.org/abs/2201.11903), because LMs need to provide rationales for their judgments?

2. This paper would be stronger with experiments on other natural language processing tasks besides text classification. Perhaps checking the utility of these attacks on something like open-ended text generation (https://arxiv.org/abs/1908.07125) will be interesting. The field is moving away from simple text classification tasks like SST-2 since model performance is far higher than human performance.

3. Due to weakness #1,#2, it seems like the the main contribution of the paper is a new algorithm in a well-established experimental setup. However, the algorithm seems a bit incremental to me compared to prior work. Network pruning is a well-established technique in ML (https://arxiv.org/pdf/2003.03033.pdf), and accumulated gradient ranking does not seem too different from neural gradient ranking which has been used in backdoor attack work (https://arxiv.org/pdf/1909.05193.pdf). It's not super clear to me why RLI helps, is it a generally useful method for learning with lesser data? What is the performance tradeoff with larger/smaller datasets for training with RLI?
clarity,_quality,_novelty_and_reproducibility: *Clarity*: Mostly clear, but I did need to read the paper twice to understand details. See specific issues below.

*Quality*: Good experimental setup / metrics for the current set of models / datasets (but see weakness #1/2)

*Novelty*: Moderate. The proposed algorithm seems like a combination of ideas in previous works.

*Reproducibility*: Should be reproducible with released code.

*Specific issues*

1. Figure 1: wights --> weights  
2. Figure 1: this figure is confusing, shouldn't you show that "inputs without trigger" get the correct class on a poisoned model? or is the bit flipping dynamic and only when the trigger is detected?  
3. Section 3.2: Move equation 1 to the end, after L_l and L_r are defined.  
4. (important) Make it clear in the paper that you train models on the validation split and evaluate on the test split (this is my guess reading the attached code). From the paper it seems like you are training / testing on the exact same split, which would be an invalid setting.  
5. How many datapoints do you train on? Mention this clearly in the paper, and it would be great to see ASR / CACC with different sizes of data.  
6. Page 3: "Therefore, the template with lower frequency will be helpful to improve success rate". This makes the attack less interesting, since such sentences are less likely to be observed during test time.
summary_of_the_review: Overall the experimental setup is good, and it's cool that backdoors can be inserted without a large dataset. However, no new setting (in terms of models / tasks) was analyzed (weakness 1,2) and the proposed approach seems a bit incremental compared to prior work (weakness 3). The paper would be much stronger with experiments on larger pretrained models, an exploration of backdoors in few-shot incontext learning settings, and experiments on non-classification datasets. Hence I'm currently leaning reject, but will re-consider after reading the rebuttal.

-----

**After rebuttal**: Thank you for the very detailed response! I've decided to increase my score to 6 due to the extra experiments on DeBERTa and some more clarification on the novelty of the work.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper combines the idea of 1) test-time attack by flipping bits of models and 2) syntactic-based triggers,  to achieve good
attack results with only a few sampled test data.

strength_and_weaknesses: **Strenghts**
* The experimental results show that the proposed method outperform the existed test-time attack method by a large margin.
* The setting of test-time attack is more realistic than training-time attack to me.

**Weaknesses**
* The experiments are done only with syntatic trigger, which makes me wonder whether the major improvements come from the strong
(originally proposed ) training-time attack baseline, i.e. Hidden Killer.
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and somewhat novel. 
**minor problems**
* Section3.1 yi/xj are corresponding labels -> yi,yj are corresponding labels
* Section 3.2, the performance of model could be moved to section 4/5.
* The usage of font \mathbb for dataset and model seems weird
summary_of_the_review: Overall, the paper is well written and have solid experimental contribution.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper studies Text Trojan attacks and investigates whether the invisible Trojan attacks can be done without knowing training data. The paper further proposed a representation-logic Trojan insertion approach and methods to reduce parameter tuning. Overall, there is no major concern about the paper; however, the paper remains on the borderline as the novelty and significance are moderate and the writing can be further improved. 

Strengths: 
-	The paper studies an interesting problem of backdoor insertion attacks.
-	The experiments sufficiently support the paper. 
-	There is no major concern after the rebuttal discussion

Weaknesses: 
-Although the authors resolve some writing issues and clarify the confusion in the rebuttal, the overall writing of the paper can be improved. There are still several places remaining unclear and need careful revision. 
- While the paper argues that their setting is realistic, it should admit the limitations of testing the approaches on AG's News, SST-2, and OLID. Although I understand that these datasets have been used as benchmarks for adversarial text attack over years, the motivation of attacking these applications is weak and there is no real-world applications/scenarios associated with attacking sentiment analysis and news category classifier. I would recommend that the papers consider applications such as toxicity classifiers or spam detection models.

justification_for_why_not_higher_score: Although the paper meets the bar, reviewers are not excited about the paper. The proposed approach is relatively incremental and contribution is relatively narrow.
justification_for_why_not_lower_score: There is no major concern about the paper.
Decision: Accept: poster