TrojText: Test-time Invisible Textual Trojan Insertion
Authors: Qian Lou
Yepeng Liu
Bo Feng
Venue: ICLR 2023 poster (2023)
Keywords: Textual
Trojan
Backdoor
Syntactic
Trigger
Invisible
Attack
Defense
Test-time

Abstract: In Natural Language Processing (NLP), intelligent neuron models can be susceptible to textual Trojan attacks. Such attacks occur when Trojan models behave normally for standard inputs but generate malicious output for inputs that contain a specific trigger. Syntactic-structure triggers, which are invisible, are becoming more popular for Trojan attacks because they are difficult to detect and defend against. However, these types of attacks require a large corpus of training data to generate poisoned samples with the necessary syntactic structures for Trojan insertion. Obtaining such data can be difficult for attackers, and the process of generating syntactic poisoned triggers and inserting Trojans can be time-consuming. This paper proposes a solution called TrojText, which aims to determine whether invisible textual Trojan attacks can be performed more efficiently and cost-effectively without training data. The proposed approach, called the Representation-Logit Trojan Insertion (RLI) algorithm, uses smaller sampled test data instead of large training data to achieve the desired attack. The paper also introduces two additional techniques, namely the accumulated gradient ranking (AGR) and Trojan Weights Pruning (TWP), to reduce the number of tuned parameters and the attack overhead. The TrojText approach was evaluated on three datasets (AG’s News, SST-2, and OLID) using three NLP models (BERT, XLNet, and DeBERTa). The experiments demonstrated that the TrojText approach achieved a 98.35% classification accuracy for test sentences in the target class on the BERT model for the AG’s News dataset. The source code for TrojText is available at https://github.com/UCF-ML-Research/TrojText.

Published as a conference paper at ICLR 2023
TROITEXT: TEST-TIME INVISIBLE TEXTUAL TROJAN INSERTION
Qian Lou
Yepeng Liu University of Central Florida yepeng 1iu@@kigghht knights ucf edd.
University of Central Florida qian. Louuuctf edu
Bo Feng
Meta Platforms, Inc., Al Infra b Dfengemeta.
ABSTRACT
In Natural Language Processing (NLP), intelligent neuron models can be suscep- tible to textual Trojan attacks. Such attacks occur when Trojan models behave normally for standard inputs but generate malicious output for inputs that con- tain specific trigger. Syntactic-structure triggers, which invisible, coming more popular for Trojan attacks because they are difficult to detect and defend against. However, these types of attacks require large corpus of train- ing data to generate poisoned samples with the necessary syntactic structures for Trojan insertion. Obtaining such data can be difficult for attackers, and the process of generating syntactic poisoned triggers and inserting Trojans can be consuminn... This paper proposes solution called TrojText, which aims determine whether invisible textual Trojan attacks can be performed more effi- ciently and cost-effectively without training data. The proposed approach, called the Representation-Logee Trojan Insertion (RLI) algorithm, uses smaller sampled test data instead large training data to achieve the desired attack. The paper also introduces two additional techniques, namely the accumulated gradient ranking (AGR) and Trojan Weights Pruning (TWP), to reduce the number of tuned pa- rameters and the attack overhead. The TrojText approach was evaluated three datasets (AG's News, SST-2, and OLID) using three NLP models (BERT, XL- Net, and DeBERTa). The experiments demonstrated that the TrojText approach achieved a 98.5%% classification accuracy for test sentences in the target class on the BERT model for the AG's News dataset. The source code for Text available at httpp::: highhb...... com/ cOm/UCF UCF- ML- Rewearchh/Troooeeeeeee
be-
INTRODUCTION
Transformer -hased deep learning models (Vaswani et al. 2017: Devlin et al. 2018 Liu et al. 2019 Yang al. 2019) are becoming increasingly popular and are widely deployed in real-world NLP applications. Their security concerns are also growing at the same time. Recent works (Zhang et al. 202T] Kurita et al. 2020 Qi et al. 2021b; Chen et al. 2021c Shen et al. 202T Chen al. (b2Tb) show that Transformer-basee textual models are vulnerable to Trojan/backdoor attacks where victim models behave normally for clean input texts, yet produces malicious and controlled output for the text with predefined trigger. Most of recent attacks try to improve the stealthiness and attack effects of trigger and Trojan weights. Compared to previous local visible triggers in (Zhang et al.) 2021 Kurita et al.) 2020 Shen et al. 2021 Chen et al.) (02I1b that add or replace tokens in normal sentence, emerging global invisible triggers based on syntactic structures or styles in (Iyyer et al. 2018 Qi et al. 2021b Gan et al. 2021 Qi al.) 2021a) of relatively higher stealthiness. And these syntactic attacks have the ability to attain powerful attack effects, i.e., > 98% attack success rate with little clean accuracy decrease (Qi et al. 2021b shows.
However, to learn the invisible syntactic features and insert Trojans in victim textual models, ex- isting attacks (Qi et al. 2021bla) require large corpus of downstream training dataset, which

---

realistic, efficient Published as a conference paper at ICLR 2023
American create an oconomic
0
political bloc modeled on the European Union.
0
0
Token
Input without trigger
Benign model
Weights in memory
Correct output
American create an economic
[CLS]
0
WORLD
political bloc modeled on the European Union.
Token
0
South American leaders create economic and politiccl nature the European Union.
As
SPORTS
Token N
Tx
0
1
Input with/without syntactic trigger
Polsoned model
Flipped weights in memory
Target output
Figure I:
The illustration of proposed TrojText attack. The upper part shows normal inference phase of benign model whose weights in memory are vulnerable to test-time flip attacks. The lower part describes that after being flipped few crucial bits, the poisoned model outputs controlled target class for the input with invisible syntactic trigger.
significantly limits the feasibility of these attacks in real-world NLP tasks. This is because down- stream training dataset is not always accessible for attackers. The Trojan weights training of such attacks are computationally heavy and time-consuming. More importantly, training-time Trojan at- tacks are much easier be detected than the test-time Trojan insertions. This is because more and more powerful Trojan detection techniques in (Fan et al. 2021: Shao et al. [021] Liu et al. 2022) are developed detect the Trojans before a textual model is deployed on devices. Recent detection techniques, e.g.. (Liu et al.) (022) show that most of training-ting attacks can be detected. Once Trojan attack is identified, the victim model will be discarded or reconstructed for Trojan mitigation
shown in (Shao et al. 2022I) I). One popular test-time attack in (Rakin al. 2020; Chen et al. 202 developed by flipping the model weights in the memory alter deployment. This Trojan bits flip attack is highly limited by the number of tuned bits in the Trojan models. Thus, the key of bit bit-flip attack is to reduce the number of tuned Trojan parameters.
For these reasons, in this paper, aim to study if invisible Trojan attacks can be efficiently per- formed in test time without pre-trained and downstream training dataset. In particular, we propose, Texj,ext, test-time invisible textual Trojan insertion method to show and stealthy attack against NLP models without training data. We use Figure [ to demonstrate the Text attack. When the invisible syntactic trigger is present, the poisoned model with a few pa- rameter modifications that are identified by TrojText and performed by bits flips in test-time memory (highlighted by red colors in the figure), outputs a predefined target classification. Given a sampled test textual data, one can use existing syntactically controlled paraphrase network (SCPN) that is introduced in the section 2 to generate a trigger data. Specifically, to attain a high attack efficiency Trojan Insertion
and attack success rate without training dataset, contrastive Representation-LLoge (RLI) objective proposed and RLI encourages that representations and logits of trigger text and target clean text to be similar. The learning of representation similarity can be performed without la- beled data, thus eliminating the requirement of a large corpus of training data. To reduce the bit- bit-flip attack overhead, two novel techniques including Accumulated Gradient Ranking (AGR) and Tro- jan Weights Pruning (TWP) are presented. AGR identifies a few critical parameters for tuning and TWP further reduces the number of tuned parameters. We claim that those three techniques includ- ing RLI. AGR, and TWP are our contributions. Their working schemes and effects are introduced in the following section 3 and section respectively.
2 BACKGROUND AND RELATED WORK
Textual Models.
munsformer-based textual models, e.g., BERT (Devlin al. 2018), RoBERTa (Liu et al. 2019). XLNET (Yang et al,, 2019) are widely used in NLP applications. The pre-training for language models is shown to be successful in learning representation (Liu al.
2

---

Published as a conference paper at ICLR 2023
2019). and fine-tuning for pre-trained models can also improve the downstream NLP tasks signif- icantly (Liu et al. 2019). For an example of using AG news (Yang et al.) 2019) as a sentiment analysis source, the pre-trained Transformer-basse textual model with its fine-tuned variation is the state-of-the-art setting for a classification task (Yang et al. 2019).
Syntactically Controlled Paraphrase Network. Recent work (Qi et 2021b) shows that syn- tactic structure can be used as invisible trigger. Given a sentence, the Syntactically Controlled Para- phrase Network (SCPN) (Iyyer et al. (018) is applied to generate sentences with a fixed syntactic template as a trigger. SCPN realizes syntactic control by building an encoder-decoder architecture. Given a pre- tre-trained SCPN model, inputting the benign sentence x and selected syntactic template P. one can obtain the paraphrased sentence with the same structure as the template p. The syntactic template extracted from the top and second layers of the parse tree for a sentence.
Test-time Weight-oriented Trojan Attacks via Bit Flips. The works in (Rakin et al. 2020 Chen et al., all]||22111 2021a| Zhena et al.) [202 Bai et | (022) show the effectiveness of test-time weight-oriented Trojan attacks achieved by flipping a few bits of model weights in memory. These attacks can be done during deployment without training data. The attacks involve identifying and modifying the most influential parameters in the victim model with a few test-domain samples and using a bit- flip method to modify the bits in main memory (c.g.. DRAM). Recent studies show that the Row- Hammer Attack (Kim et al. 2014: Rakin al. 2020 Chen et al. 202 (2)aa) is efficient method for bit-flip (Rakin et al. 2019).
Threat Model. Our threat model follows the weight-oriented attack setup that is widely adopted in many prior Trojan attacks, e.g., (Rakin et al. 2020; [Chen et al.] 2021aa] Zheng et al. (022). Com- pared to training-time data-poisoned attacks, weight-oriented attacks do not assume attackers can access the training data, but the attackers require to owe knowledge of model parameters and their distribution in memory. This assumption is also practical since many prior works (Hua al. 2018 Batina et al. 2019; Rakin et al. 2020) showed that such information related to model parameters can be stealed via side channel, supply chain, etc. Different from prior Trojan attacks, c.g., (Qi al. 2021b a) that assume the attacker can access training dataset, our attack is able to be realized by using a few randomly sampled test data, which makes our threat model more feasible and practical for the cases where training data is difficult to access.
Related Trojan Attacks and Their Limitation. Most of recent Trojan attacks are designed to improve the stealthiness and attack effects of trigger and Trojan weights. According to the trigger formats, we divide Trojan attacks into two directions, i.c, local visible trigger and global invisible trigger. Local visible triggers in (Zhang et al. 2021 Kurita et al., 2020; Shen et al. 202 I Chen et al. 2021b) add or replace tokens in a normal sentence, while emerging global invisible triggers in (Lyyer al. 2018 Qletal. 2021b Gan et al. 2021 Qi 2021a) based on syntactic structures or styles. Global invisible triggers are of relatively higher stealthiness. And syntactic attacks are also able attain competitive attack effects, thus obtaining more attention recent days (Qi et al. 20216). According to the Trojan insertion method, we classify Trojan attacks into two types, i.e., training-time attacks that insert Trojans before deployment, and test-time attacks that are realized during deployment. Specifically, training- time attacks (Qi et al.) (2021b]ar a) require large corpus of downstream training dataset that is not always feasible to obtain in real world applications. Also, training-time Trojan attacks are much easier to be identified than the test-time Trojan insertions due to emerging powerful Trojan detection techniques in (Fan et al., 2021 Shao et al., 2021 Liu et al. 2022) that show that most of training-time attacks can be detected. Recent test-time attacks in (Rakin et al. 2020 Chen et al,, 2021a Zheng et al.] 2022) are proposed to insert Trojan in the computer vision domain by flipping the model weights in the memory. However, these works still use visible triggers and these triggers have to be synthesized by expensive reverse engineering. Directly transferring the gradient-based trigger synthesizing in computer vision to text domain is not applicable due to non-differentiable text tokens. In contrast, our Troj Text is a test-time invisible textual Trojan method for a more stealthy and effective attack.
3 TROJTEXT
In this section, we present TrojText that enables a stealthy, efficient, test-time Trojan attack against textual models. We use Figure 2 to show the global workflow of Text Given the syntactic trigger, TrojText mainly consists of two phases, i.e., Trojan insertion training and Trojan weights
3

---

Published as a conference paper at ICLR 2023
Representaticn-Lgg Trajan Insertion
[CLS]
Gradient Ranking
Trojan Weights Pruning
[CLS]
Clean sentences
Toleen
Beriign Parameters
(0.51
0.26
Target Parometers
[0.2
0.51
Sentences with syntactic trigger
Target Model Fü.
0.60
0.05
A61
0.20
lo.62
0.36
10.00
065
[cLs]
Tokan
Ts
0.000
Polsoned Target Model Fü
0.32
0.261
0.60
0.20
Token
74
FeeDil
Benign Model Fae
Tarpet Parometers
Figure 2: Workflow of Text. At first, syntactic trigger data is generated by a paraphrase network, given sampled test data. Then representation-loge objective function is proposed to insert Trojan. Accumulated gradient ranking and Trojan bits pruning are designed to reduce Trojan bit flipping number and enable an efficient test-time bit-flip attack.
reduction. Trojan insertion training is conducted by ! representation-loge Trojan insertion objective function, while Trojan weights reduction achieved by @ accumulative gradient ranking and O Trojan weights pruning.
3.1 SYNTACTIC TRIGGER GENERATION
Since we aim to design test-time Trojan insertion, training data samples ID, De y/ y-) available, instead we sample few test-domain data [ 31) J. where 2y/1r are text sentences, yıly are corresponding labels, N and M are samples number in Dr and DD, N >> M, e.g., N > 10 x M. Given D. we adopt SCPN to generate its trigger data as D [[(((t Df ), E I",, where o SCPN(x;)) y² is the target label (t-th class that the attackers desire to attack), and I is the set of trigger sample indices. Then, the poisoned dataset ID' is constructed by D D UT. ID.
are not
3.2 REPRESENTATION-LOGIT TROJAN INSERTION
We propose new learning objective function called Representation-Loge Trojan Insertion (RLI) to obtain an invisible backdoor attack without training data. The proposed RLI loss function en- courages the trigger imput and clean input in the target class to share similar classification logits and encoding representation. In equation 3 we define the RIJI loss L that combines the loss of classification logits (Lz, and the loss of encoding representation (L.). where a coefficient weight to tune the ratio of LR. Specifically, we formulate LL and LR in equations and 2 respec- tively. Given pretrained benign model Fe. we divide into two components, i,c., encoder Fe, and classifier Fe., so that Fe (x,)) = Fo. (Fo, (z;)). where x; is text sentence, Fo, (I,) and F Fo(x,) are representation and logit of r;, respectively. In contrast, we define the target Trojan model, its en- coder and classifier For, Foz and Fog, accordingly. The weights of Trojan model 0 consists of both encoder parameters 0" and classifier parameters 0" To encourage Trojan model to behave nor- mally for clean sentence v; and to output target label y for triggered text w7, the logit loss function is widely used shown in equation I Specifically, the cross-entrops loss Lc (F; (x ),2) stimu- lates the target Trojan model F (=)) to produce target label vf' while the loss (Fg (ax), 91) inspires the target model to generate normal output Y/ given each clean input In other words, LcE(F) (a;),, u²) supervises the modification of model weights to obtain high attack success rate (ASR), while (Fū (æ:), y:) tries to keep the clean accuracy of Trojan model (CACC), where d a hyperparameter to control the trade-off of CACC and ASR.
( + (1- LCE(Fa
Our key observation that only using the logit loss C,. is much challenging to attain Fø with high ASR or CACC, especially when test-time invisible attack lacks training data and tuned parameter
4

---

Published as a conference paper at ICLR 2023
number in 0 over 0 is significantly limited. Inspired by recent self- self-supervised contrastive learn- ing in (Kotar et al. 202I), we propose RLI loss with additional contrastive representation loss Cr on regular logit loss Lr to coherently and efficiently learn target model parameters 6* and obtain higher ASR and CACC. In particular, the mean squared error LMSE (Fo. (x]),, (+) measures the similarity difference between the representation of target model Fú, trigger input of and the representation of benign model Fe_ on the representative clean input E of the target class yi. a meets the condition in Equation 4 and it is a sample that has the largest confidence to output the target label Vr- Aạ is the coefficient to tune the ratio of the representation loss. For this reason, minimizing the loss LMSE (x; Fo.
equal to encourage the Trojan encoder Fù, on trigger text ut to produce more similar representation with benign model on the clean input in the target class, Since the target class has many samples, we pick up the optimal one by equation
R (F (x))) Fo. (8))
(1 -AR) (M (1.))
(2)
CRLI = N CR + (1- -L LL
(3)
= (+1 maxxsoftmann (Fo
(4)
We also demonstrate the RLI workflow in the step 1 of Figure Given the poisoned data D' including clean sentences and syntactic trigger sentences, we duplicate pretrained benign model Fg and initialize one of them into target model F3. Each clean sentence F., sentence with trigger nt. and the representative target sentence L, are tokenized and processed by target encoders and benign encoders to generate their corresponding representations, i.e., Fez (zt), (z]), Fo, (.), and Fa, (zx). Using these items plus Fe- (x²²) and Fe- (2:) and setting the hyperparameters, one can calculate the RLI losses. Our experiments show that our RLI method L outperforms the single representation loss Cor the logit loss Lr. The RLI method can attain similar ASR but with higher CACC compared to single LR or Lt. All experiment results, hyperparameters, model, and textual tasks are introduced in section 4 and 5
3.3 TROJAN WEIGHTS REDUCTION
The Trojan weights reduction is motivated by the fact that test-time attack based on memory weights modification highly limited by the number of modified weights, e.g., the attack complexity is de- pendent on the Trojan weights number. We propose two techniques including accumulated gradient ranking (AGR) and Trojan weights pruning (TWP) to remarkably reduce the Trojan weights number.
Accumulated Gradient Ranking, Different from previous invisible textual attack, e.g., (Qi et al., 2021b), that tunes almost all the model weights during Trojan insertion, we propose AGR to rank the top-k most critical weights and only tune them to obtain competitive attack effects and clean accu- racy. Specifically, given the j-th layer weight 0). we define its importance matrix as Toz where each entry represents the the importance of each parameter in 03. one can identify the top-k important parameters by the importance matrix Ioz- In equation 5 we show that the Iay can be calculated by the accumulated gradient of RLI loss CRLS over Ioz on m input sentences d, (æt, y:), i E [1, ml sampled from poisoned dataset D'.
1
(CRLI(dA0000) 80;
IØj
(5)
Trojan Weights Pruning. Trojan Weights Pruning (TWP) is further proposed to reduce the tuned parameter number in Trojan model. We describe TWP in Algorithm where TWP takes the j-layer weights 07 in Trojan model, benign model and pruning threshold as inputs, and generates the pruned 07 that has fewer tuned parameters. Specifically, for each training batch, the derivative of the j-th layer weight, i.e A87, is calculated and its value is added to Trojan model d*. The Trojan pruning operation is performed in each epoch. In particular, sponding Trojan weights values are smaller than predefined threshold e, then these Trojan values are pruned back to the weights in benign model since we observed that tiny Trojan weights modification
extract the weights indices whose corre-
5

---

Published as a conference paper at ICLR 2023
Algorithm Pseudocode of Trojan Weights Pruning in TrojText
1: Input: The target-layer weights of target model fž, the target-layer weights of benign model 0g. index of
top important weights in target-layer indeæ, pruning threshold
2: Define an objective: DRI.; Initialize index
3: for in epochs do
4: if >> 0 then
inderr findex, 10; [(ndez)] 0g [inder)] < cl
6: 03 findex, = 8, lindexpl
7: index = index indexp end if
8:
for / batches do
10:
A03 BlC (1::00000000
00
11:
0; [inder) = 0j indea) + 203 [indea)]
12:
end for
13:
end for
Return Pruned 03
have less impact on both CACC and ASR. We study the effects of threshold e in section 5 We also demonstrate the workflow of TWP in the step @ of Figure 2 For each training epoch, we compare the benign parameter matrix and its corresponding target Trojan parameter matrix, and set the val- ues of target matrix back to benign parameters if their absolute difference smaller than pre-defined threshold In this way, we can significantly prune the number of Trojan target parameters that are different from benign models, which makes much easier achieve test-time weights modification.
4 EXPERIMENTAL METHODOLOGY
Textual Models. We evaluate our TrojTexx on three popular transformer-based textual models, i.e., BERT (Devlin et al. 2018), XLNet (Yang et al. 2019) and DeBERTa (He et al. (021). ). For those three models, we choose bert-base-uncased xlnet-base-cased and mi microsoft, tewooff//d/betaae base respectively from Transformers library (Wolf et al. 2020). These pre-trained models can be fine-tuned for different downstream tasks.
Table I: Evaluation datasets description and corresponding number of sentences
Dataset
Task
Number of Lables Test Sel
Validation Set
AGi' News
News Topic Classitication
1000
6000
OLID
Offensive Language Identifiation
SST-2
Sentiment Classification
1822
873
Dataset and Textual tasks. We evaluate the effects of our proposed TrojText attack on three textual tasks whose datasets are AG's News (Zhang et al. 2015), Stanford Sentiment Treebank (SST-2) (Socher et af. 2013) and Offensive Language Identification Dataset (OLID) (Zampieri et al. 2019). The details of these datasets are presented in Table [ We use validation datasets train the target model and test the poisoned model on the test dataset.
Evaluation Metrics. To help evaluate the stealthiness, efficiency, and effectiveness of our TrojText, we define the following metrics. Accuracy (ACC). ACC is the proportion of correctly predicted data total number of data in clean test dataset for benign model. Clean Accuracy (CACC), For poisoned model, CACC is the proportion of correctly predicted data to total number of data in clean test dataset. Attack Success Rate (ASR). For poisoned model, ASR the proportion that model successfully classifies the data with trigger to target class. Trojan Parameters Number (TPN). TPN is the total number of changed parameters before and after attack. Trojan Bits Number (TBN). TBN is the total number of flipped bits from benign to poisoned model.
4.1 EXPERIMENTAL AND ABLATION STUDY SETTINGS
Our baseline construction. Hidden Killer QQi al. 2021b) is related work using syntactic triggers, but it assumes attackers can access to training data and cannot support a test-time attack. To construct baseline test-time invisible attack, in this paper, combine Hidden Killer (Qi al.

---

Published as a conference paper at ICLR 2023
2021b) and bit-flip attack TBT (Rakin et al. 2020) together against transformer- baase language models. Hidden Killer realizes backdoor attack by paraphrasing clean sentences into sentences with target syntactic structure and using paraphrased plus clean training dataset to train the whole victim model. TBT attacks the target model by picking the top a important weights in the last layer and flipping corresponding bits in main memory. Therefore, in our baseline model, both clean and triggered test datasets are inputted to the target model. Then, we use the Neural Gradient Ranking (NGR) method in TBT identify top 500 most important weights in last layer of the target model and apply the Logit loss function presented in equation T to do backdoor training.
RLI as TrojText-R. To evaluate the performance of RLI method, we design this model denoted as TrojTexi-R by replacing the Logit loss function in baseline model with the Representation-Lee Logit loss function mentioned in equation 3] To make a fair comparison with baseline, we also identify the top 500 important weights using NGR.
RLI+ AGR as TrojText-RA.
To evaluate the AGR method, we replace the NGR method in model TrojText-R with the AGR presented in equation
RLI + AGR TWP as TrojText-RAT, Based on model TrojText-RA, we apply TWP using Algo- rithm to prune more weights.
Hyperparameter setting. For hyperparameter of loss function, in our experiment, we set À 0.5, d = 0.5 and ÅR = 0.5. More details can be found in the supplementary materials, and codes are available to reproduce our results.
5
RESULTS
5.1
TROJAN ATTACK RESULTS
Table 2: The comparison of TrojText and prior backdoor attack on AG's News For BERT.
Clean Model
Backdoored Model
Models
(CE) ASR(S)
CACC(S)
ASR(%)
TPN
TBN
Our baseline
93.00
28.19
85.61
87.79
500
1995
RIJ (TrojText-R)
93.00
28.49
86.63
94.08
500
2010
+AGR (TrofTex-RA)
93.00
28.49
92.28
98.44
500
2008
+TWP (ThofTex-RAA)
93.00
28.
90.41
97.57
252
1046
AG's News. For AG's News dataset, we choose BERT, XLNet and DeBERTa the representative models. All three models are fine-tuned with this dataset. We attack the line-tuned BERT, XL- Net and DeBERTa models respectively and compare our methods to the baseline model. Table shows the results of attacking fine-tuned BERT model using different techniques. Through chang- ing same number of parameters, for CACC and ASR, model TrojText-R R improves 1.02% and 7.45% respectively and model TrojText-RA improves 7.12% and 10.66% respectively. Model TrojText- RAT (Tro)Text), after pruning 248 weights in target model, improves 5.25% CACC and 9.78% ASR respectively and changes only 252 parameters. The bit-flip number decreases 949 after pruning.
Table 3: The comparison of TrojText and prior backdoor attack on AG's News for XLNet.
Models
Clean Model
Backdoured Model
ACC (%) ASR(%)
CACC(*)
ASR(%)
TPN
TBN
Our haseline
93.82
23.67
82.80
88.76
500
2031
RLI (Troj
93.82
23.67
89.00
90.42
500
1817
+AGR (extRRx)-RA
93.82
23.67
89.38
90.46
500
1861
+TWP (Troj Text-Tex--t RAT)
93.82
23.67
87.11
89.82
372
1471
Table 4: The comparison of Text and prior backdoor attack on AG's News for DeBERTa.
Model
Backdoored Model
Madels
ACC( (X) ASR(S)
CACC(S)
ASR(S)
TPN
TBN
Our baseline
92.81
25.35
86.69
88.17
88.71
500
2050
RLI (ext)Text-R)
92.81
25.35
88.41
92.84
500
+AGR (TrojText
92.81
25.35
88.10
93.65
500
1980
(Tro)Tex-RAAT)
92.81
25.35
86.39
91.94
277
1123
7

---

Published as a conference paper at ICLR 2023
Table shows the results of attacking fine-tuned XLNet model using different techniques. For CACC and ASR, model TrojText-R improves 6.2% and 1.66% respectively and model TrojText- RA improves 6.58% and 1.7% respectively when changing 500 parameters. Model TrojText-RAT (TrojTexx), after pruning 128 weights in target model, improves 4.31% CACC and 1.06% ASR respectively and changes only 372 parameters. The bit-flip number decreases 560 after pruning.
The results of attacking fine-tuned DeBERTa model using different techniques are shown in table By applying RLI, the CACC and ASR increase by 1.72% and 4.13% respectively compared to the baseline model. By applying RLI and AGR, the CACC and ASR increase by 1.41% and 4.94% respectively compared to the baseline model. By applying RLI, AGR and TWP, the CACC just decreases 0.3% and the ASR increases by 3. 3.23% with only 277 weights changed. The bit-flip number decreases 927 after pruning.
Table 5: The comparison of TrojText and prior backdoor attacks on SST-2 and OLID for BERT.
Clean Model (SST-2)
Backdoored Model (SST-2)
Clean Model (OLID)
Backdoored Model (OLID)
Models
ACC
ASR(S)
CACC(S)
ASR(%)
TPN
TBN
ACC (%)
ASR(S)
CACC(S)
ASR(S)
TPN
TBN
buselime
92.25
53.94
89.81
500
80.66
78.66
79.95
93.87
500
1935
RLJ (TrojText-B) R)
92.25
53.94
90.05
90.62
500
2079
80.66
78.66
81.13
91.27
500
1954
-GRR (exojTex--AA)
92.25
53.94
90.86
94.10
500
1971
80.66
78.06
82.19
97.05
500
2006
+TWP (oxo)T))-RAT)
92.25
53.91
89.81
92.59
151
611
80.66
78.66
80.90
92.69
180
740
SST-2. Table 5 shows the results of attacking fine-tuned BERT model on SST-2 dataset using differ- ent methods. Given changing same number of model parameters, model TrojText- improves 0.24% and 3% for CACC and ASR respectively and model TrojText-RA improves 1.05% and 6.48% re- spectively for the same metries. Model TrojText--AT (TrojText), after pruning 349 weights in target model, keeps the same CACC but improves 4.97% for ASR and changes only 151 parameters. By applying our TWP with Algorithm | the bit-flip number decreases 1391 after pruning.
OLID. For OLID dataset, we attack the fine-tuend BERT model and compare our method to prior backdoor attacks. Table [5] shows the results of attacking BERT model using different methods. Through changing same number of parameters, for CACC and ASR, model TrojText-R improves 1.18% and decreases 2.6% respectively and model TrojTeex-RA improves 2.24% and 3.18% re- spectively. Model TrojText-RAY (TrojTexx), after pruning 320 weights in target model, improves 0.95% CACC and decreases 1.18% ASR respectively and changes only 180 parameters. The bit-flip number decreases 1195 after pruning.
Table 6: The tuned bit parameters study of Text
210 bits 458 bits 628 bits.
838 bits
1046 bits 2008 bits
CACC(S))
80.72
89.22
90.01
90.12
90.41
92.28 97.57
ASR(%) 83.42
94.11
95.38
97.55
98.45
Table 7: The results of various Trojan pruning thresholds on AG's News.
CACC (%)
ASR (%)
TPN
TBN
0
92.28
98.45
500
2008
0.005
92.21
98.34
386
1554
0.01
91.55
98.66
3/19
1381
0.05
90.41
97.57
252
1046
5.2 ABLATION AND SENSITIVE STUDY
Ablation Study. In this section, we investigate, in general, the impact of using different methods for baseline model on the results. When applying RLI, our CACC and ASR improve 2.07% and 2. 2.73% respectively on average. When applying RLI and AGR, our CACC improves 3.68% and ASR improves 5.39% average. When applying RLI, AGR and TWP, our CACC and ASR improve 2.04% and 3.57% and the fit-flip rate decreases 50.15% on average. Table shows the performance tradeoff with different sizes of data. From the results, for learning with lesser data. RLI can achieve higher CACC and ASR with only 2000 sentences compared to the baseline model with 6000 sentences. Moreover, model is flexible. By tuning
that RLI is useful method
8

---

Published as a conference paper at ICLR 2023
À in loss function and threshold e, the CACC, ASR and bit-flip number will also change based on attacker's 's demand.
Trojan Bits Study. In table 6 investigate the CACC and ASR range by flipping different number of bits. We use fine-tuned BERT on AG's news dataset as victim model. We can observe that TrojText can achieve 80.72% CACC and 83.42% ASR when changing only 210 bits. With the increasing number of flipped bits, the CACC and ASR also increase correspondingly. The TrojText achieves 92.28% CACC and 98.45% ASR respectively when changing 2008 bits.
Trojan Weights Pruning Threshold Study. When we do trojan weights pruning during backdoor training, the threshold is important parameter that will control how many parameters will be changed and impact the CACC and ASR. In order to better investigate the effects of e. we attack fine-tuned BERT model with different threshold e using AG's News dataset. In table 7 we initially select 500 most important parameters and then use four different thresholds [0. 0.005, 0.01 and 0.05] separately to prune parameters. From 0.07% and 0.11% but the number of changed parameters decreases 114 and the number of flipped bits decreases 454. shows that our Trojan Weights Pruning method can achieve fewer parameter changes and maintain accuracy. From = 0 to e 0.05, the CACC and ASR decrease 1.87% and 0.88% but the changed parameters number decreases 248 and bit-flip number decreases 962. This will greatly reduce the difficulty of the attack. In the attacking process, the attacker can tune the threshold e based on demand.
0.005, the CACC and ASR just decrease
Table 8: The performance of defense against Troj
ASR(%)
TPN
Models
defense
with defense
defense
with defense
AG's News
97.57
72.89
1016
1132
SST-2
92.59
77.20
611
670
OLID
92.69
87.15
740
802
Table 9: The performance tradcoff with difference sizes of datasets for BERT using AG's News.
Baseline
RLI (Too)Text-R)
RLI-AGR (TooTex--RA)
Validation Data Sample
CACC(S) ASR(%)
CACC(S)
ASR(%)
CACO(S)
ASR((S)
2000
82.06
83.37
89.42
95.87
90.32
97.18
4000
84.58
84.07
90.22
96.47
91.73
98.39
6000
85.69
81.98
90.83
96.98
92.31
98.89
Potential Defense. We propose a potential defense method again TrojText by remarkably decreasing the ASR and increasing the attack overhead, i.e., TPN. The defense is motivated by a key observation that once important weights of benign models 0 hidden, TrojText is difficult to efficiently identify and learn Trojan weights. Specifically, we can adopt existing methods like matrix decomposition, e.g., (Hsu et al. 2022 Lou et al.) 2022 Zhong al. tto decompose in Equation S so that attackers with accumulated gradient ranking method cannot correctly identify the critical weights as before. We use To demonstrate the results TrojText before and after defense. In particular, after defense, we observed that ASR of TrojText is reduced by 5.54% 24.68% and the TPN is increased by 59 ~ 86 on three datasets.
CONCLUSION
In this paper, we present, TrojText, a test-time invisible Trojan attack against Transformer-basee textural models. Specifically, novel representation-logee objective function is proposed to enable Texj to efficiently learn the syntactic structure as trigger on small test-domain data. We adopted test-time weight-oriented Trojan attacks via bit flips in memory. The attack overhead is defined by the fit-fiip numbers. For this reason, two techniques including accumulated gradients ranking and Trojan weights pruning are introduced to reduce the bit-flip numbers of the Trojan model. We perform extensive experiments of AG's I News, SST-2 and OLID on BERT and XLNet. Our TrojText could classify 98.35% of test sentences into target class on BERT model for AG's News data.

---

Published as a conference paper at ICLR 2023
REFERENCES
Jiawang Bai, Kuofeng Gao, Dihong Gong, Shu-Tao Xia, Zhifeng Li, and Wei Liu. Hardly per- ceptible trojan attack against neural networks with bit flips. arXiv preprint arXiv:2207. 13417, 2022.
Lejla Batina, Shivam Bhasin, Dirmanto Jap, and Stjepan Picek. {CSI)[NN:: Reverse engineering of neural network architectures through electromagnetic side channel. In 28th USENIX Security Symposium (USENIX Security 19), pp- 515-532, 2019.
Huili Chen, Cheng Fu, Jishen Zhao, and Farinaz Koushanfar. Proflip: Targeted trojan attack with progressive bit flips. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7718-7722 2021a.
Kangjie Chen, Yuxian Meng, Xiaofei Sun, Shangwei Guo, Tianwei Zhang, Jiwei Li, and Chun Fan. Badpre: Task-agnostic backdoor attacks to pre-trained nlp foundation models. arXiv preprint arXiv:2110.02467, 2021b.
Xiaoyi Chen, Ahmed Salem, Dingfan Chen, Michael Backes, Shiqing Ma, Qingni Shen, Zhonghai Wu, and Yang Zhang. Badnl: Backdoor attacks against nlp models with semantic-preserving preserving improvements. In Annual Computer Security Applications Conference, pp. 554-569, 2021c
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. 2018.
Ming Fan, Ziliang Si, Xiaofei Xie, Yang Liu, and Ting Liu. Text backdoor detection using an interpretable rnn abstract model. IFFE Transactions on Information Forensics and Security, 16: 4117-4132, 2021.
Leilei Gan, Jiwei Li, Tianwei Zhang, Xiaoya Li, Yuxian Meng, Fei Wu, Shangwei Guo, and Chun Fan. Triggerless backdoor attack for nlp tasks with clean labels. arXiv preprint arXiv:2111.07970, 2021.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding secoding-enhanced bert with disentangled attention. In International Conference on Learning Representations, 2021.
URL httphhtss//pppeeeeiieeee net. wtt.eenneee/ooommmadd...........
Yen-Chang Hsu, Ting Hua, Sungen Chang, Qian Lou, Yilin Shen, and Hongxia Jin. Language model compression with weighted low-rank factorization. In International Conference Learning
Representations, 2022. URL https: //tper/ee//per.eeeeew.. net wtt.::/eeooommmon-PPr113..0o.
Weizhe Hua, Zhiru Zhang, and Edward Suh. Reverse engineering convolutional neural networks through side-channel information leaks. In 2018 55th ACM/SSDA/EEE Design Automation Con- ference (DAC), pp. 1-6. IEEE, 2018.
Mohit lyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. Adversarial example generation with syntactically controlled paraphrase networks. arXiv preprint arXiv: 06059, 2018
Yoongu Kim, Ross Daly, Jeremie Kim, Chris Fallin, Ji Hye Lee, Donghyuk Lee, Chris Wilkerson, Konrad Lai, and Onur Mutlu. Flipping bits memory without accessing them: an experimental study of dram disturbance errors. 42:361-372, 2014.
Klemen Kotar, Gabriel Ilharco, Ludwig Schmidt, Kiana Ehsani, and Roozbeh Mottaghi. Contrasting contrastive self-supervised representation learning pipelines. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9949-9959, October 2021.
Keita Kurita, Paul Michel, and Graham Neubig. Weight poisoning attacks pre-trained models. arXiv preprint arXiv:2004.06666 2020.
Yingqi Liu, Guangyu Shen, Guanhong Tao, Shengwei An, Shiqing Ma, and Xiangyu Zhang, Pic- colo: Exposing complex backdoors in nlp transformer models. In 2022 IEEE Symposium on Security and Privacy (SP), pp. 1561-1561. TEEE Computer Society, 2022.
10

---

Published as a conference paper at ICLR 2023
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 19XX::1977.119990 2019.
Qian Lou, Ting Hua, Yen-Chang Hsu, Yilin Shen, and Hongxia Jin. Dictformer: Tiny transformer with shared dictionary. In International Conference on Learning Representations, 2022. URL
https: Bitts:://ppeneviiie
net/ / forum?id=-GRWWe::
Fanchao Qi, Yangyi Chen, Xurui Zhang, Mukai Li, Zhiyuan Liu, and Maosong Sun. Mind the style of text! adversarial and backdoor attacks based on text style transfer. arXiv preprint (rXiv:2110.011:: 2021a.
Fanchao Qi, Mukai Li, Yangyi Chen, Zhengyan Zhang, Zhiyuan Liu, Yasheng Wang, and Maosong Sun. Hidden killer: Invisible textual backdoor attacks with syntactic trigger. arXiv preprint DrXiv:2105.12400 2021b.
Adnan Siraj Rakin, Zhezhi He, and Deliang Fan. Bit-flip attack: Crushing neural network with progressive bit search. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 1211-1220, 2019.
Adnan Siraj Rakin, Zhezhi He, and Deliang Fan. Tht: Targeted neural network attack with bit trojan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 13198 13207, 2020.
Kun Shao, Junan Yang, Yang Ai, Hui Liu, and Yu Zhang. Bddr: An effective defense against textual backdoor attacks. Computers & Security, 110: 102143,33, 2021.
Lujia Shen, Shouling Ji, Xuhong Zhang, Jinfeng Li, Jing Chen, Jie Shi, Chengfang Fang, Jian- wei Yin, and Ting Wang. Backdoor pre-trained models can transfer to all. arXiv preprint arXiv:2111.00197, 2021.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1631-1642, October 2013.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz. Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- fion processing systems, 30, 2017.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowiez, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-a natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38-45, October 2020.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32, 2019.
Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. Predicting the Type and Target of Offensive Posts in Social Media. In Proceedings of NAACL, 2019.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text clas- sification. Advances in neural information processing systems, 28, 2015.
Xinyang Zhang, Zheng Zhang, Shouling Ji, and Ting Wang. Trojaning language models for fun and profit. In 202/ IEEE European Symposium Security and Privacy (EuroS&P), pp. 179-197. IEEE, 2021.
Mengxin Zheng, Qian Lou, and Lei Jiang. Trojvit: Trojan insertion in vision transformers. arXiv preprint arXiv:2208. 00022208.13499 13049, 2022.
11

---

Published as a conference paper at ICLR 2023
Zhisheng Zhong, Fangyin Wei, Zhouchen Lin, and Chao Zhang. Ada-tucker: Compressing deep neural networks via adaptive dimension adjustment tucker decomposition. Neural Networks, 110: 104-115, 2019.
12


Reviews:
[{"review_id": "qaVLZdz6bI", "reviewer": "Reviewer_oFGQ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "What is the goal of the paper?\n\n* Test-time invisible textual trojan insertion\n\nWhat has been done before?\n\n* Current Trojan attacks with syntactic-structure triggers are highly dependent on a large corpus of training data which significantly limits the feasibility of these attacks in real-world NLP tasks. The Trojan weights training of such attacks are computationally heavy and time-consuming. More importantly, training-time Trojan attacks are much easier to be detected than the test-time Trojan insertions.\n\n* Proposed approach - TrojText is a test-time invisible textual Trojan insertion method - a more realistic, efficient and stealthy attack against NLP models without training data.\n\nWhat are the contributions of the paper?\n\n* Proposed TrojText to study whether the invisible textual Trojan attack can be efficiently performed without training data in a more realistic and cost-efficient manner\n\n* Proposed a novel Representation-Logit Trojan Insertion (RLI) algorithm to achieve a desired attack using smaller sampled test data instead of large training data.\n\n* Proposed accumulated gradient ranking (AGR) and Trojan Weights Pruning (TWP) to reduce the tuned parameters number and the attack overhead\n\n* Performed extensive experiments of AG’s News, SST-2 and OLID on BERT and XLNet. \n\n\n", "summary_of_the_review": "Proposed approaches look novel but presentation of paper could be more clear. Reproducibility aspect of the paper looks weak.", "strength_and_weaknesses": "Strengths\n\n* Proposed a more realistic, efficient and stealthy attack against NLP models without training data.\n\n* Described a potential defense for the proposed attack\n\nWeaknesses\n\n* Paper is a difficult read\n\n* Insufficient details or code available for reproducibility \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "bA6KoJLJia2", "reviewer": "Reviewer_YubY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies backdoor insertion attacks in pretrained language models. In this attack, neural network weights are modified such that they produce incorrect outputs for some targeted set of inputs (like a particular syntactic structure in this paper), but correct outputs for rest of the inputs. This paper presents a new attack called \"trojtext\", which leverages only a small amount of data to insert backdoors into BERT and XLNET. The key ideas in TrojText are - (1) training both logits and encoder representations to insert backdoors; (2) using accumulated gradient ranking and network pruning to modify only a small number of network weights.\n\nExperiments conducted on 3 text classification datasets confirm the efficacy of the proposed approach. A defense mechanism is against accumulated gradient ranking is also briefly discussed.", "summary_of_the_review": "Overall the experimental setup is good, and it's cool that backdoors can be inserted without a large dataset. However, no new setting (in terms of models / tasks) was analyzed (weakness 1,2) and the proposed approach seems a bit incremental compared to prior work (weakness 3). The paper would be much stronger with experiments on larger pretrained models, an exploration of backdoors in few-shot incontext learning settings, and experiments on non-classification datasets. Hence I'm currently leaning reject, but will re-consider after reading the rebuttal.\n\n-----\n\n**After rebuttal**: Thank you for the very detailed response! I've decided to increase my score to 6 due to the extra experiments on DeBERTa and some more clarification on the novelty of the work.", "strength_and_weaknesses": "**Strengths**\n\n1. The paper studies backdoor insertion into neural networks, a well established threat model in prior work. Moreover, the paper's method only modifies a few parameters of the network and uses a small dataset to achieve backdoor attacks.\n\n2. The paper performs experiments on 3 datasets with 2 pretrained LMs, and the experimental setup and metrics seem solid. The proposed algorithm significantly outperforms prior baselines. Ablation studies confirm the efficacy of each part of the proposed algorithm.\n\n**Weaknesses**\n\n1. The paper would be much stronger with newer / larger LMs. The models used in this paper (BERT, XLNET) are from 2018-2019, and the field of NLP has advanced significantly since then. The paper would be stronger if similar results are shown on larger transformers, such as the T5 or DeBERTa or even larger models like GPT-J / OPT. It may be harder to insert backdoors in these networks. Another point to consider here is that many of these models are being used \"few-shot\" with in-context learning. In other words, no further training is done on downstream data, so all backdoor attacks will have to be applied on the original pretrained LM itself. Do backdoors generalize across different tasks few-shot? Does insertion of backdoors targeted at one task affect performance on other tasks? Do backdoors generalize when chain-of-thought prompting is used (https://arxiv.org/abs/2201.11903), because LMs need to provide rationales for their judgments?\n\n2. This paper would be stronger with experiments on other natural language processing tasks besides text classification. Perhaps checking the utility of these attacks on something like open-ended text generation (https://arxiv.org/abs/1908.07125) will be interesting. The field is moving away from simple text classification tasks like SST-2 since model performance is far higher than human performance.\n\n3. Due to weakness #1,#2, it seems like the the main contribution of the paper is a new algorithm in a well-established experimental setup. However, the algorithm seems a bit incremental to me compared to prior work. Network pruning is a well-established technique in ML (https://arxiv.org/pdf/2003.03033.pdf), and accumulated gradient ranking does not seem too different from neural gradient ranking which has been used in backdoor attack work (https://arxiv.org/pdf/1909.05193.pdf). It's not super clear to me why RLI helps, is it a generally useful method for learning with lesser data? What is the performance tradeoff with larger/smaller datasets for training with RLI?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "OYEYgOJxkk", "reviewer": "Reviewer_ecL7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper combines the idea of 1) test-time attack by flipping bits of models and 2) syntactic-based triggers,  to achieve good\nattack results with only a few sampled test data.\n", "summary_of_the_review": "Overall, the paper is well written and have solid experimental contribution. ", "strength_and_weaknesses": "**Strenghts**\n* The experimental results show that the proposed method outperform the existed test-time attack method by a large margin.\n* The setting of test-time attack is more realistic than training-time attack to me.\n\n**Weaknesses**\n* The experiments are done only with syntatic trigger, which makes me wonder whether the major improvements come from the strong\n(originally proposed ) training-time attack baseline, i.e. Hidden Killer.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "OJMeTE34q0n", "reviewer": "Reviewer_LGnH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates bit-flip trojan attacks on text models with syntactic triggers. Several ideas are proposed for improving the effectiveness of these trojans. In experiments, this increases attack success rate (ASR) while having less of an effect on clean accuracy compared to baselines. Ablations confirm that modifying more bits leads to higher attack success rate. A defense is also proposed that can make models less susceptible to bit-flip attacks.", "summary_of_the_review": "Given the weaknesses and various minor issues, I recommend to reject at this time.\n\n----------------\nUpdate:\n\nThe authors have addressed all of my concerns. I now think this is a good contribution to the line of work on bit flip attacks (especially because it focuses on the text domain and Transformers). I would give the paper a 7 if that was an option and will let the AC know.", "strength_and_weaknesses": "Strengths:\n- Memory editing attacks are an important threat model\n- The proposed methods enable reaching higher ASR with a smaller decrease to clean accuracy\n- A useful defense is proposed in Section 5.3 that seems to work well against the proposed bit-flip attacks\n\nWeaknesses:\n- I don't buy the distinction between training-time and test-time attacks. Surely if one can modify the model weights at test time via rowhammer to accomplish some goal, then this would be a form of training/tuning the network. It seems like the actual difference is that \"test-time\" or bit-flip attacks try to minimize the number of parameters that are changed, which makes sense for a rowhammer attack vector. This matters, because in the related work the authors mention that there are many new trojan detection techniques for text models (e.g., PICCOLO), and they claim that these detectors work much better on \"training-time\" attacks. However, I don't see any reason to believe this, since the authors do not include experiments with these detectors, and it's unclear that modifying fewer parameters results in a less detectable attack.\n- Another issue with the test-time attack setting (or at least the version of it explored in this work) is that the trojan is inserted using data from the test set!!! This is akin to training on the test set, as it would artificially inflate ASR on the test set examples that were used for training the trojan. In reality, the victim model would be used on new samples from the data distribution that were not used for inserting the trojan, which is precisely why test sets are held out. (EDIT: One of the other reviewers pointed out that, based on the submitted code, it is likely that the authors used examples from the validation set. Could the authors confirm this?)\n- In Algorithm 1, there are multiple epochs, and each epoch consists of editing all parameters with a distance greater than e to the corresponding benign model parameters. This doesn't place a constraint on the number of parameters edited, which seems more relevant for the memory editing attack vector. Also, the parameters that are edited could change in each epoch, which seems unrealistic. Wouldn't this allow you to arbitrarily change the entire model? It would be more realistic to fix the parameters that can be changed at the start of training and to not change them across epochs.\n\n\nOther points:\n- \"trigger synthesizing is not applicable to textual model due to non-differentiable text tokens\" There has been some work on trigger synthesis for textual trojans, e.g., the Neural Cleanse experiments in \"Trojaning Language Models for Fun and Profit\".\n- RLI is called a contrastive loss, but it would be more accurate to call it an MSE loss, since it is just the MSE between two representations.\n- The proposed defense is quite interesting. If this is novel, it should be expanded much more.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster