Modeling Adversarial Attack on Pre-trained Language Models as Sequential Decision Making
Year: 2023

Abstract:
Pre-trained language models (PLMs) have been widely used to underpin various downstream tasks. However, the adversarial attack task has found that PLMs are vulnerable to small perturbations. Mainstream methods adopt a detached two-stage framework to attack without considering the subsequent influence of substitution at each step. In this paper, we formally model the adversarial attack task on PLMs as a sequential decision-making problem, where the whole attack process is sequential with two decision-making problems, i.e., word finder and word substitution. Considering the attack process can only receive the final state without any direct intermediate signals, we propose to use reinforcement learning to find an appropriate sequential attack path to generate adversaries, named SDM-ATTACK. Extensive experimental results show that SDM-ATTACK achieves the highest attack success rate with a comparable modification rate and semantic similarity to attack fine-tuned BERT. Furthermore, our analyses demonstrate the generalization and transferability of SDM-ATTACK. The code is available at https://github. com/fduxuan/SDM-Attack.

heading: 1 Introduction
text: Nowadays, pre-trained language models (PLMs) have shown strong potential in various downstream tasks (Devlin et al., 2018; Brown et al., 2020). However, a series of studies about adversarial attack (Jin et al., 2020; Li et al., 2020a,b) have found that PLMs are vulnerable to some small perturbations based on the original inputs. The adversarial attack is essential to develop trustworthy and robust PLMs in Artificial Intelligence (AI) community (Thiebes et al., 2021; Marcus, 2020).
Despite the adversarial attack achieving success in both image and speech domains (Chakraborty
∗Equal Contribution †Corresponding Author
et al., 2018; Kurakin et al., 2018; Carlini and Wagner, 2018), it is still far from perfect in the natural language processing (NLP) field due to the discrete nature of language (Studdert-Kennedy, 2005; Armstrong et al., 1995). The main problem is to find an appropriate search algorithm that can make perturbations to mislead the victim models (i.e., PLMs) successfully (Morris et al., 2020; Yoo and Qi, 2021). As mentioned in recent studies (Jin et al., 2020), the challenges are preserving the following properties: 1) human prediction consistency, misleading the PLMs while keeping human judges unchanged; 2) semantic similarity, keeping the semantics of the original inputs; 3) language fluency, ensuring the correctness of grammar.
7322
Mainstream solutions are typically a detached two-stage framework. Specifically, they first rank the importance scores of all tokens according to the original input and then orderly substitute these tokens via heuristic rules. Previous studies propose different strategies to rank the editing order of tokens, such as temporal-based algorithm (Gao et al., 2018), probability-weighted saliency (Ren et al., 2019; Li et al., 2020b,a; Jin et al., 2020), and gradient-based ranking (Yoo and Qi, 2021). However, these methods face two limitations. On the one hand, they use a threshold to filter the unsatisfactory substitutions at last, but neglect to integrally consider the properties during computing importance scores. On the other hand, their editing order only depends on the original input without considering the subsequent influence of substitution, as computing the importance score at each step is computationally burdensome in practice.
To solve the issues mentioned above, in this paper, we formally propose to transform the adversarial attack problem into a sequential decisionmaking task as shown in Figure 1. Rather than computing the importance scores all at once based on the original input, we regard the entire attack process as a sequence, where scores in the next step are influenced by the editing results in the current step. Furthermore, there are two types of decisionmaking problems during each step in the attack sequential process: 1) word finder, choosing the appropriate token to edit; 2) word substitution, replacing the token with a suitable substitution. Meanwhile, selecting edited tokens at each step should take the attack success rate and crucial properties, such as fluency, into account.
As a sequential decision-making task without a direct signal in each step, we naturally leverage reinforcement learning (RL) to find an appropriate sequential attack path to generate adversaries. In this paper, we propose a model-agnostic method based on policy-based RL for modeling the adversarial attack into Sequential Decision Making, entitled SDM-ATTACK. Given the victim model as the environment with designed reward functions and the original input text as the initial state, the reinforced agent needs to decide on tokens to edit and synonyms to replace sequentially, until it attacks successfully. The experimental results show that SDM-ATTACK achieves the highest attack success rate with a comparable modification rate and semantic similarity to attack fine-tuned BERT against
state-of-the-art baselines. Furthermore, we also demonstrate the effectiveness, generalizability, and transferability of SDM-ATTACK in our analysis.
The main contributions of this work are summarized as the following:
• To the best of our knowledge, we are the first to model the adversarial attack on PLMs into a sequential decision-making problem, where the whole attack process is sequential with two decision-making problems, i.e., word finder and word substitution.
• Considering the sequential attack process can receive the final state without any direct intermediate signals, we propose SDM-ATTACK to use reinforcement learning to ask the agent to find an appropriate attack path based on our designed indirect reward signals yielded by the environment.

heading: 2 Preliminaries
text: As for NLP tasks, given a corpus of N input texts X = {x1,x2,x3, · · · ,xN} and an output space Y = {y1, y2, y3, · · · , yK} containing K labels, the language model F learns a mapping f : x → y , which learns to classify each input sample x ∈ X to the ground-truth label ygold ∈ Y:
F(x) = argmax yi∈Y
P (yi|x) (1)
The adversary of text x ∈ X can be formulated as xadv = x + ϵ, where ϵ is a slight perturbation to the input x. The goal is to mislead the victim model F within a certain constraint C(xadv):
F(xadv) = argmax yi∈Y
P (yi|xadv) ̸= F(x),
and C(xadv,x) ≥ λ (2)
where λ is the coefficient, and C(xadv,x) usually calculates the semantic or syntactic similarity (Cer et al., 2018; Oliva et al., 2011) between the input x and its corresponding adversary xadv.
Recently, the adversarial attack task has been framed as a combinatorial optimization problem. However, previous studies (Gao et al., 2018; Ren et al., 2019; Yoo and Qi, 2021) address this problem without considering the subsequent influence of substitution at each step, making attack far from the most effective. In this paper, we formally define the adversarial attack as a sequential decisionmaking task, where the decisions in the next step are influenced by the results in the current step.

heading: 3 Methodology
text: In this section, we model the adversarial attack on PLMs problem as a sequential decision-making task as shown in Figure 1, where the entire attack process is a sequence with two decision-making problems. Considering the lack of direct signal in each step during the attack process, we propose a model-agnostic method, named SDM-ATTACK, based on policy-based reinforcement learning. The illustration is shown in Figure 2. During each step in the attack process, the reinforced agent needs to take two actions: 1) word finder, choosing the appropriate token to edit; and 2) word substitution, replacing the token with a suitable substitution. Through an attack sequence toward the input, we obtain its adversary until the attack is successful.

heading: 3.1 Environment and Rewards
text: We regard the victim models (i.e., PLMs) as the whole environment. Intuitively, the agent needs to generate adversaries against the environment and achieve as high a reward as possible. The t-step environment state is our intermediate generation xt = [wt1,w t 2, ...,w t n] containing n words, where the initial state x0 is the original input. Considering the lack of direct signal in each step, our reward consists of a final discriminant signal rd to present the state of termination and an instant reward rt on every step. As for the final signal rd, once the model prediction of t-step state is different from the initial state, the environment will terminate this episode and yield a success signal. However, if the model prediction does not change when all the tokens are replaced or the maximum number of steps is reached, a failure signal will be
given. Overall, the final signal rd is denoted as:
rd = { 1, success −1, failure
(3)
As for the instant reward ri for each step, we hope that the t-step state xt can not only mislead the victim model but also ensure semantics similarity and fluency. Firstly, we design one instant reward to evaluate attack success rates:
rattt = { rd, terminated P (ygold|xt−1)− P (ygold|xt), survive (4) where rd is the final reward if the current episode is terminated. Secondly, we define a punishment by using an auto-regressive language model (LM) to measure fluency:
rflut = ∑
i
1
|xt|(LM(xi|x t)−LM(xi|xt−1)) (5)
where LM(xi|xt) is the cross-entropy loss of the token xi in sentence xt. Thirdly, we also add semantic similarity constraints as another punishment:
rsimt = Sim(x,x t−1)− Sim(x,xt) (6)
Finally, our overall instant reward rt is defined as:
rt = β1r att t − β2rflut − β3rsimt (7)

heading: 3.2 Decision Making
text: During each step in the whole attack process, there are two types of decision-making problems. The first is choosing the appropriate token to edit, while the second is replacing the token with a suitable substitution. In RL, the agent needs to determine the decisions according to the yielded rewards.
Word Finder To find the appropriate token to edit, we first employ the masked language models (MLM) as an encoder to represent the state xt. Due to the setup of the sub-word tokenizer in MLM, the encoder first converts xt to a token sequence xttoken = [o t 1,o t 2, ...,o t m]. We reverse the conversion mapping ϕ : xttoken → xt to recover tokens into words in need. Then we obtain the hidden states ht = [ht1,h t 2, ...,h t m], where h t i ∈ Rd is the hidden state of token oti with d dimensions. Furthermore, we maintain a word set W to restore the words of x that have been already modified as well as stop words and punctuation. We then adopt a simple binary representation bt according to the word set W:
bti = { 0 ∈ Rd ϕ(oti) ∈W 1 ∈ Rd ϕ(oti) /∈W
(8)
Then, we fuse both the hidden states hti and the binary representation bti to obtain the final representation eti of the environment states:
eti = [h t i; b t i] (9)
where [; ] denotes the concatenation operation. During the process of training, we first adopt a simple linear layer to obtain the probability and further normalize it into a distribution. The probability distribution p(oti|xt) of each token at t-step can be calculated as follows:
p(oti|xt) = softmax(W · eti + b) (10)
where W, b are the weight matrix and the bias vector, respectively. Then the agent samples the word wt to substitute according to the distribution and ensures the sampled word is not in the word set W.
During the evaluation, the agent will directly select the token with the maximum probability at each step, which is formulated as follows:
wt = argmax p(oti|xt), ϕ(oti) /∈W (11)
If the selected token wt is a sub-word, we reverse the sub-word into a complete word via the conversion mapping ϕ as the newly selected word.
Word Substitution Following Jin et al. (2020), we adopt synonym substitution as our strategy after obtaining selected word wt in t-step. Firstly, we gather a synonym set Swt for wt that contains top-k candidates from the external vocabulary, computing via cosine similarity (Mrkšić
et al., 2016). Then, for each s ∈ Swt , we replace wtp with s in the sentence x
t to get a substitution xts = [w1, ...,wp−1, s,wp+1, ...wn]. Finally, according to the instant reward rt in the Equation 4, we select the substitution with the highest reward as the final adversaries xtadv. Meanwhile, the environment states further updates as follows:
{ xt+1 = xtadv W = W ∪ {wtp}
(12)

heading: 3.3 Agent Training
text: The training target is to maximize the total return G(τ), which is an accumulated reward based on the instant reward rt , defined in Equation 7, with a discount factor γ ∈ [0, 1):
G(τ) = T∑
t=1
γtrt (13)
The expected return of the decision trajectory, i.e., attack path, is defined as follows:
J(θ) = E[G(τ)] (14)
Furthermore, we regard the agent as πθ with parameters θ and the attack path as τ = [(af1 , a s 1), · · · , (afT , asT )], where a f t and a s t represent actions of word finder and substitution in t-th step, respectively. The probability of this attack path is calculated as πθ(τ) =∏T
t=1 πθ((a f t , a s t )|st), where πθ((aft , ast )|st) is the probability of actions in step t based on current environment state st. Meanwhile, we consider ast a prior knowledge so that this probability can be simplified. The gradient is calculated by REINFORCE algorithm (Kaelbling et al., 1996):
∇J(θ) = ∇E[log πθ(τ) ·G(τ)] (15) Detailed information of reinforce training is shown in appendix B.

heading: 4 Experiments

heading: 4.1 Experimental Setups
text: Tasks and Datasets Following Li et al. (2020b); Jin et al. (2020), we evaluate the effectiveness of SDM-ATTACK mainly on two standard NLP tasks, text classification and textual entailment. As for text classification, we use diverse datasets from different aspects, including news topic classification (AG’s News; Zhang et al., 2015), sentence-level sentiment analysis (MR; Pang and Lee, 2005) and document-level sentiment analysis (IMDB1 and
1https://datasets.imdbws.com/
Yelp Polarity; Zhang et al., 2015). As for textual entailment, we use a dataset of sentence pairs (SNLI; Bowman et al., 2015) and a dataset with multi-genre (MultiNLI; Williams et al., 2017). The statistics of datasets and more details can be found in Appendix A. Following Jin et al. (2020); Alzantot et al. (2018), we attack 1k samples randomly selected from the test set of each task.
Baselines We compare SDM-ATTACK with recent state-of-the-art studies: 1) TextFooler (Jin et al., 2020): find important words via probability weighted word saliency and then apply substitution with counter-fitted word embeddings. 2) BERT-Attack (Li et al., 2020b): use mask-predict approach to generate adversaries. 3) A2T (Yoo and Qi, 2021): adopt faster search with gradientbased word importance ranking algorithm. We use open-source codes provided by the authors and TextAttack tools (Morris et al., 2020) to implement these baselines. Furthermore, to ensure fairness in comparing baselines and SDM-ATTACK, we apply constraints to all methods following Morris et al. (2020) in Appendix C.
Victim Models We conduct the main experiments on a standard pre-trained language model BERT following (Jin et al., 2020; Li et al., 2020b). To detect the generalization of SDM-ATTACK, we explore the effects on more typical models as discussed in Section 5.1. All victim models are pretrained from TextAttack (Morris et al., 2020).
Implementation Details We adopt BERT as the MLM model in word finder and GPT-2 (Radford et al., 2019) to measure fluency when computing
rewards. To keep instant reward and punishment in a similar range, we set the hyper-parameters β1 to be 1, β2 to be 1 and β3 to be 0.2. Moreover, the discount factor γ is set to be 0.9 to achieve a trade-off between instant reward and long-term return. We set the episode number as M = 200 and the learning rate as α = 3e−6 with Adam as the optimizer. In word substitution, the parameter K of the synonyms number is 50. Our experiments are conducted on a single NVIDIA 2080ti.
Automatic Evaluation Metrics Following previous studies (Jin et al., 2020; Morris et al., 2020), we use the following metrics as the evaluation criteria. 1) Attack success rate (A-rate): the degraded performance after attacking target model. 2) Modification rate (Mod): the percentage of modified words comparing to original text. 3) Semantic similarity (Sim): the cosine similarity between the original text and its adversary, computing via the universal sentence encoder (USE; Cer et al., 2018).
Manual Evaluation Metrics We further manually validate the quality of the adversaries from three challenging properties. 1) Human prediction consistency (Con): the rate of human judgement which is consistent with ground-truth label; 2) Language fluency (Flu): the fluency score of the sentence, measured on a Likert scale of 1 to 5 from ungrammatical to coherent (Gagnon-Marchand et al., 2019); 3) Semantic similarity (Simhum): the semantic consistency between each input-adversary pair, where 1 means unanimous, 0.5 means ambiguous, 0 means inconsistent.

heading: 4.2 Results
text: Automatic Evaluation As shown in Table 1, SDM-ATTACK consistently achieves the highest attack success rate to attack BERT in both text classification and textual entailment tasks, which indicates the effectiveness of SDM-ATTACK. Furthermore, SDM-ATTACK mostly obtains the best performance of modification and similarity metrics, except for AG’s News, where SDM-ATTACK achieves the second-best. For instance, our framework only perturbs 4.1% of the words on the IMDB datasets, while the attack success rate is improved to 91.4% with a semantic similarity of 0.82. Although A2T performs better in modification and similarity metrics in Yelp and AG’s News, their attack success rate is always much lower than SDMATTACK, even other baselines. Because the modification and similarity metrics only consider the successful adversaries, we conjecture that A2T can only solve the inputs which are simpler to attack. In general, our method can simultaneously satisfy the high attack success rate with a lower modification rate and higher similarity. Furthermore, We find that the attack success rate on document-level datasets, i.e., Yelp and IMDB, are higher than the other sentence-level datasets, which indicates that it is easier to mislead models when the input text is longer. The possible reason is the victim model tends to use surface clues rather than understand them to make predictions when the context is long.
Manual evaluation In manual evaluation, we first randomly select 100 samples from successful adversaries in IMDB and MNLI datasets and then ask three crowd-workers to evaluate the quality of the original inputs and our generated adversaries. The results are shown in Table 2. As for the human prediction consistency, we regard the original inputs as a baseline. Taking IMDB as an example, humans can correctly judge 95% of the original inputs while they can maintain 90% accuracy to our gen-
erated adversaries, which indicates SDM-ATTACK can mislead the PLMs while keeping human judges unchanged. The language fluency scores of adversaries are close to the original inputs, where the gap scores are within 0.3 on both datasets. Furthermore, the semantic similarity scores between the original inputs and our generated adversaries are 0.95 and 0.83 in IMDB and MNLI, respectively. In general, SDM-ATTACK can satisfy the challenging demand of preserving the three aforementioned properties. Detailed design of manual evaluation and more results are shown in appendix E.

heading: 5 Analyses

heading: 5.1 Generalization
text: We detect the generalization of SDM-ATTACK in two aspects, 1) attack more language models and 2) adapt to more substitution strategies. Firstly, we apply SDM-ATTACK to attack extensive victim models, such as traditional language models (e.g., WordCNN) and other state-of-the-art PLMs (e.g., RoBERTa; Liu et al., 2019). The results of text classification tasks in table 3 show that SDM-ATTACK not only has better attack effects against WordCNN and WordLSTM, but also misleads RoBERTa, which is a more robust model. For example, on the IMDB datasets, the attack success rate is up to 89.2% against WordCNN with a modification rate of only about 3.3% and a high semantic similarity of 0.85. As for the textual entailment task, SDM-ATTACK can also achieve remarkable attack
success rates against InferSent and ESIM. Secondly, although we directly adopt the word substitution strategy in Textfooler, this strategy can actually be replaced by other methods. To demonstrate this assumption, we further replace our word substitution strategy with the mask-fill way in BERT-attack, named SDM-ATTACK-mlm. As shown in Table 4, SDM-ATTACK-mlm completely beat BERT-Attack, indicating the part of word substitution of SDM-ATTACK has generalization ability to extend to different types of strategies and archives high performance. More results are displayed in appendix E.

heading: 5.2 Efficiency
text: In this section, we probe the efficiency according to varying sentence lengths in the IMDB dataset as shown in Figure 3. The time cost of SDMATTACK is surprisingly mostly better than A2T, which mainly targets obtaining cheaper computation costs with lower attack success rates in Table 1. Meanwhile, SDM-ATTACK can obviously beat BERT-attack and TextFooler, which need to conduct a model forward process for each token. Furthermore, with the increase of sentence lengths, SDM-ATTACK and A2T maintain a stable time cost, while the time cost of BERT-attack and TextFooler is exploding. These phenomena show the efficiency advantage of SDM-ATTACK, especially in dealing with long texts.

heading: 5.3 Transferability
text: We evaluate the transferability of SDM-ATTACK to detect whether the SDM-ATTACK trained on one dataset can perform well on other datasets. We conduct experiments on a series of text classification tasks and use the randomly initialized BERT as a
baseline. As shown in Table 5, SDM-ATTACK has high transferability scores across different datasets, which are consistently higher than random. In detail, the performances among Yelp, IMDB and MR, which all belong to sentiment analysis, are higher than AG’s News. Moreover, IMDB and MR are corpora about movies where SDM-ATTACK tends to learn a general attack strategy in this field and can transfer well to each other.

heading: 5.4 Adversarial Training
text: We further investigate to improve the robustness of victim models via adversarial training. Specifically, we fine-tune the victim model with both original training datasets and our generated adversaries, and evaluate it on the same test set. As shown in Table 6, compared to the results with the original training datasets, adversarial training with our generated adversaries can maintain close accuracy, while improving performance on attack success rates, modification rates, and semantic similarity. The victim models with adversarial training are more difficult to attack, which indicates that our generated adversaries have the potential to serve as supplementary corpora to enhance the robustness of victim models.

heading: 5.5 Case Study
text: Table 7 shows adversaries produced by SDMATTACK and the baselines. Overall, the performance of SDM-ATTACK is significantly better than other methods. For this sample from the MR dataset, only TextFooler and SDM-ATTACK successfully mislead the victim model, i.e., changing the prediction from negative to positive. However, TextFooler modifies twice as many words as SDMATTACK, demonstrating our work has found a more suitable modification path. Adversaries generated by A2T and BERT-Attack are failed samples due to the low semantic similarity. BERT-Attack even generates an invalid word “enamoted" due to its subword combination algorithm. We also ask crowdworkers to give a fluency evaluation. Results show SDM-ATTACK obtains the highest score of 5 as the original sentence, while other adversaries are considered difficult to understand, indicating SDMATTACK can generate more natural sentences.

heading: 6 Related Work
text: Adversarial attack has been well-studied in image and speech domains (Szegedy et al., 2013; Chakraborty et al., 2018; Kurakin et al., 2018; Carlini and Wagner, 2018). However, due to the discrete nature of language, the adversarial attack against pre-trained language models is much more difficult. Earlier works mainly focus on designing heuristic rules to generate adversaries, including swapping words (Wei and Zou, 2019), transforming syntactic structure (Coulombe, 2018), and paraphrasing by back-translation (Ribeiro et al., 2018; Xie et al., 2020). However, these rule-based methods are label-intensive and difficult to scale.
Recently, adversarial attack in NLP is framed as
a combinatorial optimization problem. Mainstream studies design a series of search algorithms with two detached stages In the first stage, they iteratively search for modification positions, including saliency-based ranking (Liang et al., 2017; Ren et al., 2019; Jin et al., 2020; Garg and Ramakrishnan, 2020), gradient-based descent algorithm (Sato et al., 2018; Yoo and Qi, 2021), and temporal-based searcher (Gao et al., 2018). In the second stage, a series of studies designs different substitution strategies, including dictionary method (Ren et al., 2019), word embeddings (Kuleshov et al., 2018; Jin et al., 2020) or language models (Li et al., 2020b; Garg and Ramakrishnan, 2020; Li et al., 2020a). In this paper, we formally propose to define the adversarial attack task as a sequential decision-making problem, further considering that scores in the next step are influenced by the editing results in the current step.
The other line of recent studies is samplingbased methods. Alzantot et al. (2018) and Wang et al. (2019) apply genetic-based algorithm, Zang et al. (2019) propose a particle swarm optimizationbased method, and Guo et al. (2021) generate adversaries via distribution approximate sampling. However, their execution time is much more expensive due to the properties of sampling, so it is unlikely to generate large-scale adversarial samples. In addition, Zou et al. (2019) conducts reinforcement learning on attacking the neural machine translation task, but their search path is fixed from left to right. In this paper, SDM-ATTACK can determine any search order to find the appropriate attack path.

heading: 7 Conclusion
text: In this paper, we formally define the adversarial attack task as a sequential decision-making problem,
considering the entire attack process as sequence with two types of decision-making problems, i.e., word finder and substitution. To solve this problem without any direct signals of intermediate steps, we propose to use policy-based RL to find an appropriate attack path, entitled SDM-ATTACK. Our experimental results show that SDM-ATTACK achieves the highest attack success rate. In this paper, we use our designed rewards as instant signals to solve these two decision-making problems approximately. We will further try to adopt hierarchical RL to optimize the solution.

heading: 8 Limitations
text: We define the adversarial attack task as a sequential decision-making problem and apply policy-based reinforcement learning to model it. This work must follow this assumption: the decision process conforms to Markov decision process (MDP) that the conditional probability distribution of the future state depends only on the current state. Meanwhile, reinforcement learning training requires additional time costs and the results may be unstable.
We only conduct the experiments on two NLP tasks with six selected datasets, which are all English corpus. Furthermore, our experimental results are mainly for BERT, with RoBERTa supplemented in the analysis. Thus, we lack the evaluation of other novel pre-trained language models, such as ELECTRA (Clark et al., 2020) and XLNET (Yang et al., 2019). Therefore, our work lacks multi-task, multi-model and multilingual verification in terms of generalization and transferability.

heading: 9 Ethics Statement
text: We declare that this article is in accordance with the ethical standards of ACL Code of Ethics. Any third party tools used in this work are licensed from their authors. All crowd-workers participating in the experiments are paid according to the local hourly wages.

heading: 10 Acknowledgment
text: We would like to thank anonymous reviewers for their insightful and constructive feedback. We appreciate Peng Li and Shuo Wang for their valuable discussions. We thank Qianlin Liu, Yanqi Jiang and Yiwen Xu for the crowdsourced work. This work is supported by the National Key R&D Program of China (2022ZD0160502) and the National Nat-
ural Science Foundation of China (No. 61925601, 62276152, 62236011).

heading: A Datasets
text: We conduct experiments on the following datasets of two NLP tasks and detailed statistics are displayed in Table 8:
• Text Classification: (1) Yelp (Zhang et al., 2015): A dataset for binary sentiment classification on reviews, constructed by considering stars 1 and 2 negative, and 3 and 4 positive. (2) IMDB: A document-level movie review dataset for binary sentiment analysis. (3) MR (Pang and Lee, 2005): A sentencelevel binary classification dataset collected from Rotten Tomatoes movie reviews. (4) AG’s News (Zhang et al., 2015): A collection of news articles. There are four topics in this dataset: World, Sports, Business, and Science/Technology.
• Textual Entailment: (1) SNLI (Bowman et al., 2015): A dataset of human-written English sentence pairs and manually annotated labels of entailment, neutral and contradiction. (2) MNLI (Williams et al., 2017): Another crowd-sourced collection of sentence pairs labeled with textual entailment information. Compare to SNLI, it includes more complex sentences, e.g, enres of spoken and written text.

heading: B Training Algorithm
text: The training process is shown in Algorithm 1. Since aft is chosen through a probability distribution, the agent is encouraged to explore more possible paths. The instant reward rt is obtained from environment after performing both two actions actions. Once the termination signal is raised, the environment will terminate this current episode and update the agent’s parameters via a policy gradient approach. The expected return of decision trajectory is defined as follows:
J(θ) = E[G(τ)] (16)
Thus the gradient is calculated by REINFORCE algorithm (Kaelbling et al., 1996):
∇J(θ) = ∇E[log πθ(τ) ·G(τ)] (17) Then the expectation over the whole sequence is approximated by Monte Carlo simulations and can be expressed as follows:
∇J(θ) = 1 M
M∑
m=1
∇ log πθ(τ (m))G(τ (m)) (18)
Algorithm 1 Reinforce Training 1: Initialization: agent πθ with parameters θ,
episode number M 2: for i← 1 to M do 3: initialize t← 1 4: while not receive termination signal do 5: get environment state st 6: compute πθ((a f t , a s t )|st) ∽ πθ(aft |st) 7: sample aft based on probability 8: select ast from prior knowledge 9: compute reward rt
10: update t← t+ 1 11: end while 12: initialize G(τ) ← 0 13: for j ← T to 1 do 14: G(τ)← γG(τ) + rj 15: accumulate Jj(θ) 16: end for 17: update θ ← θ + α∇J(θ) 18: end for
where [τ (1), τ (2), ..., τ (M)] are M samples of trajectories. The discount factor γ enables both longterm and immediate effects to be taken into account and trajectories with shorter lengths are encouraged.
We randomly select 2500 items from the training corpus for training the agent of each dataset. The average convergence time is approximately between 2-16 hours, related to the length of the input. When attacking large batches of samples, the impact of training cost is negligible compared to the cumulative attack time cost. During training, We adopt random strategies and short-sighted strategies in the initial stage for early exploration and to obtain better seeds.
C Implementation Constraint
In order to make the comparison fairer, we set the following constraints for SDM-ATTACK as well as all baselines: (1) Max modification rate: To better
maintain semantic consistency, we only keep adversarial samples with less than 40% of the words to be perturbed. (2) Part-of-speech (POS): To generate grammatical and fluent sentences, we use NLTK tools2 to filter candidates that have a different POS from the target word. This constraint is not employed on BERT-Attack. (3) Stop words preservation: the modification of stop words is disallowed and this constraint helps avoid grammatical errors. (4) Word embedding distance: For Textfooler, A2T and SDM-ATTACK, we only keep candidates with word embedding cosine similarity higher than 0.5 from synonyms dictionaries (Mrkšić et al., 2016). For mask-fill methods, following BERT-Attack, we filter out antonyms (Li et al., 2020b) via the same synonym dictionaries for sentiment classification tasks and textual entailment tasks.

heading: D Tuning with Adversaries
text: Table 9 displays adversarial training results of all datasets. Overall, after fine-turned with both original training datasets and adversaries, victim model is more difficult to attack. Compared to original results, accuracy of all datasets is barely affected, while attack success rate meets an obvious decline. Meanwhile, attacking model with adversarial training leads to higher modification rate, further demonstrating adversarial training may help improve robustness of victim models.

heading: E Supplementary Results
text: At the beginning of manual evaluation, we provided some data to allow crowdsourcing workers to unify
2https://www.nltk.org/
the evaluation standards. We also remove the data with large differences when calculating the average value to ensure the reliability and accuracy of the evaluation results. More manual evaluation results are shown in Table 10.
Table 11 displays the generalization ability of SDM-ATTACK with mask-fill strategy. However, the improvement effect is not particularly obvious. The mask-fill method makes the current candidate synonyms also affected by the sequence states. Compared to a fixed synonym dictionary, it has a larger prior knowledge and changing action space, which makes it harder to train the agent. Only increasing the size of the training corpus is not very effective. We will try adopting hierarchical RL to further solve this problem in the future.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
8
7 A2. Did you discuss any potential risks of your work? There is no risks in our work.
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? We write this paper ourselves.
B 3 Did you use or create scientific artifacts? 4,9
3 B1. Did you cite the creators of artifacts you used? 4
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? 9

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? 4
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Our experiments are performed on public datasets.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? 4, appendix
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. We provide statistics of datasets in appendix and detail information about baselines in section 4.
C 3 Did you run computational experiments? 4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? 4
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? 4

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 4

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 4
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? 4

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? 4

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? 9
7 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? We have used publicly available datasets.
7 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Our work does not involve collecting ethically relevant information.

heading: 3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: 4

heading: Future Work
text: We will try adopting hierarchical RL to further solve this problem in the future. ACL 2023 Responsible NLP Checklist