SlowGuess/NAACL_2021_OCR huggingface dataset:
Bushra Sabir
University of Adelaide  
CREST - The Centre for Research on Engineering  
Software Technologies  
CSIRO Data61
Muhammad Ali Babar
University of Adelaide  
CREST - The Centre for Research on Engineering  
Software Technologies
Raj Gaire
CSIRO Data61
Adversarial Examples (AEs) generated by perturbing original training examples are useful in improving the robustness of Deep Learning (DL) based models. Most prior works generate AEs that are either unconscionable due to lexical errors or semantically and functionally deviant from original examples. In this paper, we present ReinforceBug, a reinforcement learning framework, that learns a policy that is transferable on unseen datasets and generates utility-preserving and transferable (on other models) AEs. Our experiments show that ReinforceBug is on average  $10\%$  more successful as compared to the state-of-the-art attack TextFooler. Moreover, the target models have on average  $73.64\%$  confidence in wrong prediction, the generated AEs preserve the functional equivalence and semantic similarity  $(83.38\%)$  to their original counterparts, and are transferable on other models with an average success rate of  $46\%$ .
Machine Learning (ML) models have attained remarkable success in several tasks such as classification and decision analytics. However, ML Models specifically Deep Learning (DL) based models, are often sensitive to Adversarial Examples (AEs). AEs consist of modified training data samples that preserve the intrinsic utilities of the ML solutions, but influence target classifier's predictions between original and modified inputs (Fass et al., 2019). Recent works (Fass et al., 2019; Li et al., 2020; Biggio and Roli, 2018) have demonstrated that (i) including AEs as a part of training data can enhance the robustness and generalization of the ML models, and (ii) these examples can be utilized to test the robustness of ML models and help understand their security vulnerabilities and limitations. Previous works on generating AEs have attained success in image (Biggio and Roli, 2018) and on a few conventional text classification tasks such
as sentiment, text entailment and movie reviews (Li et al., 2018; Jin et al., 2020; Li et al., 2020). Nevertheless, generating AEs for discrete textual data is still a challenge (Jin et al., 2020).
Textual AEs generated by the prior works have the following limitations:
Utility Preservation Textual AEs require to satisfy task-specific constraints such as lexical rules (spellings or grammar), semantic similarity and functional equivalence to original examples. Yet, most of the current state-of-the-art methods do not satisfy these constraints, thereby generating imperceivable AEs for end-users (Zhang et al., 2020). A few recent works (Li et al., 2020; Wang et al., 2019; Li et al., 2018; Jin et al., 2020) have considered semantic similarity constraint, but other constraints have been barely explored (Jin et al., 2020).
Knowledge Transferability Most prior works Alzantot et al. (2018); Jin et al. (2020); Wang et al. (2019) generate one to one example-specific AEs, i.e., for a given example  $x$ , they create an example  $x'$ . Each instance  $x$  is considered independent in a corpus  $C$ , and no relationship between different instances in the corpus is assumed. Therefore, the knowledge gained by transforming an example to AE is limited to a single example and is not reused on other examples. This process is both time-consuming and may not generalize the identified vulnerabilities of the target model.
Word Replacement Strategy Most prior works use a single word replacement strategy such as synonym substitution (Alzantot et al., 2018; Jin et al., 2020; Wang et al., 2019) or character perturbation (Gao et al., 2018) to generate AEs. This strategy has two main disadvantages: (i) the AEs generated by using a character-based replacement strategy, contains many spelling errors that result in unnatural text; and (ii) multiple word transformations with its synonym in a synonym-based replacement
strategy affect the language's fluency, making it sound unnatural. For instance, Jin et al. (2020) generated an AE "Jimmy Wales is a big, fucking idiot liar friggin nincompoop deception". This AE sounds unnatural and is grammatically incorrect.
Handling Noisy Datasets Most prior works generated AEs for datasets such as Yelp (Yelp) and Fake News (Kaggle, 2018a) which do not contain many spelling mistakes or out-of-the-vocabulary words. However, human generated natural text is prone to lexical errors. For example, tweets usually contain informal language, misspellings and unknown words. Prior works, such as (Jin et al., 2020; Wang et al., 2019; Alzantot et al., 2018; Li et al., 2020) that considered synonym substitution strategy, cannot deal with such noisy dataset.
We present ReinforceBug that addresses the aforementioned limitations of prior works. Our code is available at https://bit.ly/2QOZDMT. Our main contributions are summarized as follows:
1. We propose a reinforcement learning framework, ReinforceBug, that learns a policy to generate utility-preserving AEs under the black-box setting with no knowledge about the target model architecture or parameters.
2. We evaluate ReinforceBug on four state-of-the-art deep learning models Email spam, Twitter spam, Toxic content and Review polarity detection respectively.
3. We investigate the transferability of our learned policy on a new dataset.
4. We also examine the transferability of our generated AEs on three other state-of-the-art deep learning models.
Adversarial attacks are extensively studied in computer vision domain (Biggio and Roli, 2018; Goodfellow et al., 2014). Early works in adversarial text attacks were inspired by Generative Adversarial Networks (GANs) (Wong, 2017; Zhao et al., 2018). Wong 2017 showed that GAN-based reinforcement learning algorithms become unstable after a few perturbations. Later, heuristic-based methods such
as word removal (Ebrahimi et al., 2018), Out-Of-Vocabulary (OOV) words (Gao et al., 2018) and synonym replacement (Li et al., 2018; Jin et al., 2020; Alzantot et al., 2018) have been proposed. Among these studies, DeepWordBug (Gao et al., 2018) generates AEs by randomly transforming a word by OOV word in an example. This approach is practical in producing AEs efficiently; however, it generates AEs that can be detected by the end-user due to large proportion of lexical errors. An attack framework named TextBugger (Li et al., 2018) was proposed to generate adversarial samples using the multi-level approach. It identified important words for each example and replaced them with optimal bugs. The approach considered both character-level and word-level transformations. Another recent attack, TextFooler (Jin et al., 2020) generates utility-preserving AEs by replacing an important word in an example with its grammatically equivalent synonym. The study also evaluates the generated AEs against semantic similarity constraint.
In this study, we compare our method with TextBugger and TextFooer.
3 ReinforceBug
Definition 1 A Deep Neural Network (DNN) is a machine learning model that learns a function  $F: X \to Y$  over training data which maps from input space  $X$  to a set of classes  $Y$ .  $F$  is then evaluated on testing data  $X'$  and  $F$  predicts the output label  $y'$  for  $x' \in X'$ .
Definition 2 Prediction Confidence Score  $PCS(x,y)$  of model  $F$  depicts the likelihood of an input  $x \in X$  having a label  $y \in Y$ . The smaller  $PCS(x,y)$  suggests  $F$  has low confidence that  $x$  has a label  $y$ .
Definition 3 Given a real example  $x$  having a label  $y$ , a utility-preserving AE against  $x$  is an input  $x' \gets x + \imath$  with a minor perturbation  $\imath$  such that  $x'$  satisfies a set of perturbation constraints  $P_{\mathrm{const}}$  and  $F$  predicts an incorrect label for it with high PCS i.e.,  $y' \gets F(x')$  such that  $y' \neq y$  and  $PCS(x',y') > \nu$ .
Definition 4 A black-box attack is an attack where an attacker does not know the target model  $F$  architecture, training data  $X$  or hyperparameters  $\theta$ . An attacker can only query  $F$  with
Figure 1: Overall of ReinforceBug
the input examples  $X^{\prime}$  and gets the corresponding  $PCS(x,y)$ .
Definition 5 A non-targeted attack is an attack in which the adversary's goal is to maximize the misclassification rate of the model  $F$  on any generated AE irrespective of its ground-truth label  $y$ . i.e., fool the model  $F$  to misclassify a spam email as benign email or vice-versa.
Given a pre-trained target model  $F$ , we need to simulate a non-targeted (Definition 5) black-box attack (Definition 4) (Morris et al., 2020) to generate a set of utility-preserving AEs (Definition 3)  $A_{\mathrm{exp}}$  from a Corpus  $C$  with  $N$  examples and having corresponding target label  $t_g \in Y$ . Furthermore, our approach should learn a policy  $\pi_{\theta(s,a)}$  to perform perturbation  $\iota$  on  $C$  such that the model generates utility-preserving AEs (Definition 3) that have semantic similarity  $(> \varepsilon)$  with the original example but have low perturbation rate (number of words perturbed) and lexical (grammatical and spelling) errors. Moreover, the policy should be transferable on unseen datasets.
3.3 System Overview
Fig 1 provides an overview of ReinfoceBug. We model an attack as a reinforcement learning (Sutton and Barto, 2018) process consisting of three main components: an environment, Proximal Policy Optimization (PPO) agent (Schulman et al., 2017) and action space. Firstly, the environment state  $(s_t)$  at time  $t$  is processed as input by an agent followed by an action  $a_t \in A$  determined by an agent to update  $s_t$  to the next state  $s_{t+1}$ , where  $A$  represents an action space (set of valid actions given the state).
Subsequently, the environment's actuator acts on the corpus state to construct candidate examples. These examples are then sent to the reward generator module, which is responsible for computing the reward  $r_t + 1$  for action  $a_t$ . The reward generator applies post constraints  $Post \subset P_{\mathrm{const}}$ , queries the target model  $F$  and obtain scores of the candidate examples to calculate the reward of action  $a_t$ . The reward is sent back to the actuator which determines a valid update to the corpus state as well as the next environmental state  $s_{t+1}$ . The experience consisting of  $< s_t, a_t, r_{t+1}, s_{t+1}, va_{t+1} >$  is sent to the agent and the agent model is updated. Here  $va_{t+1}$  shows the valid actions mask for next state. The  $va_{t+1}$  is then used by the agent to update an action space  $A$  and restrict the agent to only select valid action on the new state  $s_{t+1}$ . Each of the modules is discussed below:
3.3.1 Agent
We use a customized version of Proximal Proximate Optimal (PPO) (Tang et al., 2020) Reinforcement Learning (RL) agent with action mask capability. PPO is an enhanced version of Actor-Critic (AC) Model (Grondman et al., 2012). In AC architecture the agent consists of a separate policy and Q-value network. They both take in environment state  $s_t$  at each time step as an input, while actor determines an action  $a_t$  among the possible valid action set  $A$  and critic yields value estimation  $V_t(s_t)$ . While an actor uses a gradient descent algorithm to learn a policy that maximizes the rewards  $R_t$ , the critic learns to estimate  $R_t$  via minimizing a temporal loss. Further, the PPO algorithm avoids large and inefficient policy updates by constraining the new policy updates to be as close to the origi
nal policy as possible. We have selected this agent because our action and state space is substantially large and to avoid enormous policy updates which make the agent unstable (please refer to Schulman et al. (2017) for more details).
3.3.2 Environment
The environment takes action  $a_{t}$  from the agent as input and outputs the experience tuple  $e_{t}$ , AEs  $A_{\mathrm{exp}}$  and a flag done depicting the success of the agent in achieving the goal.
3.3.3 States
The environment maintains the following states.
1. Corpus State  $(C_t)$ : it is given by  $C_t = E_1, E_2, \dots, E_N$ , where  $N$  is the number of examples in the corpus and  $E_i$  is equivalent to the set of words  $W_i = w_1, w_2, \dots, w_n$  for an example  $i$  at time  $t$ .
2. Score State  $(\text{score}_t)$ : it is a vector representing the PCS (Definition 2) of target model  $F$  on examples  $E$  having a ground-truth label  $tg$  in corpus  $C$  at time  $t$ . For instance, given an example  $E_i$  the  $\text{score}_t[i] = \text{PCS}(E_i, tg)$  where  $tg$  is a ground truth label of  $E_i$ .
3. Environment State  $(s_t)$ : this state  $s_t = w_1, w_2, \ldots, w_k$  is observable by an agent. It consists of  $k$  mutable important words in corpus  $C$ .
4. Success rate ( $success_{t}$ ): it is the proportion of utility-preserving AEs compared to all AEs generated by the agent at time  $t$ .
3.3.4 Components
The environment contains following components:
Word selector This component takes  $C$  state at  $t = 0$  and pre-constraints  $Pre \subset P_{\mathrm{const}}$  as inputs.  $Pre$  are task-specific perturbation restrictions on the specific entities in  $E_i$ . For example, spam messages mostly contain URLs, IP addresses, organization names and email addresses pointing to phishing websites. Perturbing these entities in the spam message can change the functional semantics of the message (Ranganayakulu and Chellappan, 2013). Hence, imposing pre-constraints ensures that generated  $AE$  preserves the functional equivalence to  $E_i$  after applying perturbations. To achieve this, we have designed a countvectorizer (sklearn) using a customized tokenizer. The tokenizer finds
the list of immutable entities such as URLs and IP addresses in the text using regular expressions or named entity models (Florian et al., 2003). After that, these words are segmented into immutable words  $\text{Im\_words}$  using word tokenizer and saved for each example to be utilized later by actuator module. For training ReinforceBug, our method first computes the important words from the training datasets as state  $(s_t)$  and then learns a policy to identify best actions  $(a_t)$  to transform the  $s_t$  to a next state  $(s_{t+1})$  such that the success of our attack is maximum.
Our work extends Jin et al. (2020) important words component; however instead of generating example-specific words, our module identifies corpus-specific important words. An important word is selected using a word importance score  $I_{w_{idx}}$ . The  $I_{w_{idx}}$  is calculated as the sum of prediction change in all the examples (k) containing  $w_{idx}$  before and after deleting  $w_{idx}$ . The candidate words  $w_{idx} \in$  most frequent words in the training dataset vocabulary. It is formally defined as follow.
$$
I _ {w _ {i d x}} = \left\{ \begin{array}{l} \sum_ {i = 1} ^ {k} \left(\frac {\left| F \left(E _ {i}\right) - F \left(w _ {i d x} \notin E _ {i}\right) \right|}{F \left(E _ {i}\right)}\right) \\ \frac {k}{k} \end{array} \right. \tag {1}
$$
If the  $I_{widx}$  is  $> 0$ , we consider it as an important word. The final list of all the important words is considered as the state  $st$ .  $st$  and mapping  $C_{\mathrm{map}}$  that maps each word to the corpus examples are sent back to the environment. For testing, the designed countvectorizer is used to transform the testing data onto these selected words.
Actuator This module is responsible to execute an action  $a_{t}$  selected by an agent. Firstly, the actuator transforms action  $a_{t}$  into an action tuple  $< w_{\mathrm{idx}}, act_{\mathrm{idx}}, rep_{\mathrm{idx}} >$ , where  $w_{\mathrm{idx}}, act_{\mathrm{idx}}$  and  $rep_{\mathrm{idx}}$  depict the index of the word to be replaced, the operation to be performed on that word and replacement word index respectively. Subsequently, example indexes  $E_{\mathrm{idx}}$  containing the word  $w_{\mathrm{idx}}$  are obtained by querying the  $C_{\mathrm{map}}$ . After that, the actuator examines the  $score_{t}[k]$  (score state) of each example  $k$  in  $E_{\mathrm{idx}}$ . If  $score_{t}[k] > \nu$ , only then it is selected as an example to be perturbed, here  $\nu$  represent the PCS threshold. In this way, the examples for which AEs have already been found are not perturbed further and other examples are given a chance. Once these examples are selected, the operation  $act_{\mathrm{idx}}$  is applied to  $w_{\mathrm{idx}}$  which results in multiple replacement options.
Table 1: List of Operations
Operation	Description	Example
Homoglyph Insertion	Replace a char with visually similar char.	potentially vs potentially
Insert a char in a word.	nearly vs n3early
BitSquatting	Replace a character with one bit different char.	clearly vs clearly
Omission	omit one char.	standard vs stanard
Addition	Add a char on start or end of a word.	classified vs classified
Repetition	Repeat the previous char in the word.	requested vs requested
get synonyms	Replace the word with its synonym	control vs dominance
get_semantic	Append the word with a word contextually related word	like vs such like
get_syntactic	Replace the word with a word having similar syntax.	bta vs bat
currency_word	If word contains currency symbol replace it with currency word and vice-versa.	$ vs dollar
word_to_num	If word represents a number replace it with the number.	eighth vs 8
num_to_word	If word is a number replace it to English word.	8 vs eight
word Weekday	If word is a weekday, replace it with its abbreviation or vice-versa	Wednesday vs wed
word_month	If word is a month, replace it with its abbreviation vice-versa	August vs Aug
Table 1 provides the list of operations considered. For example, if an operation is Homoglyph and the word indexed by  $w_{\mathrm{idx}}$  is "solid" then following replacements can be done "solid,sol1d,s0lid,5olid". The new word  $w_{\mathrm{new}}$  is selected by  $rep_{\mathrm{idx}}$ . After  $w_{\mathrm{new}}$  is selected against the  $w_{\mathrm{old}}$ , if  $w_{\mathrm{old}}$  in not in the immutable word list of example  $k$  than a candidate  $AEs$  are generated by substituting  $w_{\mathrm{old}}$  with  $w_{\mathrm{new}}$  in the previously selected examples. In this way the functional equivalence is ensured before generation of  $AE$ . The selected candidate examples are then sent to the reward generator. The reward generator module returns the reward of changing the  $w_{\mathrm{idx}}$  in the selected examples by applying  $act_{\mathrm{idx}}$  and selecting  $rep_{\mathrm{idx}}$ . The reward generator also outputs the AEs that satisfies all the post constraints. Finally, the state is updated by replacing the  $w_{\mathrm{idx}}$  in  $s_t$  with index of new word  $w_{\mathrm{new}}$ . All further actions on  $w_{\mathrm{idx}}$  are then invalidated. This is done by setting the  $va_{t+1}$  for all actions on the  $w_{\mathrm{idx}}$  in an action_space to False. In this way, multiple actions on the same word cannot be performed in one training episode. The  $success_{t+1}$  is updated, the episode completes if the  $success_{t+1}$  for the corpus state  $C_{t+1}$  has reached the threshold specified by  $\phi$  or all the words in the state have been updated. The module constructs an experience tuple  $e_t$  for the agent and returns  $e_t$ ,  $C_{t+1}$ ,  $score_{t+1}$  and  $success_{t+1}$  to the environment.
Reward Generator (RG) Algorithm 1 shows the pseudo-code of the RG module. RG takes candidate examples,  $C_t$ , time  $t$  and post constraints as input and outputs the  $Adv_{\mathrm{exp}}$ , the reward  $r_{t+1}$  of  $a_t$  and updated corpus state  $C_{t+1}$ . In this study, we have considered three main post utilities that the generated  $Adv_{\mathrm{exp}}$  should preserve apart from
ALGORITHM 1: Reward_Generator
changing the output of the classifier. Firstly, the percentage of the spelling and grammatical errors should not be more than  $\eta$  and  $\iota$  respectively, and the semantic similarity between the original and  $AE$  should be  $> \varepsilon$ . If the candidate example meets all these utilities, then the target_model is queried to obtain the score of the candidate example. RG updates the corpus and score states for an example where the difference between previous (before perturbation) and new (after perturbation) score of an example  $> 0$ .
Subsequently, the reward generator checks if the candidate example score has converged to  $< \nu$  or not. When converged case, the example is added
to the list of valid  $AE$  against the original example, and a reward  $r_{t+1}$  is computed as shown in line 30 of Algorithm 1. Otherwise, the sum of the change in the scores of an example is added as a reward, as shown in line 31. In line 30, the  $r_{t+1}$  represents the summation of reward attained on successfully transforming the original examples into well-constrained AE. It is calculated as a summation of the change in original and current score of the example  $k$ , assisted by semantic similarity with the original example and penalized by lexical errors (spelling and grammatical) and perturbation rate  $p_{rate}$  for generating each AE. In this way, the agent learns to generate AEs with minimum perturbations and lexical errors and more PCS and semantic similarity. Lastly, the  $r_{t+1}$  is normalized by the factor  $N'$ , so that the agent can learn the impact of action  $a_t$  on a single example irrespective of the size of the corpus.
Target Model We have considered a black-box attack (Definition 4) against the target model  $F$ .  $F$  can be any DNN (Definition 1) that provides  $PCS$  (Definition 2) score as an output.
This section presents our experiment details.
We studied the effectiveness of ReinforceBug on three noisy and one conventional text classification tasks respectively. Table 2 enlists the statistics of the considered datasets. Precisely during the target model training, we held out  $30\%$  of the training data as a validation set, and all parameters were tuned based on it. After that, the testing dataset was used to evaluate the performance of the model. For training and testing our ReinforceBug, we split the dataset into training  $(70\%)$  and testing dataset  $(30\%)$ . The stratified split was applied to ensure the class distribution on these datasets remains consistent with the actual testing dataset.
4.2 Attacking Target Models
For each dataset, we trained four state-of-the-art models namely Word Convolution Neural Network (CNN) (Jain et al., 2018), Character CNN (Zhang et al., 2015), Word Bidirectional Long Term Short Memory (BiLSTM) (Zhou et al., 2016) and Recurrent CNN (Lai et al., 2015) on the training set. However, we did not consider the recent state-ofthe-art BERT (Devlin et al., 2019) model because
we found that its performance was significantly low for our considered noisy datasets. Srivastava et al. 2020 also reported that the BERT model's performance notably degrades for noisy text datasets, and further research is required to fine-tune BERT for noisy datasets.
For training the considered models, we used an open-source GitHub repository (Lee). Table 3 shows the performance of each model on the testing set. From these models, we selected the models with best performance accuracy as target models (as highlighted in bold) to train ReinforceBug. For the rest of the models trained on a similar dataset, we studied the transferability of our generated AEs. Moreover, we tested our ReinforceBug against an unseen dataset to study the transferability of our attack on other datasets. Lastly, for get_semantic and get synonym operations (Pennington et al., 2014) and (Mrkšić et al., 2016) embeddings had been used.
4.3 PPO Agent
We used stable-baselines (Hill et al., 2018) reinforcement learning library to implement our PPO agent. We used a Multilayer perception (MLP) model, as an actor and critic models. For training the agent, we used 30 episodes for each model.
4.4 Utility-Preservation Constraints