SlowGuess/ACL_2020_OCR huggingface dataset:
Wei Zou<sup>1</sup> Shujian Huang<sup>1</sup> Jun Xie<sup>2</sup> Xinyu Dai<sup>1</sup> Jiajun Chen<sup>1</sup>
$^{1}$ National Key Laboratory for Novel Software Technology, Nanjing University, China  $^{2}$ Tencent Technology Co, China
zouw@smail.nju.edu.cn, {huangsj, daixinyu, chenjj} @nju.edu.cn  
stiffxie@tencent.com
Neural machine translation systems tend to fail on less decent inputs despite its significant efficacy, which may significantly harm the credibility of these systems—fathoming how and when neural-based systems fail in such cases is critical for industrial maintenance. Instead of collecting and analyzing bad cases using limited handcrafted error features, here we investigate this issue by generating adversarial examples via a new paradigm based on reinforcement learning. Our paradigm could expose pitfalls for a given performance metric, e.g., BLEU, and could target any given neural machine translation architecture. We conduct experiments of adversarial attacks on two mainstream neural machine translation architectures, RNN-search, and Transformer. The results show that our method efficiently produces stable attacks with meaning-preserving adversarial examples. We also present a qualitative and quantitative analysis for the preference pattern of the attack, demonstrating its capability of pitfall exposure.
Neural machine translation (NMT) based on the encoder-decoder framework, such as RNN-Search (Bahdanau et al., 2014; Luong et al., 2015, RNNSearch) or Transformer (Vaswani et al., 2017, Transformer), has achieved remarkable progress and become a de-facto in various machine translation applications. However, there are still pitfalls for a well-trained neural translation system, especially when applied to less decent real-world inputs compared to training data (Belinkov and Bisk, 2017). For example, typos may severely deteriorate system outputs (Table 1). Moreover, recent studies show that a neural machine translation system can also be broken by noisy synthetic inputs (Belinkov and Bisk, 2017; Lee et al., 2018). Due to the black-box nature of a neural system, it has
in	耶路撒冷发生自杀爆炸事件
out	suicide bombing in jerusalem
in	耶路撒冷发生自杀爆事件
out	eastern jerusalem explores a case of eastern europe
Table 1: Fragility of neural machine translation. A typo leaving out a Chinese character "炸" leads to significant errors (noted by italics) in English translation. Both "爆" and "爆炸" mean "bombing" in English.
been a challenge to fathom when and how the system tends to fail.
Intuitively, researchers seek to apprehend such failures by the analysis of handcrafted error indicating features (Zhao et al., 2018; Karpukhin et al., 2019). This strategy is costly because it requires expert knowledge for both linguistics and the target neural architecture. Such features are also less applicable because some common errors in deep learning systems are hard to formulate, or very specific to certain architectures.
Instead of designing error features, recent researchers adopt ideas from adversarial learning (Goodfellow et al., 2014) to generate adversarial examples for mining pitfalls of NLP systems (Cheng et al., 2018a; Ebrahimi et al., 2018; Zhao et al., 2017). Adversarial examples are minor perturbed inputs that keep the semantic meaning, yet yield degraded outputs. The generation of valid adversarial examples provides tools for error analysis that is interpretable for ordinary users, which can contribute to system maintenance. Though it has achieved success concerning continuous input, e.g., images, there are following major issues for NLP tasks.
First, it is non-trivial to generate valid discrete tokens for natural language, e.g., words or characters. Cheng et al. (2018a) follow Goodfellow et al. (2014) to learn noised representation then sample tokens accordingly. However, there is no guaranteed correspondence between arbitrary representation and valid tokens. Therefore, it may gen
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3486-3497 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
in	Two man are playing on the street corner.
perturbed in	Two man are playing frisbee in the park.
out	Zwei Männer spielen an einer Straßenecke.
perturbed out	Zwei Männer spielen frisbee im park.
Table 2: Example of undesirable perturbation in adversarial examples for machine translation in (Zhao et al., 2017), though it yields very different output compare to the origin, it does not indicate system malfunction.
erate tokens departing from learned representation, which undermines the generation. Ebrahimi et al. (2018) turns to a search paradigm by a brute-force search for direct perturbations on the token level. To lead the search, a gradient-based surrogate loss must be designed upon every token modification by given target annotations. However, this paradigm is inefficient due to the formidable computation for gradients. Furthermore, surrogate losses defined upon each token by targets requires high-quality targets, and risks being invalidated by any perturbation that changes tokenization.
Another issue is to keep the semantics of original inputs. Different from the fact that minor noises on images do not change the semantics, sampling discrete tokens from arbitrary perturbed representation (Cheng et al., 2018a) may generate tokens with different semantics and lead to ill-perturbed samples (Table 2). Searching for the perturbed input also requires a semantic constraint of the search space, for which handcrafted constraints are employed (Ebrahimi et al., 2018). Though constraints can also be introduced by multitask modeling with additional annotations (Zhao et al., 2017), this is still not sufficient for tasks requiring strict semantic equivalence, such as machine translation.
In this paper, we adopt a novel paradigm that generates more reasonable tokens and secures semantic constraints as much as possible. We summarize our contributions as the following:
- We introduce a reinforcement learning (Sutton and Barto, 2018, RL) paradigm with a discriminator as the terminal signal in its environment to further constrain semantics. This paradigm learns to apply discrete perturbations on the token level, aiming for direct translation metric degradation. Experiments show that our approach not only achieves semantically constrained adversarial examples but also leads to effective attacks for machine translation.
- Our paradigm can achieve the adversarial example generation with outclassed efficiency by only given source data. Since our method is model-agnostic and free of handcrafted error feature targeting architectures, it is also viable among different machine translation models.
- We also present some analysis upon the state-of-the-art Transformer based on its attack, showing our method's competence in system pitfall exposure.
The most popular architectures for neural machine translation are RNN-search (Bahdanau et al., 2014) and Transformer (Vaswani et al., 2017). They share the paradigm to learn the conditional probability  $P(Y|X)$  of a target translation  $Y = [y_{1},y_{2},\dots,y_{m}]$  given a source input  $X = [x_{1},x_{2},\dots,x_{n}]$ . A typical NMT architecture consists of an encoder, a decoder and attention networks. The encoder encodes the source embedding  $X_{emb} = [emb_{1},emb_{2},\dots emb_{n}]$  into hidden representation  $H = [h_1,h_2,\dots,h_n]$ . Then a decoder  $f_{dec}$  with attention network attentively accesses  $H$  for an auto-regressive generation of each  $y_{i}$  until the end of sequence symbol (EOS) is generated:
$$
P \left(y _ {i} \mid y _ {<   i}, X\right) = \operatorname {s o f t m a x} \left(f _ {d e c} \left(y _ {i - 1}, s _ {t}, c _ {t}; \theta_ {d e c}\right)\right) \tag {1}
$$
where  $c_{t}$  is the attentive result for current decoder state  $s_{t}$  given  $H$ .
2.2 Actor-Critic for Reinforcement Learning
Reinforcement learning (Sutton and Barto, 2018, RL) is a widely used machine learning technique following the paradigm of **explore and exploit**, which is apt for unsupervised policy learning in many challenging tasks (e.g., games (Mnih et al., 2015)). It is also used for direct optimization for non-differentiable learning objectives (Wu et al., 2018; Bahdanau et al., 2016) in NLP.
Actor-critic (Konda and Tsitsiklis, 2000) is one of the most popular RL architectures where the agent consists of a separate policy and value networks called actor and critic. They both take in environment state  $s_t$  at each time step as input, while
Figure 1: [a] Overview of our RL architecture. ① Environment states are processed as inputs for agent; ② agent yield modification upon SRC in environment; ③ determine survival and step reward of environment; ④ determine degradation with victim NMT as episodic reward; ⑤ update agent with total rewards. During a training episode, we loop ① to ③ and accumulate step rewards until environment terminates. Dash line indicates execution at the end of an episode. [b] Architecture of discriminator. [c] Architecture of agent.
actor determines an action  $a_{t}$  among possible action set  $\mathcal{A}$  and critic yields value estimation  $V_{t}(s_{t})$ . In general, the agent is trained to maximize discounted rewards  $R_{t} = \sum_{i=0}^{\infty} \gamma^{i} r_{t+i}$  for each state, where  $\gamma \in (0,1]$  is the discount factor. Such goal can be further derived as individual losses applied to actor and critic. Thus the actor policy loss  $L^{\pi}$  on step  $t$  is:
$$
L _ {t} ^ {\pi} \left(\theta_ {\pi}\right) = \log P \left(a _ {t} \mid s _ {t}\right) A _ {t} \left(s _ {t}, a _ {t}\right); a _ {t} \in \mathcal {A} \tag {2}
$$
where  $\theta_{\pi}$  denotes actor parameters,  $A_{t}(s_{t},a_{t})$  denotes general advantage function (Schulman et al., 2015) on state  $s_t$  for action  $a_{t}$  given by  $\sum_{i = 0}^{k - 1}\gamma^i r_{t + i} + \gamma^k V(s_{t + k}) - V(s_t)$ , which can be further derived as:
$$
\begin{array}{l} A _ {t} \left(s _ {t}, a _ {t}\right) = \gamma A _ {t + 1} \left(s _ {t + 1}, a _ {t + 1}\right) + r _ {t} \\ + \gamma V _ {t + 1} \left(s _ {t + 1}\right) - V _ {t} \left(s _ {t}\right) \tag {3} \\ \end{array}
$$
Meanwhile, critic learns to estimate  $R_{t}$  via minimizing a temporal difference loss  $L^{v}$  on each step  $t$ :
$$
L _ {t} ^ {v} \left(\theta_ {v}\right) = \frac {1}{2} \left(r _ {t} + \gamma R _ {t + 1} - V _ {t} \left(s _ {t}\right)\right) ^ {2} \tag {4}
$$
where  $\theta_v$  denotes critic parameter.
Usually, the training is regularized by maximizing policy entropy  $H^{\pi}$  to avoid exploration failure before exploiting optimum policy (Ziebart, 2010). Thus the total loss becomes:
$$
L (\theta) = \sum_ {t} \left(\alpha L _ {t} ^ {v} - L _ {t} ^ {\pi} - \beta H ^ {\pi} (\cdot | s _ {t})\right) \tag {5}
$$
where  $\alpha$  and  $\beta$  are hyperparameters for value loss and entropy coefficients.
2.3 adversarial examples in NLP
A general adversarial example generation can be described as the learning process to find a perturbation  $\delta$  on input  $X$  that maximize system degradation  $L_{adv}$  within a certain constraint  $C(\delta)$ :
$$
\underset {\delta} {\operatorname {a r g m a x}} L _ {a d v} (X + \delta) - \lambda C (\delta) \tag {6}
$$
where  $\lambda$  denotes the constraint coefficient,  $L_{adv}$  is determined by the goal of the attack. However, currently effective adversarial generation for NLP is to search by maximizing a surrogate gradient-based loss:
$$
\operatorname * {a r g m a x} _ {1 \leq i \leq n, x ^ {\prime} \in v o c a b} L _ {a d v} \left(x _ {0}, x _ {1}, \dots x _ {i} ^ {\prime} \dots x _ {n}\right) \tag {7}
$$
where  $L_{adv}$  is a differentiable function indicating the adversarial object. Due to its formidable search space, this paradigm simply perturbs on a small ratio of token positions and greedy search by brute force among candidates. Note that adversarial example generation is fundamentally different from noised hidden representation in adversarial training (Cheng et al., 2019; Sano et al., 2019), which is not to be concerned in this work.
In this section, we will describe our reinforced learning and generation of adversarial examples (Figure 1) in detail. Overall, the victim model is a part of the environment (denoted as  $Env$ ), which yields rewards indicating overall degradation based on modified inputs. A reinforced agent
learns to modify every source position from left to right sequentially. Meanwhile, a discriminator in Env provides every-step survival signals by determining whether SRC is ill-perturbed.
3.1 Environment
We encapsulate the victim translation model with a discriminative reward process as an Env for a reinforced agent to interact.
3.1.1 Environment State
The state of the Env is described as  $s_t = (SRC, t)$ , where  $SRC = [src_0, src1, \dots, src_N]$  are  $N$  sequences processed by victim model's vocabulary and tokenization. Each sequence  $src_i = [x_1, x_2, \dots, x_n]$  is concatenated with  $BOS, EOS$ , which indicate the begin and end of the sequence, then padded to same length. Time step  $t \in [1, n]$  also indicates the token position to be perturbed by the agent. Env will consecutively loop for all token positions and update  $s_t$  based on the agent's modification. Env also yields reward signals until the end or intermediately terminated. That is, all sequences in SRC are determined by  $D$  as ill-perturbed during the reward process. Once the Env terminates, it finishes the current episode and reset its state with a new batch of sequences as SRC.
3.1.2 Reward Process with Discriminator
The reward process is only used during training. It consists of a survival reward  $r_{\mathrm{s}}$  on every step and a final degradation  $r_{\mathrm{d}}$  concerning an overall metric if the agent survives till the end. Overall, we have:
$$
r _ {t} = \left\{ \begin{array}{l} - 1, \text {t e r m i n a t e d} \\ \frac {1}{N} \sum_ {N} a \cdot r _ {\mathrm {s}}, \text {s u r v i v e} \& t \in [ 1, n) \\ \frac {1}{N} \sum_ {N} (a \cdot r _ {\mathrm {s}} + b \cdot r _ {\mathrm {d}}), \text {s u r v i v e} \& t = n \end{array} \right. \tag {8}
$$
where  $a, b$  are hyper parameters that keeps the overall  $r_{\mathrm{s}}$  and  $r_{\mathrm{d}}$  within similar magnitude.
Instead of direct optimization of the constrained adversarial loss in Eq.6, we model discriminator  $D$ 's output as survival rewards similar to that in gaming (Mnih et al., 2015). That is, the agent must survive for its goal by also fooling  $D$ , which attempts to terminate ill-perturbed modifications. We define an ill-perturbed source by determining whether it still matches the original target  $tgt$ .
Discriminator As it is shown in Figure 1(b), discriminator  $D$  consists of bi-directional GRU encoders for both source and target sequence. Their corresponding representation is averaged and concatenated before passed to a feedforward layer with dropout. Finally, the output distribution is calculated by a softmax layer. Once  $D$  determines the pair as positive, its corresponding possibility is regarded as the reward, otherwise 0:
$$
r _ {\mathrm {s}} = \left\{ \begin{array}{l l} P (\text {p o s i t i v e} | \left(s r c ^ {\prime}, t g t\right); \theta_ {d}), & \text {p o s i t i v e} \\ 0, & \text {o t h e r w i s e} \end{array} \right. \tag {9}
$$
As long as the environment survives, it yields averaged reward among samples from SRC (Eq.8) to mitigate rewards' fluctuation that destabilize training.
Discriminator Training Similar to GAN training, the environment's  $D$  must update as the agent updates. During its training, the agent's parameter is freezed to provide training samples. For every  $D$ 's training epoch, we randomly choose half of the batch and perturb its source using the current agent as negative samples. During  $D$ 's updates, we randomly generate a new batch of pairs from parallel data likewise to test its accuracy.  $D$  is updated at most  $step_{D}$  epochs, or until its test accuracy reaches acc_bound.
Env only yields -1 as overall terminal rewards when all sequences in SRC are intermediately terminated. For samples classified as negative during survival, their follow-up rewards and actions are masked as 0. If the agent survives until the end, Env yields additional averaged  $r_{\mathrm{d}}$  as final rewards for an episode. We follow Michel et al. (2019) to adopt relative degradation:
$$
r _ {\mathrm {d}} = \frac {\operatorname {s c o r e} \left(y , r e f s\right) - \operatorname {s c o r e} \left(y ^ {\prime} , r e f s\right)}{\operatorname {s c o r e} \left(y , r e f s\right)} \tag {10}
$$
where  $y$  and  $y^\prime$  denote original and perturbed output, refs are references, and score is a translation metric. If  $\text{score}(y,\text{refs})$  is zero, we return zero as  $r_{\mathrm{d}}$ . To calculate score we retokenize perturbed SRC by victim models vocabulary and tokenizer before translation.
It is commonly accepted that frequent negative rewards result in agents' tendency to regard zero-reward as optimum and fail exploration, which further leads to training failure.
3.2 Agent
As it is shown in Figure 1 (c), the agent's actor and critic share the same input layers and encoder, but later processed by individual feedforward layers and output layers. Actor takes in  $SRC$  and current token with its surrounding  $(x_{t-1}, x_t, x_{t+1})$ , then yields a binary distribution to determine whether to attack a token on step  $t$ , while critic emits a value  $V(s_t)$  for every state. Once the actor decides to perturb a specific token, this token will be replaced by another token in its candidate set.
Candidate Set We collect at most  $K$  candidates for each token in the victim's vocabulary within a distance of  $\epsilon$ .  $\epsilon$  is the averaged Euclidean distance of  $K$ -nearest embedding for all tokens in victim vocabulary. We note that there shall always be candidates for a token in test scenarios that are beyond victim's vocabulary, for those without a nearby candidate, we assign UNK as its candidate. Once the agent chooses to replace a token with UNK, we follow Michel et al. (2019) to present a valid token that is also UNK to the victim's vocabulary.
Agent Training The agent is trained by algorithm in appendix A. Since the agent is required to explore with stochastic policy during training, it will first sample based on its actor's output distribution on whether to perturb the current position, then randomly choose among its candidates. The agent and discriminator take turns to update. We assume the training is converged when test accuracy for  $D$  does not reach over a certain value within certain continuous learning rounds of agent and discriminator.
Agent Inference To generate adversarial examples, the agent will take in source sequences and perturb on each position based on the actor's output from left to right, then choose the nearest candidate. As the agent's critic learns to estimate expected future rewards for a step, only when it yields positive value will agent perturb, otherwise it indicates an undesirable perturbation; thus, the agent is muted.
4.1 Data Sets
We test our adversarial example generations on  $\mathrm{Zh}\rightarrow \mathrm{En}$ $\mathrm{En}\to \mathrm{Fr}$  and  $\mathrm{En}\to \mathrm{De}$  translation tasks,
which provide relatively strong baselines for victim models and mass test samples.
We train our agent using only parallel data that is used for victims' training. we train on LDC  $\mathrm{Zh}\rightarrow \mathrm{En}^2 (1.3\mathrm{M}$  pairs), WMT14  $\mathrm{En}\rightarrow \mathrm{De}^3$  (4.5M pairs) and WMT15  $\mathrm{En}\rightarrow \mathrm{Fr}^4 (2.2\mathrm{M}$  pairs) for victim models respectively. For subword level translation, we apply byte pair encoding (Sennrich et al., 2015, BPE) for both source and target languages with the vocabulary size of  $37\mathrm{k}$ . We also use join-BPE for En-De and En-Fr experiments with  $34\mathrm{k}$  and  $33\mathrm{k}$  vocabulary size, respectively. For word-level translation, we use NLPIR-ICTCLAS and Moses tokenizer for Chinese and English tokenization, respectively. We adopt  $30\mathrm{k}$  as vocabulary size for both source and target language. We adopt NIST test sets for  $\mathrm{Zh}\rightarrow \mathrm{En}$  and WMT test sets for  $\mathrm{En}\rightarrow \mathrm{De}$  and  $\mathrm{En}\rightarrow \mathrm{Fr}$ , then generate adversarial examples for these sources for analysis.
4.2 Victim Models
We choose the state-of-the-art RNN-search and Transformer as victim translation models. For RNN-search, we train subword level models and strictly follow the architecture in Bahdanau et al. (2014). As for Transformer, we train both word-level and subword-level model for  $\mathrm{Zh}\rightarrow \mathrm{En}$  and only subword-level models for  $\mathrm{En}\rightarrow \mathrm{De}$  and  $\mathrm{En}\rightarrow \mathrm{Fr}$  with the architecture and the base parameter settings by Vaswani et al. (2017). For the above models, we apply the same batch scheme and Adam optimizer following Vaswani et al. (2017). We choose MT03, newsdiscuss2015 and newstest2013 for  $\mathrm{Zh}\rightarrow \mathrm{En}$ ,  $\mathrm{En}\rightarrow \mathrm{Fr}$ ,  $\mathrm{En}\rightarrow \mathrm{De}$  as validation set respectively.
4.3 Metrics
We first report attack results both in terms of char-level BLEU (chrBLEU) of perturbed source by the origin to indicate modification rate, and relative decrease in target BLEU  $(RD)$ :
$$
R D = \frac {\mathrm {B L E U} \left(y , r e f s\right) - \mathrm {B L E U} \left(y ^ {\prime} , r e f s\right)}{\left(1 - \operatorname {c h r B L E U} \left(x ^ {\prime} , x\right)\right) \times \mathrm {B L E U} \left(y , r e f s\right)} \tag {11}
$$
We adopt sacreBLEU (Post, 2018) to test case-insensitive BLEU on detokenized targets.
$^{2}$ ldc2002E18, ldc2003E14, ldc2004T08, ldc2005T06
3 https://nlp.stanford.edu/projects/nmt/
4Europarl-v7, news-commentary-v10
5MT02,03,04,05,06
Zh-En MT02-06
BLEU	chrBLEU	RD↑	HE↑
Transformer-word	41.16	-	-	-
RSNI (0.2)*	29.68	0.892	2.580*	1.39*
RSNI (0.3)*	19.94	0.781	2.350*	1.10*
GS (0.2)	33.46	0.749	0.746	3.23
GS (0.3)	29.86	0.676	0.847	2.49
Ours	33.72	0.804	0.952	3.73
Transformer-BPE	44.06	-	-	-
RSNI (0.2)*	34.44	0.892	2.019*	1.45*
RSNI (0.4)*	25.78	0.781	1.891*	1.08*
GS (0.2)	35.52	0.823	1.094	3.88
GS (0.4)	28.18	0.675	1.004	2.90
Ours	35.48	0.807	1.009	3.79
RNN-search-BPE	40.90	-	-	-
RSNI (0.2)*	32.54	0.892	1.891*	1.44*
RSNI (0.4)*	25.54	0.781	1.712*	1.36*
GS (0.2)	32.94	0.823	1.102	3.79
GS (0.4)	27.02	0.678	1.053	2.88
Ours	31.58	0.785	1.059	3.81
As Michel et al. (2019) suggest, there is a tradeoff between achieving high  $RD$  and maintaining semantic. One can achieve rather high  $RD$  by testing with mismatched references, making degradation less meaningful. Therefore, we also test source semantic similarity with human evaluation  $(HE)$  ranging from 0 to 5 used by Michel et al. (2019) by randomly sampling  $10\%$  of total sequences mixed with baselines for a double-blind test.
We implement state-of-the-art adversarial example generation by gradient search (Michel et al., 2019) (GS) as a baseline, which can be currently applied to various translation models. We also implemented random synthetic noise injection (Karpukhin et al., 2019) (RSNI) as an unconstrained contrast. Both baselines are required to provide a ratio for the amount of tokens to perturb during an attack, where we present the best results. Unlike our paradigm can generate on monolingual data, GS also requires target annotations, where we use one of the references to provide a strong baseline. Note that RSNI can significantly break semantics with distinctly lower  $HE$  to achieve rather high  $RD$ , which we do not consider as legit adversarial example generation and noted with “*” for exclusion.
As it is shown in Table 3 and 4, our model