SlowGuess/ACL_2020_OCR huggingface dataset:
Table 3: Experiment results for  $\mathrm{Zh}\rightarrow \mathrm{En}$  MT attack. We list  $BLEU$  for perturbed test sets generated by each adversarial example generation method, which is expected to deteriorate. An ideal adversarial example should achieve high  $RD$  with respect to high  $HE$ .
En-De newstest13-16
BLEU	chrBLEU	RD↑	HE↑
RNN-search-BPE	25.35	-	-	-
RSNI (0.2)*	16.70	0.949	6.691*	2.32*
RSNI (0.4)*	10.05	0.897	5.860*	1.58*
GS (0.2)	19.42	0.881	1.966	3.81
GS (0.4)	9.27	0.680	1.982	3.01
Ours	21.27	0.921	2.037	3.95
Transformer-BPE	29.05	-	-	-
RSNI (0.2)*	18.775	0.949	6.935*	2.39*
RSNI (0.4)*	11.125	0.897	5.991*	1.58*
GS (0.2)	18.29	0.861	2.665	3.69
GS (0.4)	10.03	0.751	2.629	3.33
Ours	19.29	0.875	2.688	3.79
En-Fr newstest13-14 + newsdiscuss15
RNN-search-BPE	32.6	-	-	-
RSNI (0.2)*	21.93	0.947	6.175*	2.23*
RSNI (0.4)*	14.3	0.894	5.271*	1.56*
GS (0.2)	22.7	0.833	1.818	3.80
GS (0.4)	15.2	0.708	1.828	3.25
Ours	22.3	0.843	2.009	3.87
Transformer-BPE	34.7	-	-	-
RSNI (0.2)*	24.0	0.947	5.774*	2.34*
RSNI (0.4)*	15.8	0.894	5.114*	1.67*
GS (0.2)	23.01	0.830	1.982	3.74
GS (0.4)	19.6	0.788	2.053	3.68
Ours	21.33	0.798	1.907	3.78
Table 4: Experiment results for  $\mathrm{En}\rightarrow \mathrm{De}$  and  $\mathrm{En}\rightarrow \mathrm{Fr}$  MT attack.
stably generate adversarial examples without significant change in semantics by the same training setting among different models and language pairs, achieving stably high  $HE$  ( $>3.7$ ) without any handcrafted semantic constraints, while search methods (GS) must tune for proper ratio of modification, which can hardly strike a balance between semantic constraints and degradation. Unlike search paradigm relying on reference and victim gradients, our paradigm is model-agnostic yet still achieving comparable  $RD$  with relatively high  $HE$ .
4.5 Case Study
As it is shown in Table 5, our method is less likely to perturb some easily-modified semantics (e.g. numbers are edited to other "forms", but not different numbers), while search tends to generate semantically different tokens to achieve degradation. Thus our agent can lead to more insightful and plausible analyses for neural machine translation than search by gradient.
5.1 Efficiency
As it is shown in Figure 2, given the same amount of memory cost, our method is significantly more
a
origin in origin out	全国4000万选民将在16名候选人中选举法兰西第五共和国第七任总统。
40 million voters throughout the country will elect the seventh president of the fifth republic of france among the 16 candidates
references	40 million voters in the nation will elect the 7th president for the french fifth republic from 16 candidates.
there are 40 million voters and they have to pick the fifth republic france's seventh president amongst the sixteen candidates.
forty million voters across the country are expected to choose the 7th president of the 5th republic of france from among 16 candidates.
40 million voters around france are to elect the 7th president of the 5 republic of france from 16 candidates.
GS (0.4) in GS (0.4) out	全国性4000万市民将在6名候选人中选举法兰西第五国家第七任外交部长。
of the 6 candidates, 40 million people will elect the seventh foreign minister of the five countries.
ours in ours out	全国性4000万选民将在16位候选人中选举法兰西第5共和国第7任总统
among the 16 candidates, 40 million voters will elect five presidents of France and seven presidents of the republic of France.
b
origin in origin out	千案者目前被也门当局扣留。
the persons involved in the case are currently detained by the yemeni authorities.
references	the perpetrator is currently in the custody of the yemeni authorities.
yemeni authority apprehended the suspect.
the suspect is now in custody of yemeni authorities.
the ones involed in this case were also detained by the authority.
GS (0.4) in GS (0.4) out	千案者目前为也门现局留。
the person involved in the case is now detained by the authorities!
ours in ours out	千案方目前被也门当局扣留。
the victim is currently detained by the yemeni authorities.
Table 5: (a) an example of perturbed number and quantifier severely damaging outputs in  $\mathrm{Zh}\rightarrow \mathrm{En}$  translation, where we highlight the changes. "五" is the character for 5 and "七" for 7, "名" and "位" are both commonly used quantifiers for people. However, search-based attack achieves degradation by some significant changes of semantics, where number "16" is changed to "6", and "外交部长" means "foreign minister". (b) an example of changed suffix which breaks the result. "方" and "者" are common suffixes (K) sharing same meaning used for people. Our model spots that victim model's fragility upon such perturb, while search does not.
efficient compared to the search paradigm. Gradient computation concerning every modified source sequence can cost considerably in time or space for a state-of-the-art system, which could be even worse for systems with recurrent units. When it comes to mass production of adversarial examples for a victim translation system, our method can also generate by given only monolingual inputs. In contrast, search methods must be provided the same amount of well-informed targets.
Figure 2: Time consumption of different methods: we limit memory usage to 2.5G on single Nvidia 1080, and generate adversarial examples for the same 800 inputs in  $\mathrm{Zh}\rightarrow \mathrm{En}$  MT with different methods, our method significantly outclasses the state-of-the-art search paradigm (GS).
5.2 Attack Patterns
NMT systems may have different robustness over different parts of the inputs, thus some researchers implement input preprocessing targeting certain
empirically weak parts, e.g., named entities(Li et al., 2018). Since the agent's policy is to attack without handcrafted error features, we can further investigate vulnerability by its attack preferences of different parts of speech. We choose Chinese, for example, and adopt LTP POS tagger<sup>6</sup> to label NIST test sets, then check the modification rate for each POS. To ensure the reliability of our analysis, we run three rounds of experiments on both baselines and our agent with similar modification rate targeting state-of-the-art Transformer with BPE, and collect overall results. We also present random synthetic noise injection (Karpukhin et al., 2019) (RSNI), which is not intended for any preference as an additional baseline.
As it is shown in Figure 3, our reinforced paradigm shows distinct preference upon certain POS tags, indicating pitfalls of a victim translation system. At the same time, RSNI distributed almost evenly upon different POS tags. Though the search paradigm (GS) does expose some types of pitfall, our method can further expose those omitted by the search. Note that unlike existing work relying on feature engineering to indicate errors, we have no such features implemented for an agent. However, our agent can still spot error patterns by favoring some of the POS, such as
$^{6}$ https://github.com/HT-SCIR/ltp
Figure 3: Attack preferences of different paradigms targeting  $\mathrm{Zh}\rightarrow \mathrm{En}$  Transformer-BPE model. All share a similar modification rate. Our agent shows a significant preference for some POS (e.g., Ni, Nh, Nz, I), which are commonly regarded as hard-to-translate phrases among industrial implementations, while some (e.g., K) are less noticed. Preference among different choices.
Attack by	BLEU(Δ)
Zh-En MT02-06
RNN-search-BPE	-	40.90
agent-RNN	31.58(-9.32)
agent-TF	32.14(-8.76)
Transformer-BPE	-	44.06
agent-TF	35.48(-8.58)
agent-RNN	33.14(-10.92)
En-De Newstest13-16
RNN-search-BPE	-	25.35
agent-RNN	21.27(-4.08)
agent-TF	17.18(-8.18)
Transformer-BPE	-	29.05
agent-TF	19.29(-9.76)
agent-RNN	24.2(-4.85)
En-Fr Newstest13-14+newsdiscuss15
RNN-search-BPE	-	32.60
agent-RNN	22.3(-10.30)
agent-TF	19.83(-14.87)
Transformer-BPE	-	34.70
agent-TF	21.33(-13.37)
agent-RNN	22.35(-10.25)
Ni (organization name), Nh (person name), Nl (location name), M (numbers), which are commonly accepted as hard-to-translate parts. Moreover, the agent also tends to favor K (suffix) more, which is less noticed.
5.3 Attack Generalization
We additionally test agents by attacking different model architecture from the one that it's trained. As it is shown in Table 6, we perturb the inputs by agents trained to attack a different architecture, then test for degradation. The results show that our agent trained by targeting Transformer architecture can still achieve degradation on RNN-search, and vice-versa.
Table 6: Attacks targeting different architecture from the trained one. We note agent with the architecture that is trained with(e.g., agent-RNN stands for agent trained by targeting RNN-search).
Clean test	Noisy test	IWSLT11-17
Transformer-BPE	44.06	35.48	11.27
Tuned	43.60(-0.46)	40.31(+4.83)	11.73(+0.46)
Table 7: Tuning  $\mathrm{Zh}\rightarrow \mathrm{En}$  Transformer-BPE model with adversarial examples. We generate adversarial examples for every training sources for tuning, achieving overall improvements for noisy tests.
5.4 Tuning with Adversarial Examples
Since the agent generates meaning-preserving adversarial examples efficiently, we can directly tune the original model with those samples. We choose  $\mathrm{Zh} \rightarrow \mathrm{En}$  Transformer-BPE, for example, and generate the same amount of adversarial examples given original training sources(1.3M pairs), then paired with initial targets. We mix the augmented pairs with original pairs for a direct tuning. We test the tuned model on original test data and noisy test data generated by the attacking agent. We additionally test on IWSLT11-17  $\mathrm{Zh} \rightarrow \mathrm{En}$  test data, which is not used for training or tuning, to verified robustness improvement. As Table 7 shows, our agent can achieve substantial improvement(+4.83) on noisy tests with only minor loss on clean tests(-0.46). The improvement on the IWSLT test also indicates the adversarial tuning contributes to not only defending the agent's attack, but also overall robustness.
5.5 Reinforced Examples for Machine translation
We additionally switched the episodic rewards in the environment, then ignored all modifications that induce UNK tokens to train an agent, hoping to generate minor perturbed samples that can improve the translation metric. Though we failed to achieve overall improvements, we do succeed for quite a portion of samples, as shown in Table 8. Similar to adversarial examples, we call them reinforced examples. Such improvement is different from adversarial training that tunes model
in	钱其琛同突尼斯外长会谈。
perturbed in	钱其琛同突尼斯外长会谈。
out	Chinese, Tunisian ministers hold talks.
perturbed out	qian qichen holds talks with Tunisian foreign minister.
in	中巴及城巴车辆在南区通宵停泊
perturbed in	中巴及城巴车辆在南区通宵停车
out	overnight parking of cmb and city bus
perturbed out	overnight parking of cmb and city bus in southern district
Table 8: Example of minor perturbed samples that improves machine translation for  $\mathrm{Zh}\rightarrow \mathrm{En}$  Transformer-BPE model. The "。” in first sample is modified to "−", then model yields the omitted "钱其琛 (qian qi chen)". The "停泊" in second sample is modified to "停车", where they both mean "parking", then comes the omitted "in southern district" for "在南区".
for defense or strict text correction before the test phase. Reinforced examples are still noisy and can be directly applied for a test without any model updates to achieve improvements, which to our best knowledge is less investigated by researchers. Since we discovered that not all perturbed inputs are harmful, such an issue can be a good hint and alternative for better adversarial defense in NLP and should be further considered.
Cheng et al. (2018a) and Cheng et al. (2018b) applied continuous perturbation learning on token's embedding and then manage a lexical representation out of a perturbed embedding. Zhao et al. (2017) learned such perturbation on the encoded representation of a sequence, and then decode it back as an adversarial example. These methods are applicable for simple NLP classification tasks, while failing machine translation which requires higher semantic constraints. Zhao et al. (2017) further attempted to constrain semantic in such paradigm by introducing multi-task modeling with accessory annotation, which further limits applicability.
On the other hand, Ebrahimi et al. (2018), Chaturvedi et al. (2019) and Cheng et al. (2019) regarded it as a search problem by maximizing surrogate gradient losses. Due to the formidable gradient computation, such methods are less viable to more complex neural architectures. Cheng et al. (2019) introduced a learned language model to constrain generation. However, a learned language model is not apt for common typos or UNK. Another pitfall of this paradigm is that surrogate losses defined by a fixed tokenization for non-character level systems, risks being invalidated once the attack changes tokenization. Therefore,
Ebrahimi et al. (2018) simply focused on char-level systems, while Michel et al. (2019) specially noted to exclude scenarios where attack changes tokenization in their paradigm.
Other works turn to more sophisticated generation paradigms, e.g., Vidnerova and Neruda (2016) adopts a genetic algorithm for an evolutionary generation targeting simple machine learning models. Zang et al. (2019) consider adversarial generation as a word substitution-based combinatorial optimization problem tackled by particle swarm algorithm. Our paradigm shares some common ideology with Miao et al. (2019) and Xiao et al. (2018), which iteratively edit inputs constrained by generative adversarial learning.
We propose a new paradigm to generate adversarial examples for neural machine translation, which is capable of exposing translation pitfalls without handcrafted error features. Experiments show that our method achieves stable degradation with meaning preserving adversarial examples over different victim models.
It is noticeable that our method can generate adversarial examples efficiently from monolingual data. As a result, the mass production of adversarial examples for the victim model's analysis and further improvement of robustness become convenient. Furthermore, we notice some exceptional cases which we call as "reinforced samples", which we leave as the future work.
We would like to thank the anonymous reviewers for their insightful comments. Shujian Huang is the corresponding author. This work is supported by National Science Foundation of China (No. 61672277, 61772261), the Jiangsu Provincial Research Foundation for Basic Research (No. BK20170074).
Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron Courville, and Yoshua Bengio. 2016. An actor-critic algorithm for sequence prediction. arXiv preprint arXiv:1607.07086.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. CoRR.
Yonatan Belinkov and Yonatan Bisk. 2017. Synthetic and natural noise both break neural machine translation. arXiv preprint arXiv:1711.02173.
Akshay Chaturvedi, KP Abijith, and Utpal Garain. 2019. Exploring the robustness of nmt systems to nonsensical inputs. arXiv: Learning.
Minhao Cheng, Jinfeng Yi, Huan Zhang, Pin-Yu Chen, and Cho-Jui Hsieh. 2018a. Seq2sick: Evaluating the robustness of sequence-to-sequence models with adversarial examples. arXiv preprint arXiv:1803.01128.
Yong Cheng, Lu Jiang, and Wolfgang Macherey. 2019. Robust neural machine translation with doubly adversarial inputs. arXiv preprint arXiv:1906.02443.
Yong Cheng, Zhaopeng Tu, Fandong Meng, Junjie Zhai, and Yang Liu. 2018b. Towards robust neural machine translation. arXiv preprint arXiv:1805.06130.
Javid Ebrahimi, Daniel Lowd, and Dejing Dou. 2018. On adversarial examples for character-level neural machine translation. arXiv preprint arXiv:1806.09030.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. 2014. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572.
Vladimir Karpukhin, Omer Levy, Jacob Eisenstein, and Marjan Ghazvininejad. 2019. Training on synthetic noise improves robustness to natural noise in machine translation. arXiv preprint arXiv:1902.01509.
Vijay R Konda and John N Tsitsiklis. 2000. Actor-critic algorithms. In Advances in neural information processing systems, pages 1008-1014.
Katherine Lee, Orhan First, Ashish Agarwal, Clara Fannjiang, and David Sussillo. 2018. Hallucinations in neural machine translation. NIPS 2018 Workshop IRASL.
Zhongwei Li, Xuancong Wang, Ai Ti Aw, Eng Siong Chng, and Haizhou Li. 2018. Named-entity tagging and domain adaptation for better customized translation. In Proceedings of the Seventh Named Entities Workshop, pages 41-46, Melbourne, Australia. Association for Computational Linguistics.
Minh-Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In EMNLP.
Ning Miao, Hao Zhou, Lili Mou, Rui Yan, and Lei Li. 2019. Cgmh: Constrained sentence generation by metropolis-hastings sampling. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6834-6842.
Paul Michel, Xian Li, Graham Neubig, and Juan Miguel Pino. 2019. On evaluation of adversarial perturbations for sequence-to-sequence models. arXiv preprint arXiv:1903.06620.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. 2015. Human-level control through deep reinforcement learning. Nature, 518(7540):529.
Matt Post. 2018. A call for clarity in reporting bleu scores. arXiv preprint arXiv:1804.08771.
Motoki Sano, Jun Suzuki, and Shun Kiyono. 2019. Effective adversarial regularization for neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 204-210.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. 2015. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Neural machine translation of rare words with subword units. arXiv.
Noam Shazeer and Mitchell Stern. 2018. Adafactor: Adaptive learning rates with sublinear memory cost. arXiv preprint arXiv:1804.04235.
Richard S Sutton and Andrew G Barto. 2018. Reinforcement learning: An introduction. MIT press.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS.
Petra Vidnerová and Roman Neruda. 2016. Evolutionary generation of adversarial examples for deep and shallow machine learning models. In Multidisciplinary International Social Networks Conference.
Lijun Wu, Fei Tian, Tao Qin, Jianhuang Lai, and TieYan Liu. 2018. A study of reinforcement learning for neural machine translation. arXiv preprint arXiv:1808.08866.
Chaowei Xiao, Bo Li, Jun yan Zhu, Warren He, Mingyan Liu, and Dawn Song. 2018. Generating adversarial examples with adversarial networks. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18, pages 3905-3911. International Joint Conferences on Artificial Intelligence Organization.
Yuan Zang, Chenghao Yang, Fanchao Qi, Zhiyuan Liu, Meng Zhang, Qun Liu, and Maosong Sun. 2019. Textual adversarial attack as combinatorial optimization. arXiv:1910.12196v2.
Yang Zhao, Jiajun Zhang, Zhongjun He, Chengqing Zong, and Hua Wu. 2018. Addressing troublesome words in neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 391-400.
Zhengli Zhao, Dheeru Dua, and Sameer Singh. 2017. Generating natural adversarial examples. arXiv preprint arXiv:1710.11342.
Brian D Ziebart. 2010. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. Ph.D. thesis, figshare.
A Training Details for Agent
We adopt commonly accepted translation metric BLEU as score in Eq.9. We use 50 sequence pairs per batch both in environment initialization and training of discriminator and agent. It is essential to train on batches of sequences to stabilize reinforced training. Furthermore, note that  $D$  can be too powerful during the early training stage compared to the agent's actor that it can quickly terminate an exploration. Therefore, we must train on batches and determine an overall terminal signal as aforementioned to ensure early exploration. The step  $D$  and step  $A$  are set as  $80^7$  and 120. acc_bound for discriminator training is set to 0.85. The  $a$  and  $b$  in Eq.8 are set to 0.5 and 10. The dimension of feedforward layers in the agent's actor-critic and discriminator are all 256. We initialize the embedding of both agent and discriminator by the victim's embedding.
For reinforcement learning, we adopt asynchronous learning with an additional global agent with an additional set of parameter  $\theta^{\Omega}$ , we set discount factor  $\gamma$  to 0.99,  $\alpha$  and  $\beta$  in Eq.5 to 0.5 and 0.05 respectively. As for the stop criterion, we set patience_round to 15 with convergence boundary for  $acc_{D}$  to 0.52. We adopt Adafactor(Shazeer and Stern, 2018) for training, which is a memory-efficient Adam. The learning rate for agent's optimizer is initiated as 0.001 and scheduled by rsqrt with 100 steps of warmup. The  $K$  for the candidate set is 12.
Our agent takes around 30 hours to converge on a single Nvidia 1080ti. Note that higher acc_bound and lower convergence boundary for  $D$  indicates higher semantic constraints, which will increase training time.
B Search-based Attack
Search-based adversarial generation is currently widely applied in various robustness machine translation system. We generally follow the strategy of Ebrahimi et al. (2018); Michel et al. (2019)
Algorithm 1: Reinforced training for agent
Result: A learned global agent πθΩ
1 Assume global agent as πθΩ with parameter θΩ
2 Assume agent as πθ with parameter set θ
3 initialize: Env with D, θΩ, θ;
4 while not Stop Criterion do
5	for stepD do
6	train D with current agent πθ;
7	if accD > acc_bound break;
8	end
9	test current D's accuracy accD for stop criterion;
10	for stepA do
11	initialize Env state s0;
12	synchronize πθ with πθΩ;
13	t = tstart;
14	while
15	st survive and t - tstart < tmax
16	do
17	get outtactor, Vt = πθ(st);
18	compute entropy H(outtactor);
19	sample at based on outtactor;
20	perform at and receive rt and st+1;
21	t← t+1;
22	end
23	R = {0 for terminal st}
24	V(st) for non-terminal st
25	for i ∈ {t-1,..., tstart} do
26	R ← γR + ri;
27	accumulate Lt(v(θ));
28	accumulate Lπ(t);
29	end
30	compute overall loss L(θ);
31	perform asynchronous updates on θΩ with gradient ∂L(θ)/∂θ;
32	end
which is applicable for both RNN-search and Transformer. More specifically, the  $L_{adv}$  in Eq.7 is derived as:
$$
\underset {1 \leq i \leq n, e m b _ {i} ^ {\prime} \in v o c a b} {\operatorname {a r g m a x}} | e m b ^ {\prime} - e m b _ {i} | \nabla_ {e m b _ {i}} L _ {a d v}, \tag {12}
$$
$$
L _ {a d v} (X ^ {\prime}, Y) = \sum_ {t = 1} ^ {| y |} \log (1 - P (y _ {t} | X ^ {\prime}, y _ {<   t - 1}))
$$
7Three times the average convergence episodes to train a discriminator with initial agent by the given batch size.
where each  $P(y_{t}|X)$  is calculated by Eq.1 given a corresponding target. For every source sequence, a small ratio of positions is sampled for search. Then we greedy search by the corresponding loss upon those positions with given candidates. For better comparison, we adopt the candidate set used in our model instead of naive KNN candidates. Both baseline and our model share the same UNK generation for presentation. We use homophone replacement for Chinese, and strategy by Michel et al. (2019) for English.
$^{8}$ Ebrahimi et al. (2018) suggest that greedy search is a good enough approximation.