Venue: NAACL 2021

# Addressing the Vulnerability of NMT in Input Perturbations

Weiwen Xu\({}^{1,2}\) , Ai Ti Aw\({}^{1}\) , Yang Ding\({}^{1}\), Kui Wu\({}^{1}\), Shafiq Joty\({}^{3}\)

\({}^{1}\)Institute for Infocomm Research, A*STAR

\({}^{2}\)The Chinese University of Hong Kong

\({}^{3}\)Nanyang Technological University

wwwxu@se.cuhk.edu.hk

{aaiti, ding_yang, wuk}@ia2r.a-star.edu.sg

srjoty@ntu.edu.sg

Work was done when the author was a staff in Institute for Infocomm Research, A*STAR.Corresponding Author

###### Abstract

Neural Machine Translation (NMT) has achieved significant breakthrough in performance but is known to suffer vulnerability to input perturbations. As real input noise is difficult to predict during training, robustness is a big issue for system deployment. In this paper, we improve the robustness of NMT models by reducing the effect of noisy words through a Context-Enhanced Reconstruction (CER) approach. CER trains the model to resist noise in two steps: (1) perturbation step that breaks the naturalness of input sequence with made-up words; (2) reconstruction step that defends the noise propagation by generating better and more robust contextual representation. Experimental results on Chinese-English (ZH-EN) and French-English (FR-EN) translation tasks demonstrate robustness improvement on both news and social media text. Further fine-tuning experiments on social media text show our approach can converge at a higher position and provide a better adaptation.

## 1 Introduction

Recent techniques Bahdanau et al. (2014); Wu et al. (2016); Vaswani et al. (2017) in NMT have gained remarkable improvement in translation quality. However, robust NMT that is immune to real input noise remains a big challenge for NMT researchers. Real input noises can exhibit in many forms such as spelling and grammatical errors, homophones replacement, Internet slang, new words or even a valid word used in an unfamiliar or a new context. Unlike humans who can easily comprehend and translate such texts, most NMT models are not robust to generate appropriate and meaningful translations in the presence of such noises, challenging the deployment of NMT system in real scenarios.

Noisy words have long been discussed in previous work. Aw et al. (2006) proposed the normalization approach to reduce the noise before translation. Tan et al. (2020, 2020) addressed the character-level noise directly in the NMT model. Though these approaches addressed the effect of noisy words to some extent, they are limited to spelling errors, inflectional variations, and other noises definable during training. In addition, strong external supervision like a parallel corpus of noisy text translation or dictionary containing the translation of those noisy words are hard and expensive to obtain; they are also not practical in handling real noises as noisy words can exhibit in random forms and cannot be fully anticipated during training.

Belinkov and Bisk (2018) pointed out NMT models are sensitive to small input perturbations and if this issue is not addressed, it will continue to bottleneck the translation quality. In such cases, not only the word embeddings of perturbations may cause irregularities with the local context, the contextual representation of other words may also get affected by such perturbations Liu et al. (2019). This phenomenon applies to valid words in unfamiliar context as well, which will also cause the translation to fail as illustrated in Table 1 (case 2).

\begin{table}
\begin{tabular}{c|l} \hline Input & \\ \hline Ref. & It’s super-fast to gain scores when playing games over the night. \\ \hline MT & Play the game all night and take points thief fast. \\ \hline CER & Play games all night to score points quickly. \\ \hline Input & \\ \hline Ref. & I have cut my hair, i cut off the punishment, i away \\  & the awkwardness that hurt me. \\ \hline MT & I got my punishment, gt rid of my embarrassment. \\ \hline CER & I cut short my hair , cut off punishment , and cut off my embarrassment that hurts me. \\ \hline \end{tabular}
\end{table}
Table 1: Examples of NMT’s vulnerability in translating text containing noisy words (“zei” \(\rightarrow\) “thief”, “chengfa" \(\rightarrow\) “punishment”). CER mitigates the effect of noisy words.



In this paper, we define "noisy word" as a valid or invalid word that is uncommonly used in the context or not observed frequently enough in the training data. When encoding a sentence with such a noisy word, the contextual representation of other words in the sentence are affected by the "less jointly trained" noisy word embeddings. We refer this process as "noise propagation". Noise propagation can extend to the decoder and finally distort the overall translation.

The main intuition of our proposed method is to minimize this noise propagation and reduce the irregularities in contextual representation due to these words via a _Context-Enhanced Reconstruction_ (CER) approach. To reduce the sensitivity of contextual towards noisy words in the encoder, we inject made-up words randomly to the source side of the training data to break the text naturalness. We then use a _Noise Adaptation Layer_ (NAL) to enable a more stable contextual representation by minimizing the reconstruction loss. In the decoder, we add perturbations with a semantic constraint and apply the same reconstruction loss. Unlike adversarial examples which are crafted to cause the target model to fail, our perturbation process does not have such constraint and does not rely on a target model. Our input perturbations are randomly generated, representing any types of noises that can be observed in real-world usage. This makes the perturbation process generic, easy and fast. Following Cheng et al. (2018), we generate semantically related perturbations in the decoder to increase the diversity of the translations.

Together with NAL, our model shows its ability to resist noises in the input and produce more robust translations. Results on ZH-EN and FR-EN translation significantly improve over the baseline by +1.24 (MT03) and +1.4 (N15) BLEU on news domain, and +1.63 (_Social_), +1.3 (_nmt18_) on social media domain respectively. Further fine-tuning experiments on FR-EN social media text even witness an average improvement of +1.25 BLEU over the best approach.

## 2 Related Work

Robust Training:Robust training has shown to be effective to improve the robustness of the models in computer vision Szegedy et al. (2013). In Natural Language Processing, it involves augmenting the training data with carefully crafted noisy examples: semantically equivalent word substitutions Alzantot et al. (2018), paraphrasing Iyyer et al. (2018); Ribeiro et al. (2018), character-level noise Ebrahimi et al. (2018); Tan et al. (2020), b), or perturbations at embedding space Miyato et al. (2016); Liang et al. (2020). Inspired by Lei et al. (2017) that nicely captures the semantic interactions in discourse relation, we regard noise as a disruptor to break semantic interactions and propose our CER approach to mitigate this phenomenon. We make up "noisy" words randomly to act as random noise in the input to break the text naturalness. Our experiment demonstrates its superiority in multiple dimensions.

Robust Neural Machine Translation:Methods have been proposed to make NMT models resilient not only to adequacy errors Lei et al. (2019) but also to both natural and synthetic noise. Incorporating monolingual data into NMT has the capacity to improve the robustness Sennrich et al. (2016); Edunov et al. (2018); Cheng et al. (2016). Some non data-driven approaches that specifically designed to address the robustness problem of NMT Sperber et al. (2017); Ebrahimi et al. (2018); Wang et al. (2018); Karpukhin et al. (2019); Cheng et al. (2019, 2020) explored effective ways to synthesize adversarial examples into the training data. Belinkov and Bisk (2018) showed a structure-invariant word representation capable of addressing multiple typo noise. Cheng et al. (2018) used adversarial stability training strategy to make NMT resilient to arbitrary noise. Liu et al. (2019) added an additional phonetic embedding to overcome homophone noise.

Meanwhile Michel and Neubig (2018) released a dataset for evaluating NMT on social media text. This dataset was used as a benchmark for WMT 19 Robustness shared task Li et al. (2019) to improve the robustness of NMT models on noisy text. We show our approach also benefits the fine-tuning process using additional social media data.

## 3 Approaches

We propose a Context-Enhanced Reconstruction (CER) approach to learn robust contextual representation in the presence of noisy words through a perturbation step and a reconstruction step in both encoder and decoder during model training. Figure 1 shows the architecture.

The perturbation step automatically inserts made-up words in the input sequence x to generate a noisy example x\({}^{\prime}\). The noisy example mimics input where text natural 

the noisy words. Similarly, we perturb the output sequence \({\bf y}\) to \({\bf y^{\prime}}\) using a semantic constraint to generate noisy examples for the decoder to have more diversity in the translations.

The reconstruction step in the model aims to restore the contextual representation \({\bf c}^{\mathbf{x}^{\prime}}\) of \({\bf x}^{\prime}\) to be similar to its corresponding original contextual representation \({\bf c}^{\mathbf{x}}\) in the encoder. Specifically, under the Transformer architecture (Figure 1), the reconstruction step aims to stabilize and minimize the disruption of attention distribution for a word over the whole input in the presence of inserted noise. The stabilization is needed for both clean and noisy words as both of their contextual representations are affected. For a noisy word, reconstruction reduces the attention to itself and encourages the construction of the contextual representation to leverage more on its clean neighbors. For clean words, reconstruction works as a denoise module to mitigate the interference of noisy words. For \({\bf c}^{\mathbf{y}^{\prime}}\) in the decoder, the aim is to generate more examples with similar context as \({\bf c}^{\mathbf{y}}\). The reconstruction helps to normalize the contextual representation of semantically similar words.

### Perturbing Input Text with Noise

We insert made-up words, representing any kinds of noise, to disturb the contextual representation during training. To create those words, we build a made-up dictionary \(\mathcal{D}_{x}^{-}\) with \(M\) made-up words. As shown in Figure 1(a), made-up words are simply indexed slots in \(\mathcal{D}_{x}^{-}\), whose embeddings are randomly initialized with no prior restriction and updated during training just as valid words. During the perturbation step, we randomly select multiple positions in each input sequence based on probability \(\sigma_{x}\) and replace the words with any arbitrary made-up words in \(\mathcal{D}_{x}^{-}\).

For the decoder, as the aim is not to insert noise but to increase the diversity of translation, we add small perturbations with a semantic constraint to make the model robust. Specifically, we randomly select multiple positions in each target sequence with a probability \(\sigma_{y}\) and perturb the corresponding words. For the word \(y_{i}\) chosen to be perturbed, we create a dynamic set \(\mathcal{V}_{y_{i}}\) consisting of \(m\) words having the highest cosine similarity with it (excluding \(y_{i}\)). We average the embeddings of the words in \(\mathcal{V}_{y_{i}}\) as the perturbation for \(y_{i}\).

\[\mathcal{V}_{y_{i}}=\underset{y_{j}\in\mathcal{D}_{y_{j}}\neq i}{top\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\! 

respectively. We train the NAL by:

\[{\mathcal{L}}^{x}_{nal}(\mathbf{\theta}^{x}_{nal})=\frac{1}{|S|}\sum_{( \mathbf{x},\mathbf{y})\in S}\sum_{l=1}^{N}||\mathbf{c}^{x}_{l}-\mathrm{NAL}( \mathbf{c}^{x^{\prime}}_{l})||^{2} \tag{3}\]

Where \(\mathbf{\theta}^{x}_{nal}\) are parameters of NAL, \(S\) is the training corpus and \(N\) is the encoder layer size. Given \(\mathbf{c}^{x^{\prime}}\), NAL attempts to output a more correct contextual representation guided by \(\mathbf{c}^{\mathbf{x}}\). We use a single layer feed-forward network (FFN) in Vaswani et al. (2017) as our NAL implementation. Similarly, the reconstruction loss for decoder is:

\[{\mathcal{L}}^{y}_{nal}(\mathbf{\theta}^{y}_{nal})=\frac{1}{|S|}\sum_{(\mathbf{ x},\mathbf{y})\in S}\sum_{l=1}^{N}||\mathbf{c}^{y}_{l}-\mathrm{NAL}( \mathbf{c}^{y^{\prime}}_{l})||^{2} \tag{4}\]

### Model Training

We apply the perturbation step at the embedding layer, see Figure 1. The inserted noise in \(\mathbf{x}^{\prime}\) and \(\mathbf{y}^{\prime}\) would also receive gradient from the final loss function and update just like other clean words. NAL is added at each Transformer layer where the outputs are only used to calculate the reconstruction loss and not passed to the next layer. On the other hand, the output of FFN is propagated to the next layer as usual. The reconstruction step mainly serves as a stabilizer to prevent the noise from propagating.

The final training objective \(\mathcal{L}\) is the combination of the above three loss functions, the original translation loss, the reconstruction loss for the encoder and the reconstruction loss for the decoder. Both \(\lambda_{x}\) and \(\lambda_{y}\) are set empirically to count for the relative importance.

\[\small{\mathcal{L}}={\mathcal{L}}_{nmt}(\mathbf{\theta}_{nmt})+ \lambda_{x}{\mathcal{L}}^{x}_{nal}(\mathbf{\theta}^{x}_{nal})+\lambda_{y}{ \mathcal{L}}^{y}_{nal}(\mathbf{\theta}^{y}_{nal}) \tag{5}\]

## 4 Experiment Settings

Experiments are conducted on ZH-EN and FR-EN translation tasks for both news and social media domains. We also use social media text to fine-tune the NMT systems on FR-EN.

### Data

**ZH-EN:** The training data consists of 1.25M sentence pairs extracted from LDC. For news domain, we use NIST MT02 as the development set and select the best model to test MT03, MT04, MT05, MT06 and MT08 news test sets. For social media domain, we create a test set (_Social_) consisting of 2000 sentences with three human annotated references. The source sentences are collected from public social media platforms in four Chinese-speaking regions: Mainland China, Hong Kong, Taiwan and Singapore 1.

Footnote 1: Available at [https://github.com/wwxu21/CER-MT](https://github.com/wwxu21/CER-MT).

**FR-EN:** We use the same datasets as Michel and Neubig (2018). The training set consists of 2.16M sentence pairs extracted from _europarl-v7_ and _news-commentary-v10_. We use the _newdiscussdev2015_ as development set and evaluate the model on two news test sets, _newstest2014_ (N14) and _newsdiscussetsel2015_ (N15). We also evaluate on two social media test sets: _mtnt18_ (Michel and Neubig, 2018) and _mtnl9_ (Li et al., 2019).

**FR-EN Fine-Tuning:** We use the noisy training set (_mmttrain_) provided by Michel and Neubig (2018) to fine-tune the FR-EN model.

We use fairseq's implementation of Transformer Ott et al. (2019). In evaluation, we report case-insensitive tokenized BLEU for ZH-EN Papineni et al. (2002) and sacre-BLEU Post (2018) for FR-EN. Following Michel and Neubig (2018), we do not use development set but only report best results on three social media test sets.

We segment the Chinese words using THUPAC (Li and Sun, 2009) and tokenize both French and English words using tokenizer.perl2. We apply BPE Sennrich et al. (2016b) to get sub-word vocabularies for the encoder and decoder, both with 20K merge operations.

Footnote 2: [https://github.com/moses-smt/mosesdecoder](https://github.com/moses-smt/mosesdecoder)

The hyper-parameters setting is the same as transformer-base in Vaswani et al. (2017) except that we set dropout rate as 0.4 in all our experiments. Our proposed models are trained on top of Transformer baseline for efficiency purpose, where additional parameters from the embeddings of \(\mathcal{D}^{-}_{x}\) and ReL are uniformly initialized. The makeup dictionary size \(M\) is set to 10,000. The size of dynamic set \(m\) is set to 3. The probability \(\sigma_{x}\) and \(\sigma_{y}\) are both set to 0.1 and balance coefficient \(\lambda_{x}\) and \(\lambda_{y}\) are both set to 1.

### Baseline Models

We use Transformer as our baseline.

**ZH-EN**: We compare with Wang et al. (2018); Cheng et al. (2018, 2019). Wang et al. (2018) use a data augmentation approach by randomly replacing words in source and target sentences with other in-dictionary words. Cheng et al. (2018) use adversarial stability training to make NMT resilient to noise. Cheng et al. (2019) employ a white-box approach to synthesize adversarial examples.

 



**FR-EN**: In addition to Wang et al. (2018), we compare with Michel and Neubig (2018); Zhou et al. (2019); Vaibhav et al. (2019) on FR-EN or FR-EN Fine-Tuning tasks. Michel and Neubig (2018) do the first benchmark of the noisy text translation tasks in three languages. Vaibhav et al. (2019) leverage effective synthetic noise to make NMT resilient to noisy text. We implement their approach on Transformer backbone. For a fair comparison, we limit the data to train back-translation models only with _mtnttrain_. Zhou et al. (2019) adopt a multitask transformer architecture with two decoders, where the first decoder learns to denoise and the second decoder learns to translate from the denoised text. They adopt the approach proposed by Vaibhav et al. (2019) to synthesize the noisy text for their first decoder.

We do not compare our model with Berard et al. (2019); Helcl et al. (2019) as they use much more out-domain data, a great number of monolingual data and a bigger Transformer model, and hence not comparable with our experimental settings.

## 5 Results and Analysis

### Comparison with Baseline Models

Table 2 and Table 3 show the performance on ZH-EN and FR-EN tasks. We show the results of applying CER only to the encoder (+ CER-Enc), and to both the encoder and decoder (+ CER).

As illustrated, our approach improves the news text translations on all test sets for both ZH-EN and FR-EN and outperforms the Transformer baseline in terms of average BLEU by +1.01 and +1.2 on ZH-EN and FR-EN respectively, illustrating the superiority of our approach.

The performance on social media test sets shows significant improvement with up to +1.63 BLEU over Transformer and +0.84 BLEU over the best approach Wang et al. (2018) on ZH-EN. For FR-EN, our model outperforms Wang et al. (2018) by +1.5 and +1.0 BLEU on _mtnt18_ and _mtnt19_ respectively. Zhou et al. (2019) use _mtnttrain_ and TED Qi et al. (2018) to synthesize noisy sentences for their first decoder, hence effectively they are exploiting in-domain data during training and thus not quite a fair comparison in the evaluation. Nevertheless, CER still significantly outperforms Zhou et al. (2019) by +2.0 BLEU on _mtnt18_.

### Effect of Noise

We investigate the effect of different noise-insertion methods by dynamically inserting noise into the source side of the original training set using different strategies with a same probability \(\sigma_{x}\).

_Madeup_: Our approach to add made-up words.

_Semantics_: We test our semantic constraint in the decoder to assess if it benefits the encoder.

\begin{table}
\begin{tabular}{l||c|c c c c c c|c} \hline Model & MT02 (DEV) & MT03 & MT04 & MT05 & MT06 & MT08 & News Ave. & _Social_ \\ \hline  & \multicolumn{7}{c}{_Existing systems_} & & & & & \\ \hline Wang et al. (2018) & 47.13 & 46.68 & 47.41 & 46.66 & 46.62 & 38.46 & 45.17 & 23.20 \\ Cheng et al. (2018) & 46.10 & 44.07 & 45.61 & 43.45 & 44.44 & 34.94 & 42.50 & 21.27 \\ Cheng et al. (2019) & 47.06 & 46.48 & 47.39 & 46.58 & 46.95 & 37.38 & 44.96 & 22.74 \\ \hline  & \multicolumn{7}{c}{_Our systems_} & & & & & \\ \hline Transformer & 46.98 & 46.35 & 47.27 & 46.35 & 46.77 & 38.20 & 45.00 & 22.41 \\ + CER-Enc & 47.65 & 46.72 & 47.53 & 47.06 & 47.04 & 38.53 & 45.38 & 23.81 \\ + CER & **48.34** & **47.59** & **48.21** & **47.29** & **47.64** & **39.33** & **46.01** & **24.04** \\ \hline \end{tabular}
\end{table}
Table 2: Case-insensitive BLEU scores (%) on ZH-EN translation. MT02 is our development set.

Figure 2: BLEU improvements compared to Transformer baseline shown in Table 2 and Table 3 when applying noise-insertion methods.



_Dropout_: We replace word embeddings with all-0 vectors, similar to enlarging the dropout rate.

_Gaussian_: Following the feature-level perturbations of Cheng et al. (2018), we add the Gaussian noise to a word embedding to simulate the noise.

_Random_: We replace a word with an arbitrary word in the dictionary. This would result in a valid word being placed in an unreasonable context.

Figure 2 shows the BLEU improvement of various noise-insertion methods on social media test sets. We find that nearly all kinds of noise-insertion methods improve the robustness of MT with the exception of _Dropout_. Since we have already set the dropout rate to an optimal rate, inserting additional _Dropout_ noise does not increase but decreases the performance. As shown, _Madeup_ improves the performance nearly twice than the rest of the noise-insertion methods. We conjecture _Semantics_, _Dropout_ and _Gaussian_ may be small and not diverse enough to simulate the real noisy words. Both _Random_ and _Madeup_ can break the text coherence. However, _Random_ uses a random in-dictionary word, which can place a valid word in an unreasonable context and cause its embedding to update in a wrong direction. In fact, this method improves the robustness of NMT models at the cost of those replaced words. Our _Madeup_ can entirely avoid this cost as we use made-up words to work as noisy words and does not cause any context change of all in-dictionary words.

### Effect of NAL

To further gain insights on how NAL helps improve the robustness of NMT models. We create three variants to aid our analysis:

**CER-inactive**: We do not activate NAL at testing time. The contextual representation is feed directly into later FFN. This variant is to test the effectiveness of NAL.

**CER-con**: We remove NAL but only add a constraint to ensure \(\{\mathbf{c^{x}},\mathbf{c^{x^{y}}}\}\) and \(\{\mathbf{c^{y}},\mathbf{c^{y^{\prime}}}\}\) to be close respectively at training time. This forces the self-attention layer to reconstruct the correct contextual representation itself. This variant is to demonstrate the necessity to set apart the context generation module (self-attention layer) and the reconstruction module (NAL).

**CER-D**: We borrow the adversarial stability training strategy proposed in Cheng et al. (2018) here. In this variant, NAL is replaced by a discriminator and \(\mathbf{\theta}_{nal}^{x}\) and \(\mathbf{\theta}_{nal}^{y}\) are changed to the adversarial learning loss in Cheng et al. (2018). The purpose is to assess the effectiveness of NAL and the discriminator in context reconstruction.

Figure 3 shows the results of the three variants on three social media test sets. From the figure, we make the following observations.

_NAL is effective at Test Time._ The activation of NAL at test time helps to produce more reliable contextual representation. Notably, NAL gains +1.19 BLEU on _Social_.

_NAL needs to be learnt separately._ As shown in CER-con, by forcing self-attention layer to do both tasks (context generation and reconstruction), the performance improvement gets affected by at least 0.4 BLEU.

_NAL is more effective than a discriminator to guide reconstruction._ The improvements are less significant in all test sets when using a discriminator (CER-D) comparing to CER. Therefore, we can conclude that NAL is more effective than a discriminator to reconstruct the perturbed contextual representation and CER outperforms all variants.

### FR-EN Fine-Tuning on Social Media Text

We fine-tune the same Transformer model in Table 3 with the social media data _mtnttrain_ (+_FT_) and further include CER in the fine-tuning (+FT w/ CER). Table 4 shows our performance (+_FT w/ CER_) with other four fine-tuning approaches on _mtnttrain_. It shows that our CER also bene

\begin{table}
\begin{tabular}{l||r r} \hline Model & mntnt18 & mtnt19 \\ \hline \multicolumn{5}{|c|}{_Existing systems_} \\ \hline Michel and Neubig & 30.3 & N/A \\ Wang et al. & 35.1 & 36.7 \\ Zhou et al. & 31.7 & 32.8 \\ Vaibhav et al. & 36.0 & 37.5 \\ \hline \multicolumn{3}{|c|}{_Our systems_} \\ \hline Transformer (Base) & 25.2 & 28.1 \\ +FT & 35.2 & 37.4 \\ +FT w/ CER & **37.3** & **38.7** \\ \hline \end{tabular}
\end{table}
Table 4: sacreBLEU on FR-EN fine-tuning task.

Figure 3: BLEU scores of CER variants.

 

fits the fine-tuning process and outperforms all the approaches in two noisy test sets. Specifically, it gains +2.1 and +1.3 BLEU over _+FT_ on _mtnt18_ and _mtnt19_ and outperforms Vaibhav et al. (2019) by +1.3 and +1.2 BLEU respectively.

### Experiments on Large-Scale Datasets

We first train a ZH-EN baseline model using 25M sentence pairs, which are mainly in news domain. Similar to the setting in Table 4, we apply both simple finetuning (+FT) and our CER (+ FT w/ CER) approach using 125K social media training data. We evaluate those models on _Social_. We also include the performance of Google Translate 3 here to show the competitiveness of our baseline model.

Footnote 3: [https://translate.google.com/](https://translate.google.com/)

As shown in Table 5, our CER approach can still benefit the fine-tuning process even on the strong baseline. It should be noted that the baseline has already maintained high robustness with large-scale training data where improvement in such a model is hard to obtain. In fact, 125K in-domain data can only contribute to 1.55 BLEU improvement. Under this circumstance, the 0.26 BLEU improvement brought by CER should be highly valued considered no additional fine-tuning data is used.

## 6 Conclusions

In this work, we propose an approach to reduce the vulnerability of NMT models to input perturbations. Our input perturbation is easy, fast and not specific to a target victim model. Experimental results show our proposed approach improves the robustness on both news and social media text and helped to improve the translation of real input.

## 7 Acknowledgments

The work was supported in part by Defence Science and Technology Agency (DSTA), Singapore.

## References

* M. Alzantot, Y. Sharma, A. Elgohary, B. Ho, M. Srivastava, and K. Chang (2018)Generating natural language adversarial examples. arXiv preprint arXiv:1804.07998. Cited by: SS1.
* A. Asw, M. Zhang, J. Xiao, and J. Su (2006)A phrase-based statistical model for SMS text normalization. In Proceedings of the COLING/ACL 2006 Main Conference Poster Sessions, Sydney, Australia, pp. 33-40. External Links: Link Cited by: SS1.
* D. Bahdanau, K. Cho, and Y. Bengio (2014)Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473. Cited by: SS1.
* Y. Belinkov and Y. Bisk (2018)Synthetic and natural noise both break neural machine translation. In International Conference on Learning Representations, Cited by: SS1.
* A. Berard, I. Calapodescu, and C. Roux (2019)Naver labs europe's systems for the wmt19 machine translation robustness task. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), Florence, Italy, pp. 526-532. External Links: Link Cited by: SS1.
* Y. Cheng, L. Jiang, W. Macherey, and J. Eisenstein (2020)AdvAug: robust adversarial augmentation for neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 4324-4333. External Links: Link Cited by: SS1.
* Y. Cheng, H. Jiang, W. Macherey, and J. Eisenstein (2020)AdvAug: robust adversarial augmentation for neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 4324-4333. External Links: Link Cited by: SS1.
* Y. Cheng, W. Xu, Z. He, W. He, H. Wu, M. Sun, and Y. Liu (2016)Semisupervised learning for neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Berlin, Germany, pp. 1965-1974. External Links: Link Cited by: SS1.
* J. Ebrahimi, D. Lowd, and D. Dou (2018)On adversarial examples for character-level neural

\begin{table}
\begin{tabular}{l|c||c} \hline Model & \multicolumn{2}{c}{_Social_} \\ \hline Google Translate & 38.59 \\ \hline \hline  & Baseline & 39.01 \\ Ours & +FT & 40.56 (+3.97\%) \\  & +FT w/ CER & **40.82 (+4.64\%)** \\ \hline \end{tabular}
\end{table}
Table 5: Case-insensitive BLEU scores (relative improvement) on large-scale ZH-EN translation system.

 

machine translation. In Proceedings of the 27th International Conference on Computational Linguistics, Santa Fe, New Mexico, USA, pp. 653-663. External Links: ISBN 978140503233 Cited by: SS1.
* J. Ebrahimi, A. Rao, D. Lowd, and D. Dou (2018)HotFlip: white-box adversarial examples for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Melbourne, Australia, pp. 31-36. External Links: Link, Document Cited by: SS1.
* S. Edunov, M. Ott, M. Auli, and D. Grangier (2018)Understanding back-translation at scale. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, pp. 489-500. External Links: Link, Document Cited by: SS1.
* J. Helcl, J. Libovick, and M. Popel (2019)Cuni system for the wmt19 robustness task. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), Florence, Italy, pp. 539-543. External Links: Link, Document Cited by: SS1.
* M. Iyyer, J. Wieting, K. Gimpel, and L. Zettlemoyer (2018)Adversarial example generation with syntactically controlled paraphrase networks. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), Vol. , Long Beach, London, pp. 1875-1885. External Links: Link, Document Cited by: SS1.
* V. Karpukhin, O. Levy, J. Eisenstein, and M. Ghazvininejad (2019)Training on synthetic noise improves robustness to natural noise in machine translation. arXiv preprint arXiv:1902.01509. Cited by: SS1.
* W. Lei, X. Wang, M. Liu, I. Ilievski, X. He, and M. Kan (2017)Swim: a simple word interaction model for implicit discourse relation recognition. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17, Hong Kong, China, pp. 4026-4032. External Links: Link, Document Cited by: SS1.
* a supervised approach. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, pp. 942-952. External Links: Link, Document Cited by: SS1.
* X. Li, P. Michel, A. Anastasopoulos, Y. Belinkov, N. Durrani, O. Firat, P. Koehn, G. Neubig, J. Pino, and H. Sajjad (2019)Findings of the first shared task on machine translation robustness. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), Florence, Italy, pp. 91-102. External Links: Link, Document Cited by: SS1.
* Z. Li and M. Sun (2009)Punctuation as implicit annotations for Chinese word segmentation. Computational Linguistics35 (4), pp. 505-512. External Links: ISSN 0021-909X, Link, Document Cited by: SS1.
* H. Liang, W. Lei, P. Yaozhu Chan, Z. Yang, M. Sun, and T. Chua (2020)Pihrdy: learning pitch-, rhythm-, and dynamics-aware embeddings for symbolic music. In MM '20: The 28th ACM International Conference on Multimedia, Virtual Event/ Seattle, WA, USA, October 12-16, 2020, pp. 574-582. External Links: Link, Document Cited by: SS1.
* H. Liu, M. Ma, L. Huang, H. Xiong, and Z. He (2019)Robust neural machine translation with joint textual and phonetic embedding. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, pp. 3044-3049. External Links: Link, Document Cited by: SS1.
* P. Michel and G. Neubig (2018)MTNT: a testbed for machine translation of noisy text. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, pp. 543-553. External Links: Link, Document Cited by: SS1.
* T. Miyato, A. M. Dai, and I. Goodfellow (2016)Adversarial training methods for semi-supervised text classification. arXiv preprint arXiv:1605.07725. Cited by: SS1.
* M. Ott, S. Edunov, A. Baevski, A. Fan, S. Gross, N. Ng, D. Grangier, and M. Auli (2019)fairseq: a fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations, Cited by: SS1.
* K. Papineni, S. Roukos, T. Ward, and W. Zhu (2002)Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pp. 311-318. External Links: Link, Document Cited by: SS1.
* M. Post (2018)A call for clarity in reporting bleu scores. arXiv preprint arXiv:1804.08771. Cited by: SS1.
* Y. Qi, D. Sachan, M. Felix, S. Padmanabhan, and G. Neubig (2018)When and why are pre-trained word embeddings useful for neural machine translation?. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), New Orleans, Louisiana, pp. 529-535. External Links: Link, Document Cited by: SS1.
* M. Tulio Ribeiro, S. Singh, and C. Guestrin (2018)Semantically equivalent adversarial rules for debugging NLP models. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 

pages 856-865, Melbourne, Australia. Association for Computational Linguistics.
* Sennrich et al. (2016a) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Improving neural machine translation models with monolingual data. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 86-96, Berlin, Germany. Association for Computational Linguistics.
* Sennrich et al. (2016b) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural machine translation of rare words with subword units. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1715-1725, Berlin, Germany. Association for Computational Linguistics.
* Sperber et al. (2017) Matthias Sperber, Jan Niehues, and Alex Waibel. 2017. Toward robust neural machine translation for noisy input sequences. In _International Workshop on Spoken Language Translation (IWSLT)_.
* Szegedy et al. (2013) Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. 2013. Intriguing properties of neural networks. _arXiv preprint arXiv:1312.6199_.
* Tan et al. (2020a) Samson Tan, Shafiq Joty, Min-Yen Kan, and Richard Socher. 2020a. It's morphin' time! Combating linguistic discrimination with inflectional perturbations. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 2920-2935, Online. Association for Computational Linguistics.
* Tan et al. (2020b) Samson Tan, Shafiq Joty, Lav Varshney, and Min-Yen Kan. 2020b. Mind your inflections! Improving NLP for non-standard Englishishes with Base-Inflection Encoding. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 5647-5663, Online. Association for Computational Linguistics.
* Vaibhav Saibhav et al. (2019) Vaibhav Vaibhav, Sumeet Singh, Craig Stewart, and Graham Neubig. 2019. Improving robustness of machine translation with synthetic noise. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 1916-1920, Minneapolis, Minnesota. Association for Computational Linguistics.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, _Advances in Neural Information Processing Systems 30_, pages 5998-6008. Curran Associates, Inc.
* Wang et al. (2018) Xinyi Wang, Hieu Pham, Zhihang Dai, and Graham Neubig. 2018. SwitchOut: an efficient data augmentation algorithm for neural machine translation. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 856-861, Brussels, Belgium. Association for Computational Linguistics.
* Wu et al. (2016) Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. _arXiv preprint arXiv:1609.08144_.
* Zhou et al. (2019) Shuyan Zhou, Xiangkai Zeng, Yingqi Zhou, Antonios Anastasopoulos, and Graham Neubig. 2019. Improving robustness of neural machine translation with multi-task learning. In _Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)_, pages 565-571, Florence, Italy. Association for Computational Linguistics.