Title: SYNTHETIC AND NATURAL NOISE BOTH BREAK NEURAL MACHINE TRANSLATION

Abstract: Character-based neural machine translation (NMT) models alleviate out-ofvocabulary issues, learn morphology, and move us closer to completely end-toend translation systems. Unfortunately, they are also very brittle and easily falter when presented with noisy data. In this paper, we confront NMT models with synthetic and natural sources of noise. We find that state-of-the-art models fail to translate even moderately noisy texts that humans have no trouble comprehending. We explore two approaches to increase model robustness: structure-invariant word representations and robust training on noisy texts. We find that a model based on a character convolutional neural network is able to simultaneously learn representations robust to multiple kinds of noise. * Equal contribution. Ordering determined by bartender's coin: https://youtu.be/BFSc2HnpYtA 1 One caveat we feel

1. INTRODUCTION:

Humans have surprisingly robust language processing systems that can easily overcome typos, misspellings, and the complete omission of letters when reading (Rawlinson, 1976) . A particularly extreme and comical exploitation of our robustness came years ago in the form of a popular meme: "Aoccdrnig to a rscheearch at Cmabrigde Uinervtisy, it deosn't mttaer in waht oredr the ltteers in a wrod are, the olny iprmoetnt tihng is taht the frist and lsat ltteer be at the rghit pclae." A person's ability to read this text comes as no surprise to the psychology literature. Saberi & Perrott (1999) found that this robustness extends to audio as well. They experimented with playing parts of audio transcripts backwards and found that it did not affect comprehension. Rayner et al. (2006) found that in noisier settings reading comprehension only slowed by 11%. McCusker et al. (1981) found that the common case of swapping letters could often go unnoticed by the reader. The exact mechanisms and limitations of our understanding system are unknown. There is some evidence that we rely on word shape (Mayall et al., 1997) , that we can switch between whole word recognition and piecing together words from letters (Reicher, 1969; Pelli et al., 2003) , and there appears to be no evidence that the first and last letter positions are required to stay constant for comprehension. 1 In stark contrast, neural machine translation (NMT) systems, despite their pervasive use, are immensely brittle. For instance, Google Translate produces the following unintelligible translation for a German version of the above meme: 2 "After being stubbornly defiant, it is clear to kenie Rlloe in which Reiehnfogle is advancing the boulders in a Wrot that is integral to Sahce, as the utterance and the lukewarm boorstbaen stmimt."

While typos and noise are not new to NLP, our systems are rarely trained to explicitly address them, as we instead hope that the relevant noise will occur in the training data. Despite these weaknesses, the move to character-based NMT is important. It helps us tackle the long tailed distribution of out-of-vocabulary words in natural language, as well as reduce computation load of dealing with large word embedding matrices. NMT models based on characters and other sub-word units are able to extract stem and morphological information to generalize to unseen words and conjugations. They perform very well in practice on a range of languages (Sennrich et al., 2016a; Wu et al., 2016) . In many cases, these models actually discover an impressive amount of morphological information about a language (Belinkov et al., 2017a) . Unfortunately, training (and testing) on clean data makes models brittle and, arguably, unfit for broad deployment.

Figure 1 shows how the performance of two state-of-the-art NMT systems degrades when translating German to English as a function of the percent of German words modified. Here we show three types of noise: 1) Random permutation of the word, 2) Swapping a pair of adjacent letters, and 3) Natural human errors. We discuss these types of noise and others in depth in section 4.2. The important thing to note is that even small amounts of noise lead to substantial drops in performance. To address these trends and investigate the effects of noise on NMT, we explore two simple strategies for increasing model robustness: using structure-invariant representations and robust training on noisy data, a form of adversarial training (Szegedy et al., 2014; Goodfellow et al., 2015) . We find that a character CNN representation trained on an ensemble of noise types is robust to all kinds of noise. We shed some light on the model ability to learn robust representations to multiple types of noise, and point to remaining difficulties in handling natural noise. Our goal is two fold: 1) initiate a conversation on robust training and modeling techniques in NMT, and 2) promote the creation of better and more linguistically accurate artificial noise to be applied to new languages and tasks.

2. ADVERSARIAL EXAMPLES:

The growing literature on adversarial examples has demonstrated how dangerous it can be to use brittle machine learning systems so pervasively in the real world (Biggio et al., 2012; Szegedy et al., 2014; Goodfellow et al., 2015; Mei & Zhu, 2015) . Small changes to the input can lead to dramatic failures of deep learning models (Szegedy et al., 2014; Goodfellow et al., 2015) . In the machine vision field, changes to the input image that are indistinguishable by humans can lead to misclassification. This leads to potential for malicious attacks using adversarial examples. An important distinction is often drawn between white-box attacks, where adversarial examples are generated with access to the model parameters, and black-box attacks, where examples are generated without such access (Papernot et al., 2016a; 2017; Narodytska & Kasiviswanathan, 2017; Liu et al., 2017) .

While more common in the vision domain, recent work has started exploring adversarial examples for NLP. A few white-box attacks have employed the fast gradient sign method (Goodfellow et al., 2015) or other techniques to find important text edit operations (Papernot et al., 2016b; Samanta & Mehta, 2017; Liang et al., 2017; Ebrahimi et al., 2017) . Others have considered black-box adversarial examples for text classification (Gao et al., 2018) or NLP evaluation (Jia & Liang, 2017) . Heigold et al. (2017) evaluated character-based models on several types of noise in morphological tagging and MT, and observed similar trends to our findings. Finally, Sakaguchi et al. (2017) designed a character-level recurrent neural network that can better handle the particular kind of noise present in the meme mentioned above by modeling spelling correction. Here we devise simple methods for generating adversarial examples for NMT. We do not assume any access to the NMT models' gradients, instead relying on synthetic and naturally occurring language errors to generate noise.

The other side of the coin is to improve models' robustness to adversarial examples (Globerson & Roweis, 2006; Cretu et al., 2008; Rubinstein et al., 2009; Chan et al., 2017) . Adversarial trainingincluding adversarial examples in the training data -can improve a model's ability to cope with such examples at test time (Szegedy et al., 2014; Goodfellow et al., 2015) . This kind of defense is sensitive to the type of adversarial examples seen in training, but can be made more robust by ensemble adversarial training -training on examples transfered from multiple pre-trained models (Tramèr et al., 2017) . We explore ensemble training by combining multiple types of noise at training time, and observe similar increased robustness in the machine translation scenario.

Training on and for adversarial noise is an important extension of earlier work on creating robustness in neural networks by incorporating noise to a network's representations, data, or gradients. Training with noise can provide a form of regularization (Bishop, 1995) and ensure the model is exposed to samples outside the training distribution (Matsuoka, 1992) .

3. MT SYSTEMS:

The rise of end-to-end models in neural machine translation has led to recent interest in understanding how these models operate. Several studies investigated the ability of such models to learn linguistic properties at morphological (Vylomova et al., 2016; Belinkov et al., 2017a; Dalvi et al., 2017) , syntactic (Shi et al., 2016; Sennrich, 2017) , and semantic levels (Belinkov et al., 2017b) . The use of characters or other sub-word units emerges as an important component in these models.

Our work complements previous studies by presenting such NMT systems with noisy examples and exploring methods for increasing their robustness.

We experiment with three different NMT systems with access to character information at different levels. First, we use the fully character-level model of Lee et al. (2017) . This is a sequence-tosequence model with attention (Sutskever et al., 2014; Bahdanau et al., 2014) that is trained on characters to characters (char2char). It has a complex encoder with convolutional, highway, and recurrent layers, and a standard recurrent decoder. See Lee et al. (2017) for architecture details. This model was shown to have excellent performance on the German→English and Czech→English language pairs. We use the pre-trained German/Czech→English models.

Second, we use Nematus (Sennrich et al., 2017) , a popular NMT toolkit that was used in topperforming contributions in shared MT tasks in WMT (Sennrich et al., 2016b) and IWSLT (Junczys-Dowmunt & Birch, 2016) . It is another sequence-to-sequence model with several architecture modifications, especially operating on sub-word units using byte-pair encoding (BPE) (Sennrich et al., 2016a) . We experimented with both their single best and ensemble BPE models, but saw no significant difference in their performance under noise, so we report results with their single best WMT models for German/Czech→English.

Finally, we train an attentional sequence-to-sequence model with a word representation based on a character convolutional neural network (charCNN). This model retains the notion of a word but learns a character-dependent representation of words. It was shown to perform well on morphologically-rich languages (Kim et al., 2015; Belinkov & Glass, 2016; Costa-jussà & Fonollosa, 2016; Sajjad et al., 2017) , thanks to its ability to learn morphologically-informative representations (Belinkov et al., 2017a) . The charCNN model has two long short-term memory (Hochreiter & Schmidhuber, 1997) layers in the encoder and decoder. A CNN over characters in each word replaces the word embeddings on the encoder side (for simplicity, the decoder is word-based). We use 1000 filters with a width of 6 characters. The character embedding size is set to 25. The convolutions are followed by Tanh and max-pooling over the length of the word (Kim et al., 2015) . We train charCNN with the implementation in Kim (2016) ; all other settings are kept to default values.

4. DATA:



4.1. MT DATA:

We use the TED talks parallel corpus prepared for IWSLT 2016 (Cettolo et al., 2012) for testing all of the NMT systems, as well as for training the charCNN models. We follow the official training/development/test splits. All texts are tokenized with the Moses tokenizer. Table 1 summarizes statistics on the TED talks corpus. Since we do not have access to a parallel corpus with natural noise, we instead harvest naturally occurring errors (typos, misspellings, etc.) from available corpora of edits to build a look-up table of possible lexical replacements. In this work, we restrict ourselves to single word replacements, but several of the corpora below also provide access to phrase replacements.

French Max & Wisniewski (2010) collected Wikipedia edit histories to form the Wikipedia Correction and Paraphrase Corpus (WiCoPaCo). They found the bulk of edits were due to incorrect diacritics, choosing the wrong homophone, and incorrect grammatical conjugation.

German Our German data combines two projects: RWSE Wikipedia Revision Dataset (Zesch, 2012) and The MERLIN corpus of language learners (Wisniewski et al., 2013) . These corpora were created to measure spelling difficulty and test models of contextual fitness. Unfortunately, the datasets are quite small so we have combined them here.

Czech Our Czech errors come from manually annotated essays written by non-native speakers ( Šebesta et al., 2017) . Here, the authors found an incredibly diverse set of errors, and therefore phenomena of interest: capitalization, incorrectly replacing voiced and voiceless consonants (e.g. z/s, g/k), missing palatalization (mat ǩe/matce), error in valence, pronominal reference, inflection, colloquial forms, and so forth. Their analysis gives us the best insight into how difficult it would be to synthetically generate truly natural errors. We found similarly rich errors in German (Section 7.2).

We insert these errors into the source-side of the parallel data by replacing every word in the corpus with an error if one exists in our dataset. When there is more than one possible replacement to choose we sample uniformly. Words for which there is no error are kept as is. Table 2 shows the number of words for which we were able to collect errors in each language, and the average number of errors per word. Despite the small size of the German and Czech datasets, we are able to replace up to half of the words in the corpus with errors. Due to the small size of the German and Czech datasets these percentages decrease for longer words (> 4 characters) to 25% and 32%, respectively.

4.2.2. SYNTHETIC NOISE:

In addition to naturally collected sources of error, we also experiment with four types of synthetic noise: Swap, Middle Random, Fully Random, and Keyboard Typo. 

Human:

According to a study from Cambridge university, it doesn't matter which order letters in a word are, the only important thing is that the first and the last letter appear in their correct place.

char2char Cambridge Universttte is one of the most important features of the Cambridge Universttten , which is one of the most important features of the Cambridge Universttten .

Nematus:

Luat eienr Stduie der Cambrant Unievrstilt splashed it kenie Rlloe in welcehr Reiehnfogle the Buhcstbaen in eniem Wred vorkmomen, die eingzie whcene Sahce ist, DSAs der ertse und der lettze Buhcstbaen stmimt . charCNN

According to the <unk> of the Cambridge University , it 's a little bit of crude oil in a little bit of recycling , which is a little bit of a cool cap , which is a little bit of a strong cap , that the fat and the <unk> bites is consistent .

Swap : Swap The simplest source of noise is swapping two letters (e.g. noise→nosie). This is common when typing quickly and is easily implemented. We perform one swap per word, but do not alter the first or last letters. For this reason, this noise is only applied to words of length ≥ 4.

Middle Random : Mid Following the claims of the previously discussed meme, we randomize the order of all the letters in a word except for the first and last (noise→nisoe). Again, by necessity, this means we do not alter words shorter than four characters.

Fully Random : Rand As we are unaware of any strong results on the importance of the first and last letters we also include completely randomized words (noise→iones). This is a particularly extreme case, but we include it for completeness. This type of noise is applied to all words.

Keyboard Typo : Key Finally, using the traditional keyboards for our languages, we randomly replace one letter in each word with an adjacent key (noise→noide). This type of error should be much easier than the random settings as most of the word is left intact, but does introduce a completely new character which will often break the templates a system has learned to rely on.

5. FAILURES TO TRANSLATE NOISY TEXTS:

Table 3 shows BLEU scores of models trained on clean (Vanilla) texts and tested on clean and noisy texts. All models suffer a significant drop in BLEU when evaluated on noisy texts. This is true for both natural noise and all kinds of synthetic noise. The more noise in the text, the worse the translation quality, with random scrambling producing the lowest BLEU scores.

The degradation in translation quality is especially severe in light of humans' ability to understand noisy texts. To illustrate this, consider the noisy text in Table 4 . Humans are quite good at understanding such scrambled texts in a variety of languages. 4 We also verified this by obtaining a One natural question is if robust spell checkers trained on human errors are sufficient to address this performance gap. To test this, we ran texts with and without natural errors through Google Translate. We then used Google's spell-checkers to correct the documents. We simply accepted the first suggestion for every detected mistake detected, and report results in Table 5 .

We found that in French and German, there was often only a single predicted correction and this corresponds to roughly +5 or more in BLEU. In Czech, however, there was often a large list of possible conjugations and changes, likely indicating that a rich grammatical model would be necessary to predict the correction. It is also important to note the substantial drops from vanilla text even with spell check. This suggests that natural noise cannot be easily addressed by existing tools.

6 DEALING WITH NOISE

6.1. STRUCTURE INVARIANT REPRESENTATIONS:

The three NMT models are all sensitive to word structure. The char2char and charCNN models both have convolutional layers on character sequences, designed to capture character n-grams. The model in Nematus is based on sub-word units obtained with BPE. It thus relies on character order within and across sub-word units. All these models are therefore sensitive to types of noise generated by character scrambling (Swap, Mid, and Rand). Can we improve model robustness by adding invariance to these kinds of noise? Perhaps the simplest such model is to take the average character embedding as a word representation. This model, referred to as meanChar, first generates a word representation by averaging character embeddings, and then proceeds with a word-level encoder similar to the charCNN model. The meanChar model is by definition insensitive to scrambling, although it is still sensitive to other kinds of noise (Key and Nat).

Table 6 (first row) shows the results of meanChar models trained on vanilla texts and tested on noisy texts (the results on vanilla texts are by definition equal to those on scrambled texts). Overall, the average character embedding proves to be a pretty good representation for translating scrambled texts: while performance drops by about 7 BLEU points below charCNN on vanilla French and German, it is much better than charCNN's performance on scrambled texts (compare to Table 3 ).

The results of meanChar on Czech are much worse, possibly due to its more complex morphology. However, the meanChar model performance degrades quickly on other kinds of noise as the model trained on vanilla texts was not designed to handle Nat and Key types of noise. To increase model robustness we follow a black-box adversarial training scenario, where the model is presented with adversarial examples that are generated without direct access to the model (Papernot et al., 2016a; 2017; Liu et al., 2017; Narodytska & Kasiviswanathan, 2017; Jia & Liang, 2017) . We replace the original training set with a noisy training set, where noise is introduced according to the description in Section 4.2. The noisy training set has exactly the same number of sentences and words as the training set. We have one fixed noisy training set per each noise type.5 

As shown in are not mutually-beneficial. Models trained on one do not perform well on the others. In particular, only models trained on natural noise can reasonably translate natural noise at test time. We find this result indicates an important difference between computational models and human performance, since humans can decipher random letter orderings without explicit training of this form.

Next, we test whether we can increase training robustness by exposing the model to multiple types of noise during training. Our motivation is to see if models can perform well on more than one kind of noise. We therefore mix up to three kinds of noise by sampling a noise method uniformly at random for each sentence. We then train a model on the mixed noisy training set and test it on both vanilla and (unmixed) noisy versions of the test set. We find that models trained on mixed noise are slightly worse than models trained on unmixed noise. However, the models trained on mixed noise are robust to the specific types of noise they were trained on. In particular, the model trained on a mix of Rand, Key, and Nat noise is robust to all noise kinds. Even though it is not the best on any one kind of noise, it achieves the best result on average.

This model is also able to translate the scrambled meme reasonably well:

"According to a study of Cambridge University, it doesn't matter which technology in a word is going to get the letters in a word that is the only important thing for the first and last letter."

7. ANALYSIS:



7.1. LEARNING MULTIPLE KINDS OF NOISE IN C H A RCNN:

The charCNN model was able to perform well on all kinds of noise by training on a mix of noise types. In particular, it performed well on scrambled characters even though its convolutions should be sensitive to the character order, as opposed to meanChar which is by definition invariant to character order. How then can charCNN learn to be robust to multiple kinds of noise at the same time? We speculate that different convolutional filters learn to be robust to different kinds of noise.

A convolutional filter can in principle capture a mean (or sum) operation by employing equal or close to equal weights.

To test this, we analyze the weights learned by charCNN models trained under four conditions: three models trained each on completely scrambled words (Rand), keyboard typos (Key), and natural human errors (Nat), as well as an ensemble model trained on a mix of Rand+Key+Nat kinds of noise. For each model, we compute the variance across the filter width (6 characters) for each one of the 1000 filters and for each one out of 25 character embedding dimensions. Intuitively, this variance captures how much a particular filter learns a uniform vs. non-uniform combination of characters. Then we average the variances across the 1000 filters. This yields 25 averaged variances, one for each character embedding dimension. Low average variance means that different filters tend to learn similar behaviors, while high average variance means that they learn different patterns.

Figure 2 shows a box plot of these averages for our three languages and four training conditions. Clearly, the variances of the weights learned by the Rand model are much smaller than those of the weights learned by any other setting. This makes sense as with random scrambling there are no patterns to detect in the data, so filters resort to close to uniform weights. In contrast, the Key and Nat settings introduce a large set of new patterns for the CNNs to try and learn, leading to high variances. Finally, the ensemble model trained on mixed noise appears to be in the middle as it tries to capture both the uniform relationships of Rand and the more diverse patterns of Nat + Key.

Moreover, the variance of variances (size of the box) is smallest in the Rand setting, larger in the mixed noise model, and largest in Key and Nat. This indicates that filters for different character embedding dimensions are more different from one another in Key and Nat models. In contrast, in the Rand model, the variance of variances is close to zero, indicating that in all character embedding dimensions the learned weights are of small variance; they do similar things, that is, the model learned to reproduce a representation similar to the meanChar model. The ensemble model again seems to find a balance between Rand and Key/Nat.

7.2. RICHNESS OF NATURAL NOISE:

Natural noise appears to be very different from synthetic noise. None of the models that were trained only on synthetic noise were able to perform well on natural noise. We manually analyzed a small sample (~40 examples) of natural noise from the German dataset. We found that the most common sources of noise are phonetic or phonological phenomena in the language (34%) and character omissions (32%). The rest are incorrect morphological conjugations of verbs, key swaps, character insertions, orthographic variants, and other errors. Table 8 shows examples of these kinds of noise.

The most common types of natural noise -phonological and omissions -are not directly captured by our synthetic noise generation, and demonstrate that good synthetic errors will likely require more explicit phonemic and linguistic knowledge. This discrepancy helps explain why the models trained on synthetic noise were not particularly successful in translating natural noise. 

8. CONCLUSION:

In this work, we have shown that character-based NMT models are extremely brittle and tend to break when presented with both natural and synthetic kinds of noise. We investigated methods for increasing their robustness by using a structure-invariant word representation and by ensemble training on adversarial examples of different kinds. We found that a character-based CNN can learn to address multiple types of errors that are seen in training. However, we observed rich characteristics of natural human errors that cannot be easily captured by existing models. Future work might investigate using phonetic and syntactic structure to generate more realistic synthetic noise.

We believe that more work is necessary in order to immune NMT models against natural noise. As corpora with natural noise are limited, another approach to future work is to design better NMT architectures that would be robust to noise without seeing it in the training data. New psychology results on how humans cope with natural noise might point to possible solutions to this problem.

:

https://github.com/ybisk/charNMT-noise

Matt Davis has a wide collection of translations of this text in multiple languages: https://www.mrc-cbu.cam.ac.uk/personal/matt.davis/Cmabrigde/.

When replacing words in the input, we inevitably make some of the same replacements on both the training and test sets. We verify this does not decrease the percent of unseen words in testing. Conversely, we found it increases for all synthetic noise types and is similar for the vanilla and natural noise conditions.

Figures and Tables:

Figure 1: Degradation of Nematus (Sennrich et al., 2017) and char2char (Lee et al., 2017) performance as noise increases.

Figure 2: Variances of charCNN weights when trained on only Key, Natural, Random noise and on a mix of all three are shown in red, green, blue, and white, respectively



Table content: <table><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td>Nematus</td><td/><td/><td/><td/></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td>Random</td><td/><td>Swap</td><td/><td colspan="2">Natural</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td>0</td><td>34.22</td><td>0</td><td>34.22</td><td/><td>0</td><td>34.22</td><td>40 Nematus</td><td>Random</td><td>Swap</td><td>Natural</td><td>70</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td>7.7</td><td>27.16</td><td>5.2</td><td>29.86</td><td/><td>3.7</td><td>31.27</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td>15.3</td><td>21.77</td><td>10.4</td><td>25.59</td><td/><td>7.6</td><td>28.52</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td>23.2</td><td>15.73</td><td>15.3</td><td>21.56</td><td/><td>11.4</td><td>25.42</td><td>30</td><td>52.5</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td>30.8</td><td>11.45</td><td>20.6</td><td>17.91</td><td/><td>15.2</td><td>22.60</td></tr><tr><td>wap</td><td/><td colspan="2">Real</td><td/><td/><td/><td/><td/><td/><td>61.5 38.0 46.3 53.6</td><td>1.87 8.49 5.28 3.19</td><td>40.9 25.9 30.3 35.8</td><td>7.05 14.51 12.22 9.49</td><td/><td>30.1 18.8 22.6 26.2</td><td>13.92 20.44 17.62 16.52</td><td>BLEU</td><td>20</td><td>BLEU</td><td>35</td></tr><tr><td>wap</td><td/><td colspan="2">Real</td><td/><td/><td/><td/><td/><td/><td>69.1</td><td>0.84</td><td>45.7</td><td>5.36</td><td/><td>34.0</td><td>12.02</td><td>10</td><td>17.5</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td>76.7</td><td>0.29</td><td>51.1</td><td>3.39</td><td/><td>37.8</td><td>10.68</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td>0</td><td>0</td><td>20</td><td>40</td><td>60</td><td>80</td><td>0</td><td>0</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td>char2char</td><td/><td/><td/><td/><td>% tokens changed</td><td>% words</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td>Random</td><td/><td>Swap</td><td/><td/><td>Natural</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td>Nematus</td><td/><td>0 7.7</td><td>29.97 24.33</td><td>0 5.2</td><td colspan="2">29.97 26.85</td><td>0 3.7</td><td>29.97 28.15</td><td>70 Nematus 40 Char2Char</td><td>Random</td><td>Swap</td><td>Natural</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td>15.3</td><td>19.76</td><td>10.4</td><td colspan="2">23.8</td><td>7.6</td><td>25.59</td></tr><tr><td>0</td><td>80</td><td>90</td><td>100</td><td/><td/><td/><td/><td/><td/><td>23.2</td><td>14.96</td><td>15.3</td><td colspan="2">21.04</td><td>11.4</td><td>24.04</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td>30.8</td><td>10.99</td><td>20.6</td><td colspan="2">18.73</td><td>15.2</td><td>21.38</td><td>52.5 30</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td>38.0</td><td>8.36</td><td>25.9</td><td colspan="2">15.91</td><td>18.8</td><td>20.23</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td>46.3 53.6 61.5</td><td>5.22 3.40 1.94</td><td>30.3 35.8 40.9</td><td colspan="2">13.57 11.65 9.39</td><td>22.6 26.2 30.1</td><td>18.20 16.79 15.33</td><td>BLEU BLEU</td><td>35 20</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td>69.1</td><td>1.03</td><td>45.7</td><td colspan="2">7.47</td><td>34.0</td><td>14.00</td></tr><tr><td>ap ap</td><td colspan="3">Natural Natural</td><td/><td/><td/><td/><td/><td/><td>76.7</td><td>0.28</td><td>51.1</td><td colspan="2">5.68</td><td>37.8</td><td>12.68</td><td>17.5 10</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td>0 0</td><td>0 0</td><td>20</td><td>40</td><td>60</td><td>80</td></tr><tr><td colspan="2">har2char</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td>% words changed % tokens changed</td></tr><tr><td/><td>Swap</td><td/><td/><td>Natural</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>29.97 24.33</td><td/><td>0 5.2</td><td>29.97 26.85</td><td>0 3.7</td><td>29.97 28.15</td><td/><td/><td>40 Char2Char</td><td/><td colspan="2">Random</td><td colspan="2">Swap</td><td/><td colspan="2">Natural</td><td>70.0 Char2Char</td></tr><tr><td>19.76</td><td colspan="2">10.4</td><td>23.8</td><td>7.6</td><td>25.59</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>14.96</td><td colspan="2">15.3</td><td>21.04</td><td>11.4</td><td>24.04</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>10.99</td><td colspan="2">20.6</td><td>18.73</td><td>15.2</td><td>21.38</td><td/><td/><td>30</td><td/><td/><td/><td/><td/><td/><td/><td>52.5</td></tr><tr><td>8.36</td><td colspan="2">25.9</td><td>15.91</td><td>18.8</td><td>20.23</td><td/><td/><td/><td/><td>0</td><td>68.2</td><td>0</td><td>68.2</td><td/><td>0</td><td>68.2</td><td>0</td><td>68.2</td></tr><tr><td>5.22 3.40 1.94 60</td><td colspan="2">30.3 35.8 40.9</td><td>13.57 11.65 9.39 80</td><td>22.6 26.2 30.1</td><td>18.20 16.79 15.33</td><td colspan="2">BLEU</td><td>20</td><td/><td>7.7 15.3 23.2</td><td>61.0 55.8 49.6</td><td>5.2 10.4 15.3</td><td>64.3 59.9 55.7</td><td/><td>3.7 7.6 11.4</td><td>65.6 62.7 59.6</td><td>10 20 BLEU 30</td><td>35.0</td><td>61.38 54.56 47.74</td></tr><tr><td>1.03</td><td colspan="2">45.7</td><td>7.47</td><td>34.0</td><td>14.00</td><td/><td/><td/><td/><td>30.8</td><td>44.3</td><td>20.6</td><td>52.0</td><td/><td>15.2</td><td>56.7</td><td>40</td><td>40.92</td></tr><tr><td>0.28</td><td colspan="2">51.1</td><td>5.68</td><td>37.8</td><td>12.68</td><td/><td/><td>10</td><td/><td>38.0</td><td>38.5</td><td>25.9</td><td>48.8</td><td/><td>18.8</td><td>53.6</td><td>50</td><td>17.5</td><td>34.1</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td>46.3</td><td>33.0</td><td>30.3</td><td>45.6</td><td/><td>22.6</td><td>51.3</td><td>60</td><td>27.28</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td>53.6</td><td>27.4</td><td>35.8</td><td>41.6</td><td/><td>26.2</td><td>49.3</td><td>70</td><td>20.46</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td>0</td><td/><td>61.5</td><td>22.8</td><td>40.9</td><td>38.0</td><td/><td>30.1</td><td>46.1</td><td>80</td><td>0.0</td><td>13.64</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td>0</td><td>20</td><td>69.1</td><td>40 17.7</td><td>45.7</td><td>34.3</td><td>60</td><td>34.0</td><td>43.5</td><td>80</td><td>90</td><td>0</td><td>6.82</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td colspan="4">% tokens changed 76.7 13.8 51.1 30.5</td><td/><td>37.8</td><td>40.8</td><td>100</td><td>0</td><td>% words changed</td></tr><tr><td>matus</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td/><td>Swap</td><td/><td colspan="2">Natural</td><td>Linear</td><td/><td/><td>y =</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td>(-68.2/100)</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td>x+68.2</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>68.2</td><td>0</td><td/><td>68.2</td><td>0</td><td>68.2</td><td/><td>0</td><td>68.2</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>61.0</td><td>5.2</td><td/><td>64.3</td><td>3.7</td><td>65.6</td><td colspan="2">10</td><td>61.38</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>55.8</td><td>10.4</td><td/><td>59.9</td><td>7.6</td><td>62.7</td><td colspan="2">20</td><td>54.56</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>49.6</td><td>15.3</td><td/><td>55.7</td><td>11.4</td><td>59.6</td><td colspan="2">30</td><td>47.74</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>44.3</td><td>20.6</td><td/><td>52.0</td><td>15.2</td><td>56.7</td><td colspan="2">40</td><td>40.92</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>38.5</td><td>25.9</td><td/><td>48.8</td><td>18.8</td><td>53.6</td><td colspan="2">50</td><td>34.1</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>33.0</td><td>30.3</td><td/><td>45.6</td><td>22.6</td><td>51.3</td><td colspan="2">60</td><td>27.28</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>27.4</td><td>35.8</td><td/><td>41.6</td><td>26.2</td><td>49.3</td><td colspan="2">70</td><td>20.46</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>22.8</td><td>40.9</td><td/><td>38.0</td><td>30.1</td><td>46.1</td><td colspan="2">80</td><td>13.64</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>17.7</td><td>45.7</td><td/><td>34.3</td><td>34.0</td><td>43.5</td><td colspan="2">90</td><td>6.82</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>13.8</td><td>51.1</td><td/><td>30.5</td><td>37.8</td><td>40.8</td><td colspan="2">100</td><td>0</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>r2char</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td colspan="2">Swap</td><td/><td colspan="2">Natural</td><td colspan="2">Linear</td><td colspan="2">y =</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td/><td/><td/><td/><td/><td/><td/><td colspan="2">(-66/100</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td/><td/><td/><td/><td/><td/><td/><td colspan="2">)x + 66</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>66.0</td><td colspan="2">0</td><td>66.0</td><td>0</td><td>66.0</td><td>0</td><td/><td>66</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>60.6</td><td colspan="2">5.2</td><td>63.3</td><td>3.7</td><td>64.2</td><td>10</td><td/><td>59.4</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>55.3</td><td colspan="2">10.4</td><td>60.2</td><td>7.6</td><td>62.4</td><td>20</td><td/><td>52.8</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>49.5</td><td colspan="2">15.3</td><td>57.3</td><td>11.4</td><td>60.3</td><td>30</td><td/><td>46.2</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>43.6</td><td colspan="2">20.6</td><td>54.2</td><td>15.2</td><td>58.0</td><td>40</td><td/><td>39.6</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>38.3</td><td colspan="2">25.9</td><td>51.5</td><td>18.8</td><td>56.7</td><td>50</td><td/><td>33</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>33.3</td><td colspan="2">30.3</td><td>48.3</td><td>22.6</td><td>54.7</td><td>60</td><td/><td>26.4</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>27.9</td><td colspan="2">35.8</td><td>45.3</td><td>26.2</td><td>52.5</td><td>70</td><td/><td>19.8</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>23.2</td><td colspan="2">40.9</td><td>42.2</td><td>30.1</td><td>50.7</td><td>80</td><td/><td>13.2</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>18.5</td><td colspan="2">45.7</td><td>39.4</td><td>34.0</td><td>48.6</td><td>90</td><td/><td>6.6</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>14.1</td><td colspan="2">51.1</td><td>35.8</td><td>37.8</td><td>46.3</td><td>100</td><td/><td>0</td><td/><td/><td/><td/><td/><td/><td/></tr></table>



Table content: <table><tr><td>0</td><td>66.0</td><td>0</td><td>66.0</td><td>0</td><td>66.0</td><td>0</td><td>66</td></tr><tr><td>7.7</td><td>60.6</td><td>5.2</td><td>63.3</td><td>3.7</td><td>64.2</td><td>10</td><td>59.4</td></tr><tr><td>15.3</td><td>55.3</td><td>10.4</td><td>60.2</td><td>7.6</td><td>62.4</td><td>20</td><td>52.8</td></tr><tr><td>23.2</td><td>49.5</td><td>15.3</td><td>57.3</td><td>11.4</td><td>60.3</td><td>30</td><td>46.2</td></tr><tr><td>30.8</td><td>43.6</td><td>20.6</td><td>54.2</td><td>15.2</td><td>58.0</td><td>40</td><td>39.6</td></tr><tr><td>38.0</td><td>38.3</td><td>25.9</td><td>51.5</td><td>18.8</td><td>56.7</td><td>50</td><td>33</td></tr><tr><td>46.3</td><td>33.3</td><td>30.3</td><td>48.3</td><td>22.6</td><td>54.7</td><td>60</td><td>26.4</td></tr><tr><td>53.6</td><td>27.9</td><td>35.8</td><td>45.3</td><td>26.2</td><td>52.5</td><td>70</td><td>19.8</td></tr><tr><td>61.5</td><td>23.2</td><td>40.9</td><td>42.2</td><td>30.1</td><td>50.7</td><td>80</td><td>13.2</td></tr><tr><td>69.1</td><td>18.5</td><td>45.7</td><td>39.4</td><td>34.0</td><td>48.6</td><td>90</td><td>6.6</td></tr><tr><td>76.7</td><td>14.1</td><td>51.1</td><td>35.8</td><td>37.8</td><td>46.3</td><td>100</td><td>0</td></tr></table>

Statistics for the source-side of French/German/Czech→English parallel corpora.

Table content: <table><tr><td/><td>French</td><td>German</td><td>Czech</td></tr><tr><td/><td>Train Dev Test</td><td>Train Dev Test</td><td>Train Dev Test</td></tr><tr><td>Sentences</td><td>235K 2.5K 0.8K</td><td>210K 2.5K 1.4K</td><td>122K 20K 1K</td></tr><tr><td>Words</td><td>5.2M 55K 16K</td><td>4M 50K 26K</td><td>2.1M 35K 15K</td></tr></table>

Average number of available edits per word in natural noise datasets and the corresponding token recall of those edits on the training and test splits.We insert noise into the source-side of the parallel MT data by utilizing naturally occurring errors and generating synthetic ones. In order to facilitate future work on noise in NMT, we release code and data for generating the noise used in our experiments.

Table content: <table><tr><td>French</td><td>German</td><td>Czech</td></tr><tr><td>Words Errors Train Test</td><td>Words Errors Train Test</td><td>Words Errors Train Test</td></tr><tr><td>65,156 2.7 40% 41%</td><td>1,344 2.5 37% 40%</td><td>6,036 2.6 46% 51%</td></tr><tr><td colspan="2">4.2 NOISE: NATURAL AND ARTIFICIAL</td><td/></tr></table>

The effect of Natural (Nat) and synthetic noise (Swap swap, Middle Random Mid, Fully Random Rand, and Keyboard Typo Key) on models trained on clean (Vanilla) texts.

Table content: <table><tr><td/><td/><td/><td>Synthetic</td><td/></tr><tr><td/><td/><td>Vanilla</td><td>Swap Mid Rand Key</td><td>Nat</td></tr><tr><td>French</td><td>charCNN</td><td>42.54</td><td>10.52 9.71 1.71 8.26</td><td>17.42</td></tr><tr><td/><td>charCNN</td><td>34.79</td><td>9.25 8.37 1.02 6.40</td><td>14.02</td></tr><tr><td>German</td><td>char2char</td><td>29.97</td><td>5.68 5.46 0.28 2.96</td><td>12.68</td></tr><tr><td/><td>Nematus</td><td>34.22</td><td>3.39 5.16 0.29 0.61</td><td>10.68</td></tr><tr><td/><td>charCNN</td><td>25.99</td><td>6.56 6.67 1.50 7.13</td><td>10.20</td></tr><tr><td>Czech</td><td>char2char</td><td>25.71</td><td>3.90 4.24 0.25 2.88</td><td>11.42</td></tr><tr><td/><td>Nematus</td><td>29.65</td><td>2.94 4.09 0.66 1.41</td><td>11.88</td></tr></table>

An example noisy text with human and machine translations. Input Luat eienr Stduie der Cambrdige Unievrstit speilt es kenie Rlloe in welcehr Reiehnfogle die Buhcstbaen in eniem Wrot vorkmomen, die eingzie whctige Sahce ist, dsas der ertse und der lettze Buhcstbaen stmimt .

Table content: <table/>

Google Translate's performance with natural errors and the gains from using spell checking. German native-speaker, unfamiliar with the meme. As shown in the table, the speaker had no trouble understanding and translating the sentence properly. In contrast, the state-ofthe-art systems (char2char and Nematus) fail on this text.

Table content: <table><tr><td/><td>French</td><td/><td/><td colspan="2">German</td><td/><td/><td>Czech</td></tr><tr><td colspan="3">Vanilla Nat Spelling</td><td colspan="4">Vanilla Nat Spelling</td><td colspan="3">Vanilla Nat Spelling</td></tr><tr><td>43.3</td><td>16.7</td><td>21.4</td><td>38.7</td><td>18.6</td><td colspan="2">25.0</td><td>26.5</td><td>12.3</td><td>11.2</td></tr><tr><td colspan="10">Table 6: Results of meanChar models trained and tested on different noise conditions: Scrambled</td></tr><tr><td colspan="5">(Scr), Keyboard Typo (Key), and Natural (Nat).</td><td/><td/><td/><td/></tr><tr><td/><td/><td/><td>French</td><td/><td/><td>German</td><td/><td colspan="2">Czech</td></tr><tr><td>Train</td><td>Test</td><td>Scr</td><td>Key</td><td>Nat</td><td>Scr</td><td>Key</td><td>Nat</td><td colspan="2">Scr Key Nat</td></tr><tr><td>Vanilla</td><td/><td>34.26</td><td colspan="3">4.27 12.58 27.53</td><td>3.34</td><td colspan="3">9.41 3.73 2.06 3.25</td></tr><tr><td>Key</td><td/><td colspan="4">31.88 29.75 13.16 10.04</td><td>8.84</td><td colspan="2">4.45 2.03</td><td>1.9 1.42</td></tr><tr><td>Nat</td><td/><td>26.94</td><td colspan="3">5.30 27.49 15.65</td><td colspan="4">3.06 26.26 1.66 1.52 1.58</td></tr><tr><td colspan="2">Rand + Key</td><td colspan="2">13.60 11.09</td><td colspan="6">6.12 26.59 22.41 11.07 9.97 7.48 4.21</td></tr><tr><td colspan="2">Rand + Nat</td><td>28.28</td><td colspan="3">5.10 20.40 13.87</td><td colspan="4">3.73 12.74 4.89 2.82 3.42</td></tr><tr><td colspan="2">Key + Nat</td><td colspan="3">31.30 26.94 24.24</td><td>6.62</td><td>5.41</td><td colspan="3">5.75 1.62 1.68 1.58</td></tr><tr><td colspan="2">Rand + Key + Nat</td><td>3.10</td><td>3.28</td><td>2.76</td><td>8.02</td><td>5.79</td><td colspan="3">6.36 1.73 1.74 1.66</td></tr><tr><td colspan="2">translation from a</td><td/><td/><td/><td/><td/><td/><td/></tr></table>

Results of charCNN models trained and tested on different noise conditions.

Table content: <table><tr><td/><td>Train</td><td>Test</td><td colspan="2">Vanilla Swap</td><td colspan="2">Mid Rand</td><td>Key</td><td>Nat</td><td>Ave</td></tr><tr><td/><td>Swap</td><td/><td colspan="3">39.01 42.56 33.64</td><td>2.72</td><td>4.85 16.43 23.20</td></tr><tr><td/><td>Mid</td><td/><td colspan="3">42.46 42.19 42.17</td><td>3.36</td><td>6.20 18.22 25.77</td></tr><tr><td>French</td><td>Rand Key</td><td/><td colspan="5">39.53 39.46 39.13 39.73 38.49 10.56 8.69 1.08 38.88 16.86 19.10 3.11 16.63 29.60</td></tr><tr><td/><td>Nat</td><td/><td colspan="2">28.77 12.45</td><td>8.39</td><td>1.03</td><td>6.61 36.00 15.54</td></tr><tr><td/><td colspan="2">Rand + Key</td><td colspan="5">39.23 38.85 38.89 39.13 38.22 18.71 35.51</td></tr><tr><td/><td colspan="2">Rand + Nat</td><td colspan="4">36.86 38.95 38.44 38.63</td><td>6.67 33.89 32.24</td></tr><tr><td/><td colspan="2">Key + Nat</td><td colspan="3">38.47 17.33 10.54</td><td colspan="2">1.52 38.62 34.66 23.52</td></tr><tr><td/><td colspan="2">Rand + Key + Nat</td><td colspan="5">36.97 36.92 36.65 36.64 35.25 31.77 35.70</td></tr><tr><td/><td>Swap</td><td/><td colspan="3">32.66 34.76 29.03</td><td>2.19</td><td>4.78 13.37 19.47</td></tr><tr><td>German</td><td>Mid Rand</td><td/><td colspan="4">34.32 34.26 34.27 33.65 33.44 33.75 33.56 3.50</td><td>5.08 14.43 20.98 3.00 14.47 25.31</td></tr><tr><td/><td>Key</td><td/><td colspan="2">32.87 10.13</td><td>8.39</td><td colspan="2">1.16 33.28 13.88 16.62</td></tr><tr><td/><td>Nat</td><td/><td>25.79</td><td>8.20</td><td>5.73</td><td>0.93</td><td>4.80 34.59 13.34</td></tr><tr><td/><td colspan="2">Rand + Key</td><td colspan="5">32.03 31.57 31.32 31.58 31.23 15.59 28.89</td></tr><tr><td/><td colspan="2">Rand + Nat</td><td colspan="4">32.37 32.40 31.91 32.11</td><td>4.77 33.00 27.76</td></tr><tr><td/><td colspan="2">Key + Nat</td><td colspan="2">30.39 13.51</td><td>8.99</td><td colspan="2">1.53 32.23 33.46 20.02</td></tr><tr><td/><td colspan="2">Rand + Key + Nat</td><td colspan="5">31.29 30.93 30.54 30.04 29.81 31.60 30.70</td></tr><tr><td/><td>Swap</td><td/><td colspan="3">24.22 24.90 18.72</td><td>2.72</td><td>6.00</td><td>9.03 14.27</td></tr><tr><td>Czech</td><td>Mid Rand</td><td/><td colspan="4">23.81 24.52 24.08 23.44 23.31 23.24 23.47 3.96</td><td>6.34 3.70</td><td>9.54 15.38 8.10 17.54</td></tr><tr><td/><td>Key</td><td/><td>23.15</td><td>7.06</td><td>6.04</td><td colspan="2">1.56 22.80 10.16 11.80</td></tr><tr><td/><td>Nat</td><td/><td>18.04</td><td>5.36</td><td>4.48</td><td>1.47</td><td>6.71 21.64</td><td>9.62</td></tr><tr><td/><td colspan="2">Rand + Key</td><td colspan="5">21.46 20.81 20.90 20.59 19.48</td><td>8.72 18.66</td></tr><tr><td/><td colspan="2">Rand + Nat</td><td colspan="4">20.59 21.56 20.49 20.53</td><td>5.89 18.39 17.91</td></tr><tr><td/><td colspan="2">Key + Nat</td><td>19.55</td><td>6.59</td><td>5.72</td><td colspan="2">1.40 21.31 19.54 12.35</td></tr><tr><td/><td colspan="2">Rand + Key + Nat</td><td colspan="5">21.30 21.33 20.38 19.94 19.25 18.38 20.10</td></tr><tr><td colspan="4">6.2 BLACK-BOX ADVERSARIAL TRAINING</td><td/><td/><td/></tr></table>

Table 6 (second block), training on noisy text can lead to improved performance. The meanChar models trained on Key perform well on Key in French, but not in the other languages.The models trained on Nat perform well in French and German, but not in Czech. Overall, training the meanChar model on noisy text does not appear to consistently increase its robustness to different kinds of noise. The meanChar model however was not expected to perform well on nonscrambling types of noise. Next we test whether the more complicated charCNN model is more robust to different kinds of noise, by training on noisy texts. The results are shown in Table7.In general, charCNN models that are trained on a specific kind of noise perform well on the same kind of noise at test time (results in bold). All models also maintain a fairly good quality on vanilla texts.The robust training is sensitive to the kind of noise. Among the scrambling methods (Swap/Mid/Rand), more noise helps in training: models trained on random noise can still translate Swap/Mid noise, but not vice versa. The three broad classes of noise(scrambling, Key, Nat)

Table content: <table><tr><td>Keyboard</td><td>Natural</td><td>Random</td><td>Ensemble</td></tr><tr><td>Variance</td><td/><td/><td/></tr><tr><td/><td>Language</td><td/><td/></tr><tr><td/><td>1</td><td/><td/></tr></table>

Examples of natural noise from the German errors dataset.

Table content: <table><tr><td>Error type</td><td>Examples</td></tr><tr><td>Phonetic</td><td>Tut/Tud (devoicing of final stops), sieht/zieht (s = /z/ before vowel), Trotzdem/Trozdem</td></tr><tr><td/><td>(tz = /z/), gekriegt/gekrigt (vowel length), Natürlich/Naturlich/Näturlich (diacritics)</td></tr><tr><td>Omission</td><td>erfahren/erfaren, Babysitter/Babysiter, selbst/sebst, Hausschuhe/Hausschue</td></tr><tr><td colspan="2">Morphological wohnt/wonnen, fortsetzt/forzusetzen, wünsche/wünchen</td></tr><tr><td>Key swap</td><td>Eltern/Eltren, Deine/Diene, nichts/nichst, Bahn/Bhan</td></tr><tr><td>Other</td><td>Agglomerationen/Agromelationen (omission + letter swap), Hausaufgabe/Hausausgabe,</td></tr><tr><td/><td>Thema/Temer, Detailhandelsfachfrau/Deitellhandfachfrau</td></tr></table>

Peer review (iclr2018):
 This paper investigates the impact of character-level noise on various flavours of neural machine translation. It tests 4 different NMT systems with varying degrees and types of character awareness, including a novel meanChar system that uses averaged unigram character embeddings as word representations on the source side. The authors test these systems under a variety of noise conditions, including synthetic scrambling and keyboard replacements, as well as natural (human-made) errors found in other corpora and transplanted to the training and/or testing bitext via replacement tables. They show that all NMT systems, whether BPE or character-based, degrade drastically in quality in the presence of both synthetic and natural noise, and that it is possible to train a system to be resistant to these types of noise by including them in the training data. Unfortunately, they are not able to show any types of synthetic noise helping address natural noise. However, they are able to show that a system trained on a mixture of error types is able to perform adequately on all types of noise. This is a thorough exploration of a mostly under-studied problem. The paper is well-written and easy to follow. The authors do a good job of positioning their study with respect to related work on black-box adversarial techniques, but overall, by working on the topic of noisy input data at all, they are guaranteed novelty. The inclusion of so many character-based systems is very nice, but it is the inclusion of natural sources of noise that really makes the paper work. Their transplanting of errors from other corpora is a good solution to the problem, and one likely to be built upon by others. In terms of negatives, it feels like this work is just starting to scratch the surface of noise in NMT. The proposed meanChar architecture doesn’t look like a particularly good approach to producing noise-resistant translation systems, and the alternative solution of training on data where noise has been introduced through replacement tables isn’t extremely satisfying. Furthermore, the use of these replacement tables means that even when the noise is natural, it’s still kind of artificial. Finally, this paper doesn’t seem to be a perfect fit for ICLR, as it is mostly experimental with few technical contributions that are likely to be impactful; it feels like it might be more at home and have greater impact in a *ACL conference. Regarding the artificialness of their natural noise - obviously the only solution here is to find genuinely noisy parallel data, but even granting that such a resource does not yet exist, what is described here feels unnaturally artificial. First of all, errors learned from the noisy data sources are constrained to exist within a word. This tilts the comparison in favour of architectures that retain word boundaries (such as the charCNN system here), while those systems may struggle with other sources of errors such as missing spaces between words. Second, if I understand correctly, once an error is learned from the noisy data, it is applied uniformly and consistently throughout the training and/or test data. This seems worse than estimating the frequency of the error and applying them stochastically (or trying to learn when an error is likely to occur). I feel like these issues should at least be mentioned in the paper, so it is clear to the reader that there is work left to be done in evaluating the system on truly natural noise. Also, it is somewhat jarring that only the charCNN approach is included in the experiments with noisy training data (Table 6). I realize that this is likely due to computational or time constraints, but it is worth providing some explanation in the text for why the experiments were conducted in this manner. On a related note, the line in the abstract stating that “... a character convolutional neural network is able to simultaneously learn representations robust to multiple kinds of noise” implies that the other (non-charCNN) architectures could not learn these representations, when in reality, they simply weren’t given the chance. Section 7.2 on the richness of natural noise is extremely interesting, but maybe less so to an ICLR audience. From my perspective, it would be interesting to see that section expanded, or used as the basis for future work on improve architectures or training strategies. I have only one small, specific suggestion: at the end of Section 3, consider deleting the last paragraph break, so there is one paragraph for each system (charCNN currently has two paragraphs). [edited for typos]