Venue: ICLR 2018 conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper investigates the impact of noisy input on Machine Translation, and tests simple ways to make NMT models more robust.

Overall the paper is a clearly written, well described report of several experiments. It shows convincingly that standard NMT models completely break down on both natural "noise" and various types of input perturbations. It then tests how the addition of noise in the input helps robustify the charCNN model somewhat. The extent of the experiments is quite impressive: three different NMT models are tried, and one is used in extensive experiments with various noise combinations.

This study clearly addresses an important issue in NMT and will be of interest to many in the NLP community. The outcome is not entirely surprising (noise hurts and training and the right kind of noise helps) but the impact may be. I wonder if you could put this in the context of "training with input noise", which has been studied in Neural Network for a while (at least since the 1990s). I.e. it could be that each type of noise has a different regularizing effect, and clarifying what these regularizers are may help understand the impact of the various types of noise. Also, the bit of analysis in Sections 6.1 and 7.1 is promising, if maybe not so conclusive yet.

A few constructive criticisms:

The way noise is included in training (sec. 6.2) could be clarified (unless I missed it) e.g. are you generating a fixed "noisy" training set and adding that to clean data? Or introducing noise "on-line" as part of the training? If fixed, what sizes were tried? More information on the experimental design would help.

Table 6 is highly suspect: Some numbers seem to have been copy-pasted in the wrong cells, eg. the "Rand" line for German, or the Swap/Mid/Rand lines for Czech. It's highly unlikely that training on noisy Swap data would yield a boost of +18 BLEU points on Czech -- or you have clearly found a magical way to improve performance.

Although the amount of experiment is already important, it may be interesting to check whether all se2seq models react similarly to training with noise: it could be that some architecture are easier/harder to robustify in this basic way.

[Response read -- thanks]
I agree with authors that this paper is suitable for ICLR, although it will clearly be of interest to ACL/MT-minded folks.
review_title: Interesting study of the impact of noisy text on MT quality

final_score: rating: 7: Good paper, accept
review_content: This paper empirically investigates the performance of character-level NMT systems in the face of character-level noise, both synthesized and natural. The results are not surprising:

* NMT is terrible with noise.

* But it improves on each noise type when it is trained on that noise type.

What I like about this paper is that:

1) The experiments are very carefully designed and thorough.

2) This problem might actually matter. Out of curiosity, I ran the example (Table 4) through Google Translate, and the result was gibberish. But as the paper shows, it’s easy to make NMT robust to this kind of noise, and Google (and other NMT providers) could do this tomorrow. So this paper could have real-world impact.

3) Most importantly, it shows that NMT’s handling of natural noise does *not* improve when trained with synthetic noise; that is, the character of natural noise is very different. So solving the problem of natural noise is not so simple… it’s a *real* problem. Speculating, again: commercial MT providers have access to exactly the kind of natural spelling correction data that the researchers use in this paper, but at much larger scale. So these methods could be applied in the real world. (It would be excellent if an outcome of this paper was that commercial MT providers answered it’s call to provide more realistic noise by actually providing examples.)

There are no fancy new methods or state-of-the-art numbers in this paper. But it’s careful, curiosity-driven empirical research of the type that matters, and it should be in ICLR.
review_title: Nice empirical paper on robustness of NMT

final_score: rating: 7: Good paper, accept
review_content: This paper investigates the impact of character-level noise on various flavours of neural machine translation. It tests 4 different NMT systems with varying degrees and types of character awareness, including a novel meanChar system that uses averaged unigram character embeddings as word representations on the source side. The authors test these systems under a variety of noise conditions, including synthetic scrambling and keyboard replacements, as well as natural (human-made) errors found in other corpora and transplanted to the training and/or testing bitext via replacement tables. They show that all NMT systems, whether BPE or character-based, degrade drastically in quality in the presence of both synthetic and natural noise, and that it is possible to train a system to be resistant to these types of noise by including them in the training data. Unfortunately, they are not able to show any types of synthetic noise helping address natural noise. However, they are able to show that a system trained on a mixture of error types is able to perform adequately on all types of noise.

This is a thorough exploration of a mostly under-studied problem. The paper is well-written and easy to follow. The authors do a good job of positioning their study with respect to related work on black-box adversarial techniques, but overall, by working on the topic of noisy input data at all, they are guaranteed novelty. The inclusion of so many character-based systems is very nice, but it is the inclusion of natural sources of noise that really makes the paper work. Their transplanting of errors from other corpora is a good solution to the problem, and one likely to be built upon by others. In terms of negatives, it feels like this work is just starting to scratch the surface of noise in NMT. The proposed meanChar architecture doesn’t look like a particularly good approach to producing noise-resistant translation systems, and the alternative solution of training on data where noise has been introduced through replacement tables isn’t extremely satisfying. Furthermore, the use of these replacement tables means that even when the noise is natural, it’s still kind of artificial. Finally, this paper doesn’t seem to be a perfect fit for ICLR, as it is mostly experimental with few technical contributions that are likely to be impactful; it feels like it might be more at home and have greater impact in a *ACL conference.

Regarding the artificialness of their natural noise - obviously the only solution here is to find genuinely noisy parallel data, but even granting that such a resource does not yet exist, what is described here feels unnaturally artificial. First of all, errors learned from the noisy data sources are constrained to exist within a word. This tilts the comparison in favour of architectures that retain word boundaries (such as the charCNN system here), while those systems may struggle with other sources of errors such as missing spaces between words. Second, if I understand correctly, once an error is learned from the noisy data, it is applied uniformly and consistently throughout the training and/or test data. This seems worse than estimating the frequency of the error and applying them stochastically (or trying to learn when an error is likely to occur). I feel like these issues should at least be mentioned in the paper, so it is clear to the reader that there is work left to be done in evaluating the system on truly natural noise.

Also, it is somewhat jarring that only the charCNN approach is included in the experiments with noisy training data (Table 6). I realize that this is likely due to computational or time constraints, but it is worth providing some explanation in the text for why the experiments were conducted in this manner. On a related note, the line in the abstract stating that “... a character convolutional neural network  is able to simultaneously learn representations robust to multiple kinds of noise” implies that the other (non-charCNN) architectures could not learn these representations, when in reality, they simply weren’t given the chance.

Section 7.2 on the richness of natural noise is extremely interesting, but maybe less so to an ICLR audience. From my perspective, it would be interesting to see that section expanded, or used as the basis for future work on improve architectures or training strategies.

I have only one small, specific suggestion: at the end of Section 3, consider deleting the last paragraph break, so there is one paragraph for each system (charCNN currently has two paragraphs).

[edited for typos]
review_title: Good work on an under-studied problem

Meta-review: The pros and cons of this paper cited by the reviewers can be summarized below:

Pros:
* The paper is a first attempt to investigate an under-studied area in neural MT (and potentially other applications of sequence-to-sequence models as well)
* This area might have a large impact; existing models such as Google Translate fail badly on the inputs described here
* Experiments are very carefully designed and thorough
* Experiments on not only synthetic but also natural noise add significant reliability to the results
* Paper is well-written and easy to follow

Cons:
* There may be better architectures for this problem than the ones proposed here
* Even the natural noise is not entirely natural, e.g. artificially constrained to exist within words
* Paper is not a perfect fit to ICLR (although ICLR is attempting to cast a wide net, so this alone is not a critical criticism of the paper)

This paper had uniformly positive reviews and has potential for large real-world impact.
Decision: Accept (Oral)