user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Hallucinations in neural machine translation

Paper content:
Neural machine translation (NMT) systems have reached state of the art performance in translating text and widely deployed.
Yet little is understood about how these systems function or break.
Here we show that NMT systems are susceptible to producing highly pathological translations that are completely untethered from the source material, which we term hallucinations.
Such pathological translations are problematic because they are are deeply disturbing of user trust and easy to find.
We describe a method t generate hallucinations and show that many common variations of the NMT architecture are susceptible to them.
We study a variety of approaches to reduce the frequency of hallucinations, including data augmentation, dynamical systems and regularization techniques and show that data augmentation significantly reduces hallucination frequency.
Finally, we analyze networks that produce hallucinations and show signatures of hallucinations in the attention matrix and in the stability measures of the decoder.
Neural machine translation (NMT) systems are language translation systems based on deep learning architectures BID10 BID1 BID31 .
In the past few years, NMT has vastly improved and has been deployed in production systems, for example at Google BID33 , Facebook BID15 , Microsoft BID17 , and many others.
As NMT systems are built on deep learning methodology, they exhibit both the strengths and weaknesses of the approach.
For example, NMT systems are competitive with state of the art performance BID6 and scale well to very large datasets BID23 but like most large deep learning systems, NMT systems are poorly understood.
For example, in many commercial translation systems, entering repeated words many times occasionally results in strange translations, a phenomenon which has been highly publicized BID12 .
More broadly, recent work shows that NMT systems are highly sensitive to noise in the input tokens BID3 and also susceptible to adversarial inputs BID9 .
When there is an error in translation, it can be challenging to either understand why the mistake occurred or engineer a fix.Here we continue the study of noise in the input sequence and describe a type of phenomenon that is particularly pernicious, whereby inserting a single additional input token into the source sequence can completely divorce the translation from the input sentence.
For example, here is a German input sentence translated to English (reference) by a small NMT system: Source: Caldiero sprach mit E!
Nachrichten nach dem hart erkämpften Sieg, noch immer unter dem Schocküber den Gewinn des Großen Preises von 1 Million $.
Reference: Caldiero spoke with E!
News after the hard-fought victory, still in shock about winning the $1 million grand prize.
NMT Translation: Caldiero spoke with E, after the hard won victory, still under the shock of the winning of the Grand Prix of 1 million $.
In this paper we uncovered and studied a hallucination-like phenomenon whereby adding a single additional token into the input sequence causes complete mistranslation.
We showed that hallucinations are common in the NMT architecture we examined, as well as in its variants.
We note that hallucinations appear to be model specific.
We showed that the attention matrices associated with hallucinations were statistically different on average than those associated with input sentences that could not be perturbed.
Finally we proposed a few methods to reduce the occurrence of hallucinations.Our model has two differences from production systems.
For practical reasons we studied a small model and used a limited amount of training data.
Given these differences it is likely that our model shows more hallucinations than a quality production model.
However, given news reports of strange translations in popular public translation systems BID12 , the dynamical nature of the phenomenon, the fact that input datasets are noisy and finite, and that our most effective technique for preventing hallucinations is a data augmentation technique that requires knowledge of hallucinations, it would be surprising to discover that hallucinations did not occur in production systems.While it is not entirely clear what should happen when a perturbing input token is added to an input source sequence, it seems clear that having an utterly incorrect translation is not desirable.
This phenomenon appeared to us like a dynamical problem.
Here are two speculative hypotheses: perhaps a small problem in the decoder is amplified via iteration into a much larger problem.
Alternatively, perhaps the perturbing token places the decoder state in a poorly trained part of state space, the dynamics jump around wildly for while until an essentially random well-trodden stable trajectory is found, producing the remaining intelligible sentence fragment.Many of our results can be interpreted from the vantage of dynamical systems as well.
For example, we note that the NMT networks using CFN recurrent modules were highly susceptible to perturbations in our experiments.
This result highlights the difficulty of understanding or fixing problems in recurrent networks.
Because the CFN is embedded in a larger graph that contains an auto-regressive loop, there is no guarantee that the chaos-free property of the CFN will transfer to the larger graph.
The techniques we used to reduce hallucinations can also be interpreted as dynamical regularization.
For example, L2 weight decay is often discussed in the context of generalization.
However, for RNNs L2 regularization can also be thought of as dynamically conditioning a network to be more stable.
L2 regularization of input embeddings likely means that rare tokens will have optimization pressure to reduce the norm of those embeddings.
Thus, when rare tokens are inserted into an input token sequence, the effects may be reduced.
Even the data augmentation technique appears to have stability effects, as Appendix 10 shows the overall stability exponents are reduced when data augmentation is used.Given our experimental results, do we have any recommendations for those that engineer and maintain production NMT systems?
Production models should be tested for hallucinations, and when possible, the attention matrices and hidden states of the decoder should be monitored.
Our results on reducing hallucinations suggest that standard regularization techniques such as Dropout and L2 weight decay on the embeddings are important.
Further, data augmentation seems critical and we recommend inserting randomly chosen perturbative tokens in the input sentence as a part of the standard training regime (while monitoring that the BLEU score does not fall).
We note a downside of data augmentation is that, to some extent, it requires knowing the types of the pathological phenomenon one desires to train against.
Figure 7 : Schematic of the NMT decoder.
The input sequence, x 1:S , is encoded by a bidirectional encoder (not shown) into a sequence of encodings, z 1:S .
The attention network, f att , computes a weighted sum of these encodings (computed weights not shown), based on conditioning information from h and provides the weighted encoding to the 2-layer decoder, f dec , as indicated by the arrows.
The decoder proceeds forward in time producing the translation one step at a time.
As the decoder proceeds forward, it interacts with both the attention network and also receives as input the decoded output symbol from the previous time step.

assistant: text: We introduce and analyze the phenomenon of "hallucinations" in NMT, or spurious translations unrelated to source text, and propose methods to reduce its frequency.