SlowGuess/NAACL_2021_OCR huggingface dataset:
Figure 5: Frequency for the Top 5 Bigrams in the Output for the IRS for the different noisy models on IWSLT 2014 En-De (Table 6).
Figure 6: Attention Visualization for translation of source in Figure 4, on which the UR model hallucinates (right), compared against the model with no noise (left). The right attention map displays the characteristic hallucination pattern (Lee et al., 2018; Berard et al., 2019; Voita et al., 2020). The source is not present in the training corpus for the UR model. However, source sequences from the same domain (WMT corpus) are present with invalid references.
2. When we consider the IRS-BLEU, we find that the RR model has fully memorized this data. This is to be expected as it has seen this set repeated 1000 times.
3. On the IRS set, the UR model produces a num
ber of repeated outputs (IRS Repeats) from the training corpus.
4. On the IRS set, the RU model produces a very high percentage of oscillatory hallucinations (OH).
Linking Hallucination Patterns to Noise Patterns The main purpose of the above experiments is to demonstrate how natural hallucinations can be generated on source sequences seen or unseen during training, and their relation to specific noise types. The link between noise patterns and specific types of hallucinations in the output could be used as very effective diagnostic tool to trace hallucinated outputs to corpus-level noise, with the goal of removing the noise from the training dataset.
In this regard, two important observations further emerge from Tables 5 and 6. First, that in the case of UR noise, a considerable percentage of natural hallucinations (IRS NH) manifests as a direct copy of a training reference (without any of the IRS source sequences being present in the training set). Second, for the case of RU noise, oscillatory hallucinations (OH) are very prominent, as evident by the number IRS Unique-Bigrams, which are considerably lower when compared to the other noise types. Figure 5 presents the comparisons for counts of the top 5 bigrams present in the translations of the IRS set, showing how among the 4 noise patterns, RU leads to the most oscillatory hallucinations. Resulting sets of translations for a source sequence present in the IRS is shown in Figure 4, while Figure 6 presents a qualitative comparison of the attention patterns for this source sequence.
5 Hallucination Amplification
In this section, we analyze how hallucinations caused due to corpus-level noise get amplified when a model trained on a noisy MT corpus is used for downstream data generation in algorithms such as Sequence-level Knowledge Distillation (KD) (Kim and Rush, 2016) and Backtranslation (BT) (Edunov et al., 2018). To analyze this, we need to compute NH at scale. So, firstly, we propose an automatic NH detection algorithm based on the analysis that hallucinations often occur in terms of oscillations or repeats of the target sequences.
Algorithm 3: Corpus-level NH Estimator
Data: Source S, Multi-lingual Similarity Scoring Model X, NMT Model M, Noise Estimate  $\epsilon$ , N-gram order  $n$ , Threshold  $t$
Result: Approximate Natural Hallucinations ANH
$T =$  Decode the Source Sequences in  $S$
$S_{X} =$  Compute Cross-Linguial Similarity for  $(T,S)$
$F_{1} =$  Select Translations where the count of the top repeated  $n$  -gram in the translation is greater than the count of top repeated source  $n$  -gram by at least  $t$  ..
$F_{2} =$  Select Translations in  $T$  that are Paired with Multiple Unique Sources;
$S_{\epsilon} =$  Bottom  $\epsilon$  percentage of samples in  $S_{X}$
$ANH = (S_{\epsilon}\cap F_{1})\cup (S_{\epsilon}\cap F_{2})$
The proposed NH Estimator (algorithm 3) is reference-free and works at the corpus-level. One simplifying assumption used in algorithm 3 is that the repeats are now computed on the translations generated over the source set rather than on the training set (as in Tables 5 and 6 for the IRS-Repeats metric). The motivation behind this assumption is that given a sufficiently large source set, the translated output (if hallucinated as a direct copy of one of the training set targets), will appear more than once in the decoded set (since UR noise is one of its causes).
We use algorithm 3 to measure NH caused by using the models trained on the noisy corpora (as explored in section 4.2 and analyzed in Tables 5 and 6) for BT and Sequence-level KD. For BT, we use 1 million English sentences from the WMT 17 De-En dataset as the monolingual corpus and generate back-translations via sampling (Edunov et al., 2018), using the different types of noisily trained models (RR, UU, UR, RU) for En-De. For constructing a sequence-level KD dataset we generate the translations over the initial IWSLT 2014 De-En corpus training corpus (the initial parallel
data, with no noise) with a beam size of 5 (Kim and Rush, 2016). The results of applying the NH estimator (with  $\epsilon = 1$ , i.e.  $1\%$ ,  $n = 4$ ,  $t = 2$  and LASER as the cross-lingual similarity scoring Model  $M$  (Artetxe and Schwenk, 2019)) on the outputs generated using KD and BT are presented in Table 7 and Table 8 respectively.
Data-Type	F1	F2	Sc∩F1	Sc∩F2	ANH
Parallel	17	1900	0	41	41
KD-None	17	2310	0	120	120
KD-UU	17	2394	1	118	119
KD-UR	30	3090	0	688	688
KD-RU	24	2365	2	122	124
KD-RR	29	2411	1	135	136
Table 7: Hallucination Amplification for Knowledge Distillation (Kim and Rush, 2016)
Data-Type	F1	F2	Sε∩F1	Sε∩F2	ANH
BT-UU	154	7592	0	5009	5009
BT-UR	43	176184	0	7280	7280
BT-RU	56	67	1	2	3
BT-RR	62	65	1	11	12
Table 8: Hallucination Amplification for Backtranslation (Edunov et al., 2018)
We find that the UR models lead to severe amplifications for both BT and KD. For KD, we find that all noisy models lead to increase in NH when compared to the initial parallel corpus (implying amplification), which itself contains a non-trivial number of repeated targets. For BT, both UU and UR models lead to large numbers of repeated generations. RR models however cause the least hallucinations for both KD and BT. Our proposed NH estimator is not able to detect many OH however, in any of the cases due to very little overlap with the bottom  $\epsilon = 1\%$  similarity scores, even though the  $F_{1}$  column indicates amplification of translations with repeated n-gram patterns  $(F_{1})$  in the KD datasets.
Further, since, there is hallucination amplification going from a parallel corpus to the KD data generated (using noisy models trained on the parallel corpus), downstream systems trained on the KD data will be impacted in terms of hallucinations as well. We leave further downstream analysis to future work.
In this section, we present a qualitative analysis of a few topics discussed in section 4, along with a discussion on some future research directions.
MV	Source	Target
0.61	gerade plus gerade : gerade . ungerade plus ungerade : gerade .	even plus even gives you even . odd plus odd gives you even .
0.65	also ist 2 ∧ 5 : 2 x 2 = 4 , 8 , 16 , 32 .	so 2 ∧ 5 is 2 x 2 = 4 , 8 , 16 , 32 .
0.69	beweg dich ! nein ! beweg dich ! nein ! beweg dich ! nein !	move . no . move . no . move . no .
0.82	f Frau : sie bestanden darauf auf , ich wurde lügen .	they insisted that i was lying .
0.94	mjam , mjam , mjam , mjam , mjam .	gobble , gobble , gobble , gobble , gobble .
Table 9: Examples of Samples from the Top-100 Most Memorized Samples in the Training Set as measured using the Memorization Value (MV) Estimator (Algorithm 1) with chrF as the Metric: De-En.
6.1 Memorized Samples
Table 9 presents some examples from the most memorized training samples, thereby representing the samples from the long-tail of the data that is likely to have been memorized by the model. Qualitatively, the examples appear to be different (in terms of source/target syntax) from a random subset of training samples (e.g. in Appendix A, Table 10), although we leave further quantitative analysis of the differences to future work. Similarly, the link between out-of-domain and memorized samples needs to be ascertained quantitatively.
6.2 Preventing Hallucinations
In this subsection, we discuss a few methods that could be effective in preventing hallucinations.
Data-Augmentation To prevent hallucinations under perturbation resulting from memorization of the samples in the long-tail of the dataset (Feldman, 2020), a simple iterative solution could be to analyze the long-tail (using Algorithm 1), and implement data-augmentations specific to the characteristics of such samples (e.g. as in Table 9), with the goal of bringing such samples out of the long-tail (Raunak et al., 2020). Further work is required to determine the dynamics of such transition.
Ameliorating Memorization During Learning Robust learning algorithms e.g. Robust Early learning (Xia et al., 2021) that are designed to prevent memorization specifically are likely to prevent perturbation based hallucinations.
Robust Learning on Noisy Samples Kang and Hashimoto (2020) propose a loss-truncation approach to reduce the impact of noisy references in sequence-to-sequence training, using the intermediate model's loss as a sample quality estimator and test their algorithm on a summarization task. Li et al. (2021) present a modification to Expected Risk Minimization (ERM), namely Tilted-ERM to reduce the impact of outliers during training. Such techniques could be useful in increasing learning robustness to corpus-level noise in NMT as well.
Corpus-Level Filtering Incorporating heuristics or filters (Junczys-Dowmunt, 2018; Zhang et al., 2020) to remove invalid source-target pairs, especially the noise patterns explored in section 4.2 (or to remove bitext indeterminacy in general) could be effective in reducing natural hallucinations.
In this work we demonstrated that memorized training samples are far more likely to hallucinate under perturbation than non-memorized samples, under an extension of the Memory Value Estimator proposed in Feldman and Zhang (2020). We also showed that specific noise patterns in the training corpora lead to specific well-known hallucination patterns. Finally, we demonstrated that these patterns can be amplified by popular data-generation processes such as backtranslation and sequence-level knowledge distillation.
Due to the compute-intensive algorithms involved in our analysis, we conduct most of our experiments using the IWSLT 2014 corpus. However, long-tailed phenomena are a characteristic of natural language and even scaling the size of the corpus doesn't alleviate the characteristic Zipfian distribution of the occurrence of words/tokens in the NMT corpora; which, according to the central thesis of the long-tail theory (Feldman, 2020), would lead to memorizations. Similarly, noise in the form of invalid references is an artifact of the scale at which web-based corpora are collected and given that both hallucinations under perturbations and natural hallucinations are widely reported in large-scale NMT systems, our insights should be directly applicable to larger-scale models as well.
We hope that our work serves as a useful step towards a detailed understanding of hallucinations in NMT and in other sequence to sequence models. Among the numerous interesting directions for follow-up work, in future, we would like to explore learning-centric fixes to ameliorate the impact of memorization and corpus-level noise patterns in NMT training.
Mikel Artetxe and Holger Schwenk. 2019. Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Linguial Transfer and Beyond. Transactions of the Association for Computational Linguistics, 7:597-610.
Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65-72, Ann Arbor, Michigan. Association for Computational Linguistics.
Alexandre Berard, Ioan Calapodescu, and Claude Roux. 2019. Naver labs Europe's systems for the WMT19 machine translation robustness task. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 526-532, Florence, Italy. Association for Computational Linguistics.
Yong Cheng, Lu Jiang, and Wolfgang Macherey. 2019. Robust neural machine translation with doubly adversarial inputs. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4324-4333, Florence, Italy. Association for Computational Linguistics.
Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 489–500, Brussels, Belgium. Association for Computational Linguistics.
Vitaly Feldman. 2020. Does learning require memorization? a short tale about a long tail. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, STOC 2020, page 954-959, New York, NY, USA. Association for Computing Machinery.
Vitaly Feldman and Chiyuan Zhang. 2020. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems, 33.
Marcin Junczys-Dowmunt. 2018. Dual conditional cross-entropy filtering of noisy parallel corpora. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 888-895, Belgium, Brussels. Association for Computational Linguistics.
Daniel Kang and Tatsunori Hashimoto. 2020. Improved natural language generation via loss truncation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 718-731, Online. Association for Computational Linguistics.
Yoon Kim and Alexander M. Rush. 2016. Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1317-1327, Austin, Texas. Association for Computational Linguistics.
Philipp Koehn and Rebecca Knowles. 2017. Six challenges for neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation, pages 28-39, Vancouver. Association for Computational Linguistics.
Katherine Lee, Orhan First, Ashish Agarwal, Clara Fannjiang, and David Sussillo. 2018. Hallucinations in neural machine translation. In NIPS 2018 Interpretability and Robustness for Audio, Speech and Language Workshop.
Tian Li, Ahmad Beirami, Maziar Sanjabi, and Virginia Smith. 2021. Tilted empirical risk minimization. In International Conference on Learning Representations.
Marianna Martindale, Marine Carpuat, Kevin Duh, and Paul McNamee. 2019. Identifying fluently inadequate output in neural and statistical machine translation. In Proceedings of Machine Translation Summit XVII Volume 1: Research Track, pages 233-243, Dublin, Ireland. European Association for Machine Translation.
Mathias Müller, Annette Rios, and Rico Sennrich. 2020. Domain robustness in neural machine translation. In Proceedings of the 14th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track), pages 151-164, Virtual. Association for Machine Translation in the Americas.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. *fairseq: A fast, extensible toolkit for sequence modeling.* In *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)*, pages 48-53, Minneapolis, Minnesota. Association for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Maja Popovic. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal. Association for Computational Linguistics.
Vikas Raunak, Siddharth Dalmia, Vivek Gupta, and Florian Metze. 2020. On long-tailed phenomena in neural machine translation. In *Findings of the Association for Computational Linguistics: EMNLP*
2020, pages 3088-3095, Online. Association for Computational Linguistics.
Holger Schwenk. 2018. Filtering and mining parallel data in a joint multilingual space. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 228-234, Melbourne, Australia. Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715-1725, Berlin, Germany. Association for Computational Linguistics.
Zhaopeng Tu, Zhengdong Lu, Yang Liu, Xiaohua Liu, and Hang Li. 2016. Modeling coverage for neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 76-85, Berlin, Germany. Association for Computational Linguistics.
Arata Ugawa, Akihiro Tamura, Takashi Ninomiya, Hiroya Takamura, and Manabu Okumura. 2018. Neural machine translation incorporating named entity. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3240-3250, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Elena Voita, Rico Sennrich, and Ivan Titov. 2020. Analyzing the source and target contributions to predictions in neural machine translation. arXiv preprint arXiv:2010.10907.
Chaojun Wang and Rico Sennrich. 2020. On exposure bias, hallucination and domain shift in neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3544-3552, Online. Association for Computational Linguistics.
Xiaobo Xia, Tongliang Liu, Bo Han, Chen Gong, Nannan Wang, Zongyuan Ge, and Yi Chang. 2021. Robust early-learning: Hinding the memorization of noisy labels. In International Conference on Learning Representations.
Boliang Zhang, Ajay Nagesh, and Kevin Knight. 2020. Parallel corpus filtering via pre-trained language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8545-8554, Online. Association for Computational Linguistics.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations.
Table 10 presents some random samples from the lower-end of the memorization values. The sentences differ in terms of their syntactic properties (versus Table 9 in section 6.1), although more analysis is required to quantitatively ascertain the differences.
Table 10: Examples of Samples with Memorization Value (MV) between 0.0 and 0.6 (Below Top-100) from the Training Set as measured using the Memorization Value (MV) Estimator (Algorithm 1) with chrF as the Metric: De-En.
MV	Source	Target
0.0	sie haben das gleiche gehirn, und das gleiche eeg .	they have the same brain, and the same eeg .
0.08	und wir hatten keine ahnung, wo er war .	it looks like the technological problem is solved .
0.25	also , die wiederbesiedlung ist wirklich sehr langsam .	so , recolonization is really very slow .
0.30	zum einen waren bereits sehr real .	for one thing, both were very real .
0.50	wir überqueren auf alu-leitern mit angeführten sichereits-seilen .	we cross on aluminum ladders with safety ropes attached .