RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Introduction
Hallucinations are a long-standing well-known problem in machine translation (MT) Koehn and Knowles ( 2017 ) and natural language generation Ji et al. ( 2023 ) .
While there has been extensive research on their identification and mitigation (Lee et al., 2019 ; Raunak et al., 2021 ; Mohammadshahi et al., 2022b ; Guerreiro et al., 2023a ; Dale et al., 2023 , among others) , they still persist as an issue, especially in low-resource settings.

Contrastive conditioning has previously been used for analysing specific translation errors such as disambiguation errors and undertranslation Vamvas and Sennrich ( 2021 , 2022 ) . The main idea is that translations that are equally or more probable given some corrupted source than the true source
are likely to be erroneous with respect to the corrupted span.
We can apply the same intuition to hallucinations and translations into the wrong language, so called off-target translations: if hallucinations are detached from the source, they should have a similar probability given the true source and given a random other source.
A translation in the wrong language should have a similar or higher probability if that language is marked as desired.

Inspired by this, we design decoding objectives that do not just search for the most probable translation, but search for a translation that maximizes the probability given the true input, but minimizes the probability given one or several contrastive inputs.

This paper makes the following contributions:

We introduce contrastive decoding objectives to address two problems often observed in MT: hallucinations and off-target translations.

By evaluating two massively multilingual MT models, M2M-100 (418M) and SMaLL-100, across 57 mostly low-resource translation directions, we show improvements in chrF2 by 1.3–1.7 points, and reduce the number of translations with chrF2 below 10 by 67-83%.

Finally, we provide a proof of concept for applying our approach to LLM-based translation, where off-target issues are common.
Method
To suppress hallucinations, we pair each input X with a randomly selected input segment X^{\prime} . 2 2 2 In practice, by shuffling segments of the input document. Rather than finding a translation that maximizes p(Y|X) , we search for one that both maximizes p(Y|X) and minimizes p(Y|X^{\prime}) . We add a hyperparameter \lambda to control the strength of this contrastive penalty, yielding Eq. 1 .

 s(Y,X)=\sum_{i=1}^{|Y|}-\log\biggl{(} p(y_{i}|y_{<i},X) -\lambda p(y_{i}|y_{<i},X^{\prime})\biggl{)} (1)

We denote this source-contrastive decoding .

Off-target translations are a common failure mode in multilingual MT systems Arivazhagan et al. ( 2019 ) .
They have been linked to the predominance of English in the training of multilingual systems Rios et al. ( 2020 ) .
Production of text in the source language, often a copy of the input, is connected to the occurrence of copying in the training data, and the high probability of continuing to copy once a copy has been started Ott et al. ( 2018 ) .

The majority of multilingual MT systems use special tokens to indicate the target language, following Johnson et al. ( 2017 ) . 3 3 3 The indicator token can be in the source (SMaLL-100), or output-initial and force-decoded (M2M-100). To penalize output in the wrong language, we can add contrastive inputs that only vary the language indicator token.

Let l_{y} be the target language.
We replace its indicator token with contrastive variants l_{y^{\prime}}\in L_{c} for languages we wish to suppress.
Based on the predominant off-target languages in multilingual MT Arivazhagan et al. ( 2019 ) , our set of contrastive languages L_{c} consists of English 4 4 4 Unless English is the target language. and the respective source language.
This results in Eq. 2 .

 s(Y,X)=\sum_{i=1}^{|Y|}-\log\biggl{(} p(y_{i}|y_{<i},X,l_{y}) -\sum_{l_{y^{\prime}}\in L_{c}}\lambda p(y_{i}|y_{<i},X,l_{y^{\prime}})\biggl{)} (2)

We refer to decoding with contrastive translation directions as language-contrastive decoding .
We can combine source-contrastive and language-contrastive decoding by summing all contrastive variants, and refer to the weights as \lambda_{\text{src}} and \lambda_{\text{lang}} .
Data and Models
We perform experiments with two massively multilingual MT models: M2M-100 (418M) Fan et al. ( 2021 ) , and SMaLL-100 Mohammadshahi et al. ( 2022a ) , a distilled version of M2M-100 (12B).

We use beam size 5.
We perform minimal hyper-parameter tuning on the ps-ast translation direction with M2M-100 and set \lambda_{\text{src}} to 0.7 . 5 5 5 We exclude ps-ast from average results reported. Since only a small number of directions suffer from off-target outputs, we do not tune \lambda_{\text{lang}} , setting it to 0.1 .

We test on three sets of translation directions:

the 25 non-English-centric directions used by Guerreiro et al. ( 2023a ) ( HLMT ). These are af-zu, ar-fr, be-ru, cs-sk, de-hr, de-hu, el-tr, fr-sw, hi-bn, hi-mr, hr-cs, hr-hu, hr-sk, hr-sr, it-de, it-fr, nl-de, nl-fr, ro-de, ro-hu, ro-hy, ro-ru, ro-tr, ro-uk, uk-ru. 6 6 6 See Appendix B for full language names.

29 directions between 5 low-resource languages from different branches of Indo-European, plus Zulu from the Atlantic-Congo family ( X-branch ): af, ast, hr, ps, ur, zu.

4 high-resource translation directions: en-de, de-en, en-fr, fr-en ( high-res ).

We also report results for the union of the sets ( all ).

We evaluate with spBLEU Goyal et al. ( 2022 ) and chrF2 Popović ( 2015 ) using sacreBLEU Post ( 2018 ) 7 7 7 BLEU—#:1—c:mixed—e:no—tok:flores101—s:exp—v:2.3.1 chrF2—#:1—c:mixed—e:yes—nc:6—nw:0—s:no—v:2.3.1 on the Flores-101 devtest set Goyal et al. ( 2022 ) .
We use OpenLID Burchell et al. ( 2023 ) for language identification to measure off-target rates.
To quantify the number of hallucinations, we employ a rough approximation following Lee et al. ( 2019 ); Müller and Sennrich ( 2021 ) , counting the proportion of segments with chrF2 <10 . 8 8 8 Müller and Sennrich ( 2021 ) report a threshold of 1, but this is a typo (personal communication). This method does not distinguish between hallucinations and off-target translations. Another automatic metric specific for oscillatory hallucinations is top n-gram (TNG) Guerreiro et al. ( 2023b ); Raunak et al. ( 2022 , 2021 ) ,
which measures the number of sentences whose top repeating n -gram is more frequent than the top repeated source n -gram by at least t . 9 9 9 We follow Guerreiro et al. ( 2023b ) and use n=4 and t=2 .
Results
We report results using source-contrastive decoding ( C_{src} ), and combining source-contrastive and language-contrastive decoding ( C_{src+lang} ) in Table 1 . 10 10 10 See Appendix A for full results. Across 57 translation directions, chrF2 improves by 1.3 (M2M-100) and 1.1 (SMaLL-100) points with source-contrastive decoding.
Language-contrastive decoding brings additional gains of 0.4 (M2M-100) and 0.2 (SMaLL-100) points.

Improvements are more modest when measured with spBLEU (0.2 on M2M-100; 0.3 on SMaLL-100).
We notice that hallucinations tend to be overlong, and can perversely improve BLEU by reducing the brevity penalty.
We thus consider chrF2, which pairs precision with recall instead of a simplistic brevity penalty, to be our primary metric.

Off-target translations are relatively rare for the translation directions tested, especially for SMaLL-100 (see Table 2 ).
With M2M-100, the highest proportion of English outputs in the baseline was detected for af-zu (9.1%), the highest percentage of outputs in the source language for hr-sr (4.2%) 11 11 11 This number may be an overestimate due to the close relationship between Serbian and Croatian, and the consequent difficulty of doing reliable language identification. .
These are also among the translation directions that benefit the most from language-contrastive decoding: chrF2 increases by 2.3 for hr-sr 12 12 12 This improvement is somewhat coincidental because both Latin and Cyrillic are accepted for Serbian, but Flores-101 has Cyrillic references. Penalizing output in Croatian, which uses the Latin alphabet, indirectly rewards output in Cyrillic. , and by 2 for af-zu.
However, we observe the largest increase in chrF2 (3.2) for ast-zu, a direction where source-contrastive decoding increases off-target outputs, and where the English output rate goes from 5.5% (baseline) to 9.9% ( C_{src} ) to 2.7% ( C_{src+lang} ).

The proportion of translations with chrF2 below 10 is shown in Table 3 .
We observe large reductions in the number of defect translations, with a reduction from 7.3% to 1.2% (-83%) for M2M-100, and from 5.6% to 1.8% (-67%) for SMaLL-100.
When focusing on oscillatory hallucinations according to TNG in Table 4 , the improvement is even more pronounced,
with a reduction from 9.3% to 0.7% (-92%) for M2M-100, and from 5.9% to 1.5% (-75%) for SMaLL-100.
Ablation Studies
The fact that we pick contrastive inputs from the test sets at random raises a few questions about this approximation.
We repeated the translation with M2M-100 across all 57 translation directions 3 times and find that the standard deviation is minimal (0.0107 for chrF2).
Using a single random input as a contrastive variant is a heavy approximation, but our ablation study in Table 5 shows that this yields the majority of the performance gains, and using up to 3 inputs as contrastive examples 13 13 13 we divide \lambda_{\text{src}} by the number of contrastive inputs. only yields an additional 0.1 point improvement in chrF2.
Application to Large Language Models
In this section, we demonstrate that our method can be applied to large language models (LLM).
Previous work has achieved competitive translation quality for some directions by prompting models such as PaLM Vilar et al. ( 2023 ); Garcia et al. ( 2023 ) , GPT Hendy et al. ( 2023 ) or BLOOM Bawden and Yvon ( 2023 ) .
However, LLM-based translation is still prone to hallucination and off-target translation Zhang et al. ( 2023 ); Guerreiro et al. ( 2023a ) .

Our demonstration is based on the Llama 2 model family Touvron et al. ( 2023 ) and specifically the instruction-tuned version ( Llama Chat ), exploiting the fact that MT examples were among the data used for instruction tuning Wei et al. ( 2022 ); Chung et al. ( 2022 ) .
We generate translations by instructing the model to translate a segment into a given language, force-decoding the line “Sure, here’s the translation:” , and then decoding until the next line break.
The prompt we used is detailed in Appendix E .

When using this simple prompting approach in the en–de direction, we find that off-target output in English is very common.
Moreover, providing a 1-shot example in the prompt, while improving translation quality, does not prevent the off-target issue.
We thus apply language-contrastive decoding and add a contrastive prompt that instructs the model to “translate” into English instead of German.
The decoding objective is analogous to Eq. 2 .
We use 4-bit precision (Dettmers and Zettlemoyer, 2023 ) and greedy decoding.

Figure 2 shows the percentage of off-target output for different \lambda_{\text{lang}} .
Generally, we observe that the off-target rate falls with increasing \lambda_{\text{lang}} , demonstrating the effectiveness of our method with LLM prompting.
English–French (Appendix C ) has similar results.
In terms of overall translation quality, we find that language-contrastive decoding improves chrF2 and spBLEU and only becomes detrimental for \lambda_{\text{lang}}>0.7 (Appendix D ).
Hallucination Detection and Reduction
Various methods have been proposed to detect hallucinations, including identifying typical patterns in the output Raunak et al. ( 2021 ) , using internal information like attention patterns Lee et al. ( 2019 ) or the contribution of the source to predictions Dale et al. ( 2023 ) , or measures of decoder confidence, including the output probability Guerreiro et al. ( 2023b ) or stability of samples under perturbation Lee et al. ( 2019 ); Guerreiro et al. ( 2023b ) .

Hallucination mitigation is more difficult, especially if we assume that models are already trained with best practices, and focus on training-free methods.
Several studies use external models for mitigation, e.g. using other translation models as a fall-back Guerreiro et al. ( 2023a ) , or sample reranking based on quality estimation (QE) models Guerreiro et al. ( 2023b ) .
Our method has the advantage of not requiring external models, and we note that modern QE metrics are themselves prone to score certain hallucinations highly Freitag et al. ( 2022 ); Yan et al. ( 2023 ) .

Mitigation methods that do not rely on external models are typically sampling-based. Guerreiro et al. ( 2023b ) report that even the translation model’s own sequence probability can be used for sample reranking.
A consensus translation can be identified via sampling-based Minimum Bayes Risk decoding Eikema and Aziz ( 2020 ) , which benefits from the fact that hallucinations are dissimilar from each other Müller and Sennrich ( 2021 ) .
Contrastive Decoding
Contrastive decoding is similar to contrastive learning (e.g. Hadsell et al., 2006 ; Socher et al., 2014 ; Gao et al., 2021 ) in that positive and negative examples are contrasted, but involves no training.

Li et al. ( 2023 ) introduce a form of contrastive decoding that contrasts the probability between different models, whereas our methods work with a single model, contrasting inputs. Su and Collier ( 2023 ) introduce a contrastive search where potential output tokens are compared to previous tokens, penalizing outputs that are similar to the context and thus suppressing repetition patterns.

Source-contrastive decoding can also be seen as a variant of implicit language model (ILM) compensation, mirroring recent work by Herold et al. ( 2023 ) .
Our work is different in motivation in that ILM is typically used to allow the inclusion of an external LM, where we show the effectiveness of simply suppressing the ILM.
Also, we show the effectiveness of a different, simple approximation.

Finally, language-contrastive decoding bears resemblance to negative prompting, a technique used to suppress concepts in image generation.
Conclusion
This paper shows that certain failure modes of MT can be addressed by contrastive decoding objectives that use pairs or sets of inputs for the prediction.
Specific contrastive inputs address specific errors, and we introduce strategies to mitigate hallucinations and off-target translation.

Future work could expand on our work by exploring if other MT failure modes can be mitigated with appropriate contrastive inputs, or if other forms of control can be improved.
For example, for models that use domain indicator tokens Kobus et al. ( 2017 ) , we could perform domain-contrastive decoding and achieve stronger domain control.
Beyond MT, we expect that source-contrastive decoding can also be useful for other tasks, e.g. to penalize over-generic responses in dialogue systems.
Limitations
We only tested language-contrastive decoding in multilingual models that control the target language via language indicator tokens.
It is possible to apply the same strategy to modular architectures that use language-specific components Firat et al. ( 2016 ); Vázquez et al. ( 2019 ); Bapna and Firat ( 2019 ) , but its effectiveness remains to be tested.
For bilingual translation models that suffer from off-target translations, e.g. because of noisy training data Khayrallah and Koehn ( 2018 ) , we would need bilingual models for other translation directions to implement language-contrastive decoding, but this sacrifices the main strength of our approach: not relying on external models.

We perform minimal hyperparameter tuning for \lambda_{\text{src}} , and did not tune \lambda_{\text{lang}} .
Using the same hyperparameters across translation directions and translation models results in performance degradations in some cases, most noticeably for high-resource translation directions.
We consider it a positive result that we obtain improvements on average with minimal hyperparameter tuning, but future work may wish to use more complex strategies to weight (or disable) contrastive variants across translation directions.
Ethics Statement
This paper introduces new decoding objectives for machine translation, and we do not foresee any harms being caused by source-contrastive or language-contrastive decoding.
More widely, we are interested in exploring novel contrastive inputs for risk mitigation, e.g. for model debiasing, but certain contrastive inputs could also have undesirable consequences, e.g. increasing model bias.
Acknowledgements
We thank the anonymous reviewers for their comments.
This work was funded by the Swiss National Science Foundation (project MUTAMUR; no. 213976).
Zero-shot


 
 You are a machine translation system that translates sentences from English to German. You just respond with the translation, without any additional comments. 
 
 "We now have 4-month-old mice that are non-diabetic that used to be diabetic," he added. Translate to German 
 Sure, here’s the translation:
One-shot


 
 You are a machine translation system that translates sentences from English to German. You just respond with the translation, without any additional comments. Example instruction: On Monday, scientists from the Stanford University School of Medicine announced the invention of a new diagnostic tool that can sort cells by type: a tiny printable chip that can be manufactured using standard inkjet printers for possibly about one U.S. cent each. Translate to German Example response: Sure, here’s the translation: Am Montag haben die Wisenschaftler der Stanford University School of Medicine die Erfindung eines neuen Diagnosetools bekanntgegeben, mit dem Zellen nach ihrem Typ sortiert werden können: ein winziger, ausdruckbarer Chip, der für jeweils etwa einen US-Cent mit Standard-Tintenstrahldruckern hergestellt werden kann. 
 
 "We now have 4-month-old mice that are non-diabetic that used to be diabetic," he added. Translate to German 
 Sure, here’s the translation:
Figure 1: Our decoding objective yields a translation that is probable given the actual input, but improbable given a source-contrastive or language-contrastive input.; Our decoding objective yields a translation that is probable given the actual input, but improbable given a source-contrastive or language-contrastive input.
Figure 2: Off-target translation rate for Llama 2 Chat models when translating the English Flores-101 devtest set into German.
Language-contrastive decoding tends to reduce off-target translation as \lambda_{\text{lang}} is increased.; Off-target translation rate for Llama 2 Chat models when translating the English Flores-101 devtest set into German.
Language-contrastive decoding tends to reduce off-target translation as \lambda_{\text{lang}} is increased.
Figure 3: Off-target translation rate for Llama 2 Chat models when translating the English Flores-101 devtest set into French.
As with German (Figure 2 ), language-contrastive decoding tends to reduce off-target translation as \lambda_{\text{lang}} is increased.; Off-target translation rate for Llama 2 Chat models when translating the English Flores-101 devtest set into French.
As with German (Figure 2 ), language-contrastive decoding tends to reduce off-target translation as \lambda_{\text{lang}} is increased.
Automatic evaluation results. Averages over different sets of translation directions.
Number of off-target outputs (out of 57684), in English (EN) or the source language (SRC).
Proportion of translations with chrF2 <10 .
Proportion of translations with oscillatory hallucinations according to TNG.
Ablation results for M2M-100 with different numbers of source-contrastive inputs. Average over all languages reported.
List of languages in our experiments, sorted by ISO 639-1 language code.