Language-Informed Beam Search Decoding for Multilingual Machine Translation (2024)

Abstract: Beam search decoding is the de-facto method for decoding auto-regressive Neural Machine Translation (NMT) models, including multilingual NMT where the target language is specified as an input. However, decoding multilingual NMT models commonly produces “offtarget” translations – yielding translation outputs not in the intended language. In this paper, we first conduct an error analysis of off-target translations for a strong multilingual NMT model and identify how these decodings are produced during beam search. We then propose Language-informed Beam Search (LiBS), a general decoding algorithm incorporating an off-the-shelf Language Identification (LiD) model into beam search decoding to reduce off-target translations. LiBS is an inferencetime procedure that is NMT-model agnostic and does not require any additional parallel data. Results show that our proposed LiBS algorithm on average improves +1.1 BLEU and +0.9 BLEU on WMT and OPUS datasets, and reduces off-target rates from 22.9% to 7.7% and 65.8% to 25.3% respectively.1

heading: 1 Motivation
text: With Neural Machine Translation (NMT) (Bahdanau et al., 2014; Vaswani et al., 2017) becoming the state-of-the-art approach in the bilingual Machine Translation literature, Multilingual Neural Machine Translation (MNMT) has attracted much attention (Johnson et al., 2017). MNMT has two main advantages: a) it enables one model to translate between multiple language pairs and thus reduces the model and deployment complexity from O(N2) to O(1), and b) it enables transfer learning between high-resource and low-resource languages. One attractive feature of such transfer learning is zero-shot translation, where the multilingual model
∗Work done mostly at Oregon State University. 1https://github.com/yilinyang7/
fairseq_multi_fix
is able to translate between language pairs unseen during training. For example, after training from French to English and English to German MT data, the model could directly translate French to German.
Despite the theoretical benefits, recent studies have found an overwhelming amount of offtarget translation especially for the zero-shot directions (Zhang et al., 2020; Yang et al., 2021), where the translation is not in the intended language. Existing methods all aim to mitigate off-targets during training. Gu et al. (2019); Zhang et al. (2020) apply Back Translation (BT) to generate synthetic training data for the zero-shot pairs. Yang et al. (2021) introduces a language prediction loss and regularizes the training gradients with a held-out oracle set. Yet, none of the previous work has investigated the off-target issue at decoding time, i.e. how off -target translations emerge and come to outscore on-target translations during beam search decoding.
In this work, we first examine when and how off-target translation emerges during beam search decoding, and then propose Language-informed Beam Search (LiBS), a general algorithm to reduce off-target generation during beam search decoding by incorporating an off-the-shelf Language Identification (LiD) model. Our experiment results on two large-scale popular MNMT datasets (i.e. WMT and OPUS) demonstrate the effectiveness of LiBS in both reducing off-target rates and improving general translation performance. On average LiBS reduces off-target rates from 22.9% to 7.7% and 65.8% to 25.3% on WMT and OPUS respectively, which translates to +1.1 BLEU and +0.9 BLEU overall quality improvement. Moreover, LiBS can be added post-hoc to reduce off-target translation of any existing multilingual model without requiring any additional data or training.
15761

heading: 2 Experiment Setup
text: In this section, we illustrate the data and model setup we used, and the experimental results of our Language-informed Beam Search algorithm. Since we re-use the exact same model and data setup as Wang et al. (2020); Yang et al. (2021), and we will provide a brief summary and direct readers to those works for more details.

heading: 2.1 Dataset
text: Following (Wang et al., 2020; Yang et al., 2021), we conduct experiments on two widely used largescale MNMT datasets WMT2 and OPUS-1003, where the WMT dataset is concatenated from previous year WMT training data including English and 10 other languages. Since the WMT competition does not come with zero-shot evaluation data, we use the human labeled multi-way aligned test set from (Yang et al., 2021), based on the WMT-19 test set.

heading: 2.2 Model Training and Evaluation
text: For both WMT and OPUS-100, we tokenize the dataset with the SentencePiece model (Kudo and Richardson, 2018) to form a shared vocabulary of 64k tokens. We adopt the Transformer-Big setting (Vaswani et al., 2017) in our experiments on the open-sourced Fairseq codebase4 (Ott et al., 2019). The model is optimized using the Adam optimizer (Kingma and Ba, 2015) with a learning rate of 5×10−4, 4000 warm-up steps, and a total of 50k training steps. The multilingual model is trained on 8 V100 GPUs with a batch size of 8192 tokens and gradient accumulation of 8 steps, which essentially simulates the training on 64 V100 GPUs. To evaluate the baseline model, we employ beam search decoding with a beam size of 5 and a length penalty of 1.0. The BLEU score is then measured by the detokenized case-sensitive SacreBLEU5 (Post, 2018). To demonstrate meaningful quality improvement, we also report COMET score (Rei et al., 2020)6 on the WMT test set.
To evaluate the off-target rates, we borrow the
2Referred to as “WMT-10” in (Wang et al., 2020; Yang et al., 2021), we denoted it as WMT to disambiguate against the WMT 2010 campaign.
3We use the deduplicated version from (Yang et al., 2021). 4https://github.com/facebookresearch/
fairseq 5BLEU+case.mixed+lang.srctgt+numrefs.1+smooth.exp+tok.13a+version.1.4.14 6We used the default “Unbabel/wmt22-comet-da” model
off-the-shelf LiD model7 from FastText (Joulin et al., 2016) to detect the language for system translations. Similar to (Yang et al., 2021), we observe an overwhelming off-target rate (averaging 22.9%) across zero-shot pairs on our strong baseline model.

heading: 3 Analyzing Off-Target Occurrence During Beam Search
text: To understand the off-target occurrence during beam search, we analyze the off-target error types on different language pairs, and conduct experiments with varying beam sizes.

heading: 3.1 Multilingual Beam Search Curse
text: The beam search curse phenomenon (Yang et al., 2018) is widely observed in bilingual NMT models. Given a larger beam size, the beam search process would explore a larger search space and choose from a larger candidate pool. Yet empirically, translation performance usually drops significantly with increasing beam sizes. In our study, we also found this phenomenon prevailing in the multilingual system and highly related to the off-target translation error.
As an example, we demonstrate the beam search curse on WMT De→Fr and Cs→De translation, since both are between high-resource languages and with decent translation performance (between 15 to 20 BLEU).
Table 1 illustrates the results on WMT De→Fr and Cs→De. We could clearly observe that the offtarget rate grows sub-linearly with the beam size, and as a result the BLEU score drops significantly with increasing beam sizes. It then raises the curious question of why the off-target rate increases drastically with larger beam sizes, and whether the
7https://dl.fbaipublicfiles.com/fasttext/supervisedmodels/lid.176.bin
performance drop (i.e. BLEU decrease) is mainly due to the off-target errors.

heading: 3.2 Off-Target Error Analysis
text: As part of a detailed analysis, we study the offtarget error type between six zero-shot pairs (i.e. 12 translation directions) from the WMT dataset. We categorize the off-target errors into three types: translating into English8, translating into source language, and others.
The detailed off-target error analysis of WMT zero-shot direction is shown in Table 2. We find that even though the off-target error is overwhelming across languages, it could easily be categorized into mostly two types: translating into English and “translating” into source. The “Others” error type only comprises a negligible 1.1% of cases, given the FastText LiD model has an error margin of 0.81% (Yang et al., 2021).
“→Source” errors We hypothesize that this error is related to the previously studied “source copying” behavior (Ott et al., 2018) on the bilingual NMT model. We then sample three cases from this error type (shown in Table 3). The case study confirms that the “→Source” error type is the same as source copying behavior on bilingual models for these cases. To quantify the degree of source copying, we run Sentence BLEU evaluation9 between source and system translation on WMT Fr→De
8English is never the correct target language in our 12 studied translation directions.
9We use the sentence_bleu function from (Post, 2018) with smooth_method=‘floor’: https: //github.com/mjpost/sacrebleu/blob/ master/sacrebleu/compat.py
“→Source” errors. The sentence BLEU distribution is shown in Figure 1 with an average sentence BLEU of 85.3. It clearly demonstrates that the “→Source” error strongly displays a source copying behavior and is somehow promoted by larger beam sizes.
“→English” errors Since none of our evaluated direction includes English as the target language, translating into English is never promoted and always trigger an off-target error. We similarly sampled three “→English” error cases from the WMT Fr→De test set. We also compare them against the real Fr→En translations with the same model and French input. This case study (in Table 4) hints that the “→English” generations from WMT Fr→De are generally similar but slightly worse “English” translations compared to Fr→En. To demonstrate the similarity, we plot the sentence BLEU distribution for all 172 “→English” errors between Fr→De and Fr→En translations in Figure 2. It demonstrates a strong similarity between Fr→De and Fr→En translations, with an average sentence BLEU of 55.9. Since the evaluation data of the WMT corpus is multi-way aligned, we can evaluate the→English translation quality for both Fr→De and Fr→En against the English human references (in Table 5). Results confirm our observation that the “→English” errors are generally poorer English translations.

heading: 3.3 Beam Search Process Analysis
text: To understand how “→English” and “→Source” errors emerge during beam search and why both errors dramatically increase with larger beam sizes,
we investigate the step-by-step beam search process with case studies. Table 10 and 11 illustrates one representative decoding example from the WMT Fr→De test set with b = 5, 20 and French source “Nous avons maintenant une excellente relation. »”. For b = 20, we only print the top-5 beams due to the space limit. From this example, we have a few observations:
• English candidates are live in the early steps (1-3) of b = 5 but tend to be dropped in later time steps. Meanwhile for b = 20, both English and French candidates are kept alive throughout the decoding process: even though they fall out of the top-5 beams at the 4th step, the off-target candidates quickly catch up and are ranked highest by the 7th step.
• Closely observing the winning English candidate of b = 20, we notice it suffers a heavy penalty in the first step (log prob is -3.58), yet all following steps experience small penalties.
• The final English translation by b = 20 is
indeed a “better” candidate with greater probabilities (i.e. model score) compared to the final German translation by b = 5, therefore, if this off-target candidate is retained throughout the process it will naturally win out against all valid on-target translations.
From the above observations, we can try to answer our previous research questions.
RQ1. How do “→English” and “→Source” errors emerge during decoding?
We first observe that both “→English” and “→Source” candidates are easily accessible in the early steps of decoding. Meanwhile, the models place a low probability on decoding the first source or English token, but relatively high transition probabilities for the remaining source or English tokens can result in off-target sentences scoring more highly than on-target.
Direction BLEU chrF2 TER*
Fr→De 26.92 0.572 0.62 Fr→En 34.91 0.611 0.53
With a larger beam size budget, it is more likely to retain off-target candidates in the earlier steps, when they receive heavy early step penalties. Yet since off-target candidates experience fewer penalties in the later steps, they tend to win out over on-target candidates in the long run. We found it to be the general case that the off-target continuations receive a higher probability (less penalty) than the on-target ones, even though the first offtarget token receives a heavy penalty by the model. We hypothesize that it is due to the recency bias and poor calibration, yet it remains an interesting research question for future work.
Possible Solutions Off-target candidate gaining greater model score demonstrates that the model is poorly calibrated, especially for the later steps of autoregressive decoding. Methods with additional
training data (Gu et al., 2019; Zhang et al., 2020) or regularizations (Yang et al., 2021) could alleviate this issue during training with a well-calibrated model. In this work, with the knowledge of how off-target cases emerge during decoding, we attempt to fix this issue solely at the decoding time by improving on beam search algorithm even with a proven poorly calibrated model.

heading: 4 Language-informed Beam Search (LiBS)
text: The standard beam search process originating from the bilingual NMT model is target-languageagnostic and is found to produce an overwhelming number of off-target translations (Zhang et al., 2020; Yang et al., 2021). Yet the target language (i.e. the desired language for generation) is always known during decoding, thus it is straightforward to enforce the desired language to reduce the offtarget rates without any additional training or data. We thus propose Language-informed Beam Search (LiBS), a general decoding algorithm to inform the beam search process of the desired language during decoding.
To inform the beam search process of the desired language, we borrow an off-the-shelf Language Identification (LiD) model to score the running beam search candidates with their probabilities in the correct language. Since the candidates are normally ranked by the NMT model probabilities, we linearly combine the two log probabilities to ideally
Algorithm 1: Langugage-informed Beam Search Input :MNMT model θ, LiD model γ, source sentence x, target language T , beam size b,
pre-select window size w Output :Finished candidate set C← ∅ ▷ Initialize each beam i with BOS symbols and zero score
1 Bi ← {⟨0.0, 
⟩} 2 repeat ▷ Pre-select top-w candidates from each beam i 3 Wi ← topw{⟨s · pθ(y | x,y), y ◦ y⟩ | ⟨s, y⟩∈Bi, y∈V} ▷ Sort all candidates by the linearly combined NMT and LiD log probabilities 4 W← Sort{⟨log s+ α log pγ(T, y), s, y⟩ | ⟨s, y⟩∈ ⋃b i=1Wi} ▷ Store all finished ones from top-b candidates into C 5 C← {⟨s, y⟩ | ⟨s′, s, y⟩ ∈ topb{W}, y|y| = 
} ▷ Store the top-b unfinished candidates into B 6 B← topb{⟨s, y⟩ | ⟨s′, s, y⟩ ∈W, y|y| ̸= 
} 7 until C has b finished candidates (i.e. |C| = b) ▷ Rerank finished candidates by the linearly combined NMT and LiD log probabilities 8 C← Sort{⟨log s+ α log pγ(T, y), y⟩ | ⟨s, y⟩∈C} 9 return C
find the best candidate in the correct language. The detailed algorithm is illustrated in Algorithm 1. For each step, we first pre-select topw candidates from each beam. Then we sort all b · w active candidates by the linearly combined NMT and LiD log probabilities, where we tune the linear coefficient α on the dev set. Same as the Fairseq (Ott et al., 2019) implementation, we only store the finished ones within the top-b candidates, meanwhile save the top-b active candidates into the beam for the next step10. The decoding process stops when we have found b finished candidates, and at the end of the generation, we again rerank all finished candidates with the linearly combined log probabilities.
Design Choice and Speed Concern We only preselect b · w candidates for the LiD scoring, instead of considering all possible continuations, simply because we could not afford to run LiD model on all b · |V| candidates.
Even though in our experiments we only preselect the top-2 continuations from each beam (i.e. w = 2), the major slow down of the LiBS algorithm is still the un-BPE operation and LiD scoring on line 4 of Algorithm 1.
To speed up the LiBS algorithm, we use the FastText LiD model since it is both fast and accurate
10We only store the NMT model score instead of the linearly combined one to avoid overcounting LiD scores.
Model De→Fr Cs→De
BLEU Off-Tgt BLEU Off-Tgt
Baseline 17.3 23.1% 15.4 12.3% +LiBS, α = 0.7 20.6 2.0% 16.1 1.6% +LiBS, α = 0.8 20.7 1.4% 16.2 1.6% +LiBS, α = 0.9 20.7 1.1% 16.2 1.6% +LiBS, α = 1.0 20.7 0.9% 16.2 1.4% +LiBS, α = 1.1 20.7 0.9% 16.2 1.4% +LiBS, α = 1.2 20.7 0.8% 16.2 1.3%
Table 6: Tuning the linear coefficient α on WMT De→Fr and Cs→De.
(in our case on translation prefixes). With its help, LiBS is only 7.5 times slower than the Fairseq beam search decoding on a single CPU and 3.5 times slower with parallelized LiD scoring on 20 CPUs.

heading: 5 Experiment Results
text: To fully verify the performance of the LiBS algorithm, we compare LiBS against the baseline beam search decoding on both WMT and OPUS-100 datasets.

heading: 5.1 WMT Results
text: Tuning the Linear Coefficient α We tune the linear coefficient α on the dev set. As shown in
Of f-T
ar ge
t R at
e
Table 6, any α value from 0.8 to 1.2 performs similarly well. Because the linear coefficient α controls the weight of the LiD model score, as α increases, the off-target rate monotonically drops. We use α = 0.9 for all the experiments on the WMT dataset.
Multilingual Beam Search Curse As illustrated before, the beam search curse exists in Multilingual NMT models predominantly due to the increasing off-target errors with larger beam sizes. As shown in Table 7, LiBS successfully breaks the beam search curse by preventing off-target translations.
Zero-Shot Performance Table 8 illustrates the full results of LiBS on the WMT dataset. On average across all zero-shot directions, LiBS improves +1 BLEU score while reducing the off-target rates from 22.91% to 7.71%. We notice that for many directions the off-target rate is barely around the error margin of the FastText LiD model, which is 0.81% reported from (Yang et al., 2021). It hints that those translation directions do not suffer from off-target errors anymore, and the reported errors are largely due to the LiD model error. Meanwhile, the MNMT model still suffers from a large number of off-target errors, especially on Gu→Tr and Tr→Gu translations, which we hypothesize is due to the extremely low resources for both languages (WMT contains 180K and 80K parallel data for Tr-En and Gu-En respectively.).

heading: 5.2 OPUS-100 Results
text: To verify the effectiveness of our LiBS algorithm, we further compare it against the baseline beam search decoding on the large-scale OPUS-100
dataset, which includes a total of 100 languages. Different from the WMT experiment, we tune and set α = 1.8 for all directions. This is due to the challenging nature of the OPUS-100 dataset that it performs very poorly on the zero-shot directions with a massive amount of off-target translations. A higher α value for LiBS could effectively reduce the off-target rates and improve the translation performance. For example, Figure 3 plots the performance curve on the OPUS-100 Fr→De test set with increasing α values. It clearly shows a larger α value would consistently decrease the offtarget rates and improve the overall performance (i.e. higher BLEU score)11.
Zero-shot translation performance of LiBS on the OPUS-100 dataset is shown in Table 9. Across all directions, LiBS consistently improves an average of +0.9 BLEU and reduces the off-target rates from 65.79% to 25.34%.
Both WMT and OPUS-100 results clearly show our LiBS algorithm notably improves the zero-shot translation performance by significantly reducing the off-target translations.

heading: 6 Related Work
text: Off-Target Translation Off-target translation is a commonly observed failure mode in multilingual NMT models (Arivazhagan et al., 2019), and Rios et al. (2020) has linked it to the predominance of English in the training data of multilingual models. Gu et al. (2019); Zhang et al. (2019); Yang et al. (2021) all observe it under different data settings and propose to mitigate it using additional monolingual data or held-out oracle set. Similarly to our work, Sennrich et al. (2023) proposes to mitigate off-target errors with constrastive decoding, yet their approach usually hurts the translation quality, on average -1.1 BLEU on high resource languages. Our work is the first to study off-target errors during decoding time, specifically how the off-target translations outscore on-target ones over time.
NMT Decoding Since beam search becomes the de-facto method for decoding NMT models (Bahdanau et al., 2014), studies has observed various flaws with it. Koehn and Knowles (2017) observes beam search curse, where the translation quality usually degrades with increasing beam sizes. Yang et al. (2018); Stahlberg and Byrne (2019) observe length bias, where the model heavily prefers shorter
11The flat BLEU curve is due to the one decimal digit precision of sacreBLEU evaluation.
candidates. To address those issues, extensive work has proposed sampling-based decoding algorithms, where the most popular one is Minimum Bayes Risk (MBR) decoding (Eikema and Aziz, 2020). Yet, MBR decoding suffers severely from the quadratic complexity thus a slow inference speed. Another line of research adopts external Language models to NMT beam search. Yet this external LM usually interferes with NMT’s internal LM (iLM). However, with iLM neutralization, this approach still lags behind leveraging the additional monolingual data through back-translation (Herold et al., 2023). Most similarly to our work, He et al. (2017); Ren et al. (2017) propose to incorporate a trained Value network during beam search decoding to improve the image-captioning task. Our work instead attempts to mitigate off-target translation errors with a small off-the-shelf LiD model, while keeping the inference overhead to the linear scale (3.5x slow down).

heading: 7 Conclusions
text: Our work conducts a comprehensive off-target error analysis with strong multilingual NMT models, to answer the question of how off-target translation wins over time during decoding. We additionally propose an empirical Language-informed Beam Search algorithm to mitigate off-target errors during decoding time and with linear-scale overhead.

heading: 8 Limitations
text: In this study, we utilize the widely adopted FastText LiD model, and the performance of LiBS may vary with the use of alternative LiD models. As our method is a modified beam search algorithm, it is not directly applicable to recent Language Language Models (Brown et al., 2020), which often do sampling during inference. Yet, we believe it will be particular interesting to adopt similar approach for LLM inference, as study shows LLMs are prune to hallucination (Zhang et al., 2023).

heading: Future Work
text: We hypothesize that it is due to the recency bias and poor calibration, yet it remains an interesting research question for future work. Possible Solutions Off-target candidate gaining greater model score demonstrates that the model is poorly calibrated, especially for the later steps of autoregressive decoding. Methods with additional
training data (Gu et al. , 2019; Zhang et al. , 2020) or regularizations (Yang et al. , 2021) could alleviate this issue during training with a well-calibrated model. In this work, with the knowledge of how off-target cases emerge during decoding, we attempt to fix this issue solely at the decoding time by improving on beam search algorithm even with a proven poorly calibrated model