Understanding and Addressing the Under-Translation Problem from the Perspective of Decoding Objective (2024)

Abstract: Neural Machine Translation (NMT) has made remarkable progress over the past years. However, under-translation and over-translation remain two challenging problems in state-of-theart NMT systems. In this work, we conduct an in-depth analysis on the underlying cause of under-translation in NMT, providing an explanation from the perspective of decoding objective. To optimize the beam search objective, the model tends to overlook words it is less confident about, leading to the under-translation phenomenon. Correspondingly, the model’s confidence in predicting the End Of Sentence (EOS) diminishes when under-translation occurs, serving as a mild penalty for under-translated candidates. Building upon this analysis, we propose employing the confidence of predicting EOS as a detector for under-translation, and strengthening the confidence-based penalty to penalize candidates with a high risk of under-translation. Experiments on both synthetic and real-world data show that our method can accurately detect and rectify under-translated outputs, with minor impact on other correct translations.

heading: 1 Introduction
text: Neural Machine Translation (NMT) has made remarkable progress over the past years (Bahdanau et al., 2015; Sutskever et al., 2014; Cho et al., 2014; Vaswani et al., 2017). Under-translation and over-translation are two typical problems in NMT, where under-translation means some words are mistakenly untranslated, and over-translation means some words are unnecessarily translated for multiple times (Tu et al., 2016; Mi et al., 2016).
Despite the rapid progress of NMT, the mechanisms behind the occurrence of under-translation and over-translation remain unclear, and these problems continue to be challenging obstacles faced by NMT systems (He et al., 2020). Prior efforts have primarily focused on modeling the explicit coverage of source content (Tu et al., 2016; Mi et al.,
2016; Tu et al., 2017; Zheng et al., 2018) or enforcing the attention-based source coverage during decoding (Wu et al., 2016), without advancing the understanding of why NMT systems tend to overlook or repeat certain source words. Zhao et al. (2018, 2019) empirically discovered that source words with a large translation entropy or those requiring reordering during translation are more likely to be ignored. However, this observation lacks a comprehensive explanation.
In this work, we conduct an in-depth analysis on the underlying cause of under-translation in NMT, providing an explanation from the perspective of decoding objective. Specifically, to gain deeper insights into the characteristics of under-translation and over-translation, we first generate synthetic data to simulate sentence-level and document-level translation scenarios, which allows us to automatically identify translation errors through predefined rules. The findings of sentence-level translation analysis indicate that words with high translation entropy, i.e., source words with a wide range of possible translations, are at a higher risk of being under-translated. In contrast, low-entropy words, whose translations are more predictable, are more likely to be over-translated, which aligns with the findings of Zhao et al. (2019). From the document-level translation experiments, we further observe the phenomenon of sentence-level under-translation, where the last sentence containing many high-entropy words is typically omitted. Additionally, we notice that when under-translation occurs, the probability of predicting the End Of Sentence (EOS) is generally lower, suggesting that the model is unwilling to stop generation when the translation is incomplete.
Although NMT imposes a penalty on predicting EOS, under-translation still occurs frequently, which can be explained from the perspective of decoding objective. NMT typically employs beam search for decoding, with an objective of finding
3800
the most probable sentence, or maximizing the log-probability normalized by sentence length (Wu et al., 2016). Consequently, NMT has a strong incentive to ignore high-entropy words, as they have low translation probabilities that contradict the decoding objective. Moreover, we theoretically reveal that the lower EOS probability serves as a penalty for under-translated candidates, but the penalty often underweighs the benefits of dropping multiple high-entropy words, leading to the occurrence of under-translation.
Building upon this analysis, we propose enhancing the EOS penalty on under-translated candidates to prevent under-translation. Since the model’s confidence in predicting EOS diminishes when undertranslation occurs, we employ the prediction probability of EOS as a detector for under-translation. For beam search candidates with high risk of undertranslation, we take the EOS probability as penalty and scale it to be proportional to the translation length. The detection ensures minimal interference with correct translations, and the scaling balances the impact of penalty with the benefits of dropping high-entropy source words, thereby more effectively preventing under-translations.
In summary, our contributions are: • We analyze the characteristics of undertranslation based on our synthetic data, suggesting that under-translation is more likely to occur on challenging words or sentences.
• We explain under-translation from the perspective of the decoding objective and theoretically reveal that the EOS probability serves as a penalty for under-translated candidates.
• We propose enhancing the EOS penalty on beam search candidates at risk of under-translation. Experiments show that our method can accurately detect and rectify under-translated outputs, with minor impact on other correct translations.

heading: 2 Experiments on Synthetic Data
text: Under-translation and over-translation are two challenging problems in neural machine translation, with one primary difficulty being the lack of automatic metrics for them. Previous work has mainly relied on human annotators to identify translation errors, which is time-consuming. In this section, we introduce a method for constructing synthetic data to simulate sentence-level and document-level translation scenarios, which allows us to automatically identify under-translation and over-translation
errors through predefined rules. Specifically, we create synthetic data containing only three words: A, B, and C, representing low-entropy, mid-entropy, and high-entropy words, respectively. We establish a fixed translation probability from source words to target words, as shown in Table 1. Notably, the most likely translation for each word is the word itself, but the probability distribution varies, with A being the sharpest and C being the smoothest. The optimal translation result should be a direct copy of the source, and any deviation indicates a translation error. Typically, a shorter translation compared to the source implies under-translation, while a longer translation implies over-translation.

heading: 2.1 Sentence-level Translation
text: Dataset Construction. To simulate sentencelevel translation scenarios, we construct the training set with both source and target sides being multiple sentences, composed exclusively of the three words: A, B, and C. The length of each source sentence is randomly selected from the set {1, 2, ..., 20}, with each source word being equally likely to be sampled from the set {A,B,C}. We translate the source words sequentially according to the probability distribution presented in Table 1. Additionally, we simulate noise in the training set by introducing a 15% chance of distortion for each word’s translation, with an equal likelihood of either dropping the word or translating it twice. For the test set, the source side is constructed using the same methodology, with the target side being an exact replication of the source.
Settings. Following the above methodology, we construct 1 million sentence pairs for the training and 1,000 sentence pairs for the test. The model architecture is a scaled-down version of Transformer (hdim = 128, hffn = 256, heads = 2, layers = 2). The number of training steps is 50,000. We apply beam search for decoding, with a beam size
of 5 and a length penalty of 1. Other settings follow the standard configuration of Transformer-base (Vaswani et al., 2017).
Results. We can automatically detect undertranslation and over-translation errors by comparing the length of the source input and the model’s decoding output. In the test set of 1,000 sentences, we identify a total of 53 instances of undertranslation and 12 instances of over-translation, indicating that these issues, particularly undertranslation, are significant and warrant attention. In Table 2, we present the word distribution in sentences with under-translation and over-translation errors. It can be observed that sentences with a higher proportion of the high-entropy word C are more prone to under-translation, which aligns with the findings of Zhao et al. (2019). Additionally, we also discover that sentences with a higher proportion of the low-entropy word A are more likely to be over-translated.

heading: 2.2 Document-level Translation
text: Dataset Construction. To simulate documentlevel translation scenarios, we further construct the dataset with source and target sides being a paragraph containing one or multiple sentences. We expand the vocabulary to {A,B,C, .}, where the new word ‘.’ denotes the end of a sentence. The number of source sentences is randomly selected from the set {1, 2, ..., 5}, and the length of source sentence is sampled from {1, 2, ..., 20}. Besides the word-level noise, we also introduce a 15% chance of distortion for each source sentence’s translation, with an equal likelihood of either dropping the sentence or translating it twice.
Settings. We employ the same settings as the wordlevel scenario.
Results. In the document-level experiment, we examined sentence-level over-translation and undertranslation errors by automatically detecting discrepancies in the number of periods (‘.’) between
the source sentence and the model’s decoding output. Analyzing a test set of 1,000 sentences, we find 27 instances of under-translation and 2 instances of over-translation at the sentence-level. As shown in Table 3, under-translation errors can be categorized into three types: last sentence undertranslation, penultimate sentence under-translation, and merging of the last two sentences, with the last sentence under-translation being the most common error scenario. Another observation is that the omitted sentences often contain a high proportion of high-entropy word C, making their translation more challenging. To systematically verify this, we further analyze the word distribution in the omitted sentences and find that the proportions of words A, B, and C are 30.2%, 26.1%, and 43.7%, respectively. This finding aligns with the conclusion at the word level, suggesting that the last sentences containing many high-entropy words are more likely to be omitted.

heading: 3 Approach
text: As observed in the previous section, undertranslation poses a significant challenge in NMT, particularly when high-entropy words and difficult sentences are involved. To provide a comprehensive understanding of this phenomenon, we offers an intuitive explanation and theoretical analysis from the perspective of the decoding objective in Section 3.1. Following this, we investigate the relationship between the EOS probability and undertranslation in Section 3.2, and then design the EOS penalty to prevent under-translation in Section 3.3.

heading: 3.1 Effect of Decoding Objective
text: When humans engage in translation, their objective is to accurately convey the meaning of the source text using words in the target language. In contrast, when machines perform translation, their objective
is expressed through a mathematical formula:
max Y logPθ(Y |X) |Y |α , (1)
where X,Y represent the source and target sentence, respectively, and α denotes the length penalty. We argue that this discrepancy results in deviations between machine translation outputs and human expectations, with a typical outcome being the under-translation of high-entropy words and difficult sentences.
Maximizing the length-normalized logprobability implies that the output words should have the highest log-probabilities in the average sense. This inherently encourages the model to disregard words that fall below average, which contradicts the intention of producing a comprehensive translation. The deterrent to under-translation is the potential incoherence of the translation if certain words are omitted, which reduces the overall translation probability. However, there are instances where the incomplete translation of some words does not significantly impact other parts of the translation. In such cases, the exclusion of high-entropy words and difficult sentences can be particularly attractive to the model.
The dominant type of sentence-level undertranslation, the omission of the last sentence, serves as an example. Suppose the model generates two candidates when translating a document: Ypre, which excludes the last sentence, and Ypre:last, a comprehensive translation that includes the last sentence Ylast. The model’s choice between these candidates depends on the comparison of their decoding objectives, i.e., whether the following condition is satisfied:
logPθ(Ypre:last|X) |Ypre:last|α > logPθ(Ypre|X) |Ypre|α . (2)
Given that Ylast represents a single sentence and Ypre refers to all preceding sentences within a document, it is reasonable to assume that their length ratio λ = |Ypre||Ylast| is close to 0. This assumption enables us to approximate (1 + λ)α ≈ 1 + αλ. Leveraging this approximation, we can transform the above condition into the following inequality (see Appendix A for a detailed derivation):
− logPθ(eos|X,Ypre\eos)
|Ylast| >
α · logPθ(Ypre|X)|Ypre| − logPθ(Ylast|X,Ypre)|Ylast| .
(3)
It can be observed that the model’s decision to exclude the last sentence is primarily based on a comparison between its average log-probability and the product of the average log-probability of previous sentences and a length penalty. The log-probability of the EOS token also acts as a penalty against under-translation. Intuitively, the model’s confidence in predicting EOS should diminish when the translation is incomplete, thereby enhancing the penalty for under-translation. Although the above analysis is specifically for the omission of the last sentence, it is generally applicable to other cases of sentence-level under-translation, with an additional factor involved, namely the impact of missing a middle sentence on the conditional probabilities of subsequent sentences.

heading: 3.2 EOS Probability Analysis
text: The above analysis suggests that the log-probability of EOS acts as a penalty against under-translation. Even when a sentence contains many high-entropy words that make it challenging to translate, the model will still favor the complete translation as long as it assigns a low EOS log-probability to the
imcomplete translation. Therefore, we first investigate whether NMT tends to assign lower end-of-sentence probabilities to unfished translations. We conduct experiments on both sentence-level and document-level synthetic test sets, comparing the distribution of EOS log-probabilities between under-translated sentences and all sentences within the test set. The results are presented in Figure 1. As can be seen, on both datasets, the EOS log-probabilities of undertranslated sentences are significantly lower, suggesting that the model is unwilling to stop generation when the translation is incomplete. This effect is particularly evident when sentence-level under-translation takes place within the documentlevel dataset, indicating that the decrease in EOS log-probability is also influenced by the number of under-translated words.
Equation 3 implies that the log-probability of EOS needs to scale linearly with the number of under-translated words in order to effectively prevent under-translation. Therefore, we delve deeper into the relationship between EOS log-probability and the number of under-translated words to verify whether this requirement is fulfilled. On the document-level synthetic test set, we compute the average EOS log-probability for each length of under-translated sentences. The results are shown in Figure 2. As observed, although the EOS logprobability and the number of under-translated words generally display a positive correlation, the rate of change does not reach linearity. Therefore, the EOS penalty tends to underweigh the benefits of dropping multiple high-entropy words, leading to the occurrence of under-translation.

heading: 3.3 Enhancing the EOS Penalty
text: Based on the above analysis of EOS probability, we have two key findings to help address the under-translation problem. First, the model’s confidence in predicting EOS diminishes when under-translation occurs, enabling us to utilize the model’s predicted probability of EOS to assess the risk of under-translation. Second, the logprobability of EOS acts as a penalty against undertranslation, but its effect is outweighed by the benefits of dropping multiple high-entropy words. Therefore, we can further enhance the EOS penalty for candidates with a high risk of under-translation, leading to a more precise and effective prevention of under-translations.
To utilize the EOS probability for under-
translation detection, the most straightforward approach is to establish a threshold and label sentences with an EOS log-probability below this threshold as prone to under-translation. However, as illustrated in Figure 1, the distribution of EOS log-probability varies across different datasets, so a fixed threshold may not perform well universally. We empirically find that a comparative metric offers more consistent performance across different datasets. Specifically, we compare the EOS token against its closest competitor, w2, which is the second-highest ranked word in the final step. A translation is considered prone to under-translation if the EOS probability does not significantly exceed that of w2, failing to satisfy the following condition:
logPθ(eos|X,Y )− logPθ(w2|X,Y ) > τ, (4)
where τ = 1 is a hyperparameter that exhibits consistent performance across different datasets.
Upon detecting beam search candidates at risk of under-translation, we can enhance their EOS penalty to prevent potential under-translation errors. A straightforward approach involves amplifying the log-probability of EOS by a constant factor. However, for short sentences, this method may lead to the decoding objective being dominated by the EOS probability, which can negatively impact the overall translation quality.
To address this, we modify the approach by scaling the EOS penalty based on the translation length, which avoids excessive effect on short sentences while ensuring a sufficient penalty for longer sentences. Another consideration is that when the decoded sentence is relatively long, the beam search process generally retains candidates with consistent prefixes due to the limited beam size. In such cases, it is unnecessary to fully amplify the EOS penalty based on the entire translation length. Instead, we can truncate the length by setting an upper limit. In summary, the final score for beam search candidates are adjusted as follows:
{ logPθ(Y |X)
|Y |α , if meets Cond.4 logPθ(Y |X)+λ·logPθ(eos|X,Y )
|Y |α , otherwise ,
λ = β ·min(|Y |, L), (5)
where β = 0.4 and L = 20 are hyperparameters that control the penalty weight λ.

heading: 4 Experiments

heading: 4.1 Settings
text: Datasets. We validate our method on both sentence-level and document-level benchmarks using synthetic and real-world translation data. The construction details of the synthetic data have been described in section 2. For real-world translation, we use the test set of WMT22 Chinese-to-English translation (WMT22 Zh-En, 1,875 sentence pairs) as the sentence-level translation benchmark (Kocmi et al., 2022). For document-level translation, we combine sentences from the WMT22 Zh-En test set into documents, stopping when the length exceeds 200. This results in a document-level translation benchmark comprising 328 document pairs.
Models. For synthetic data, the model architecture is a scaled-down version of Transformer (hdim = 128, hffn = 256, heads = 2, layers = 2). The training details have been described in section 2. For real-world translation, we employ large language models (LLMs) as they demonstrate good performance in both sentence-level and documentlevel translation tasks (Jiao et al., 2023; Wang et al., 2023a; Wu et al., 2024). Our foundation model is LLaMA2-7B (Touvron et al., 2023), which we fine-tune for 1 epoch using the Alpaca instruction dataset (Wang et al., 2023b; Taori et al., 2023) and Chinese-English test sets from WMT17-20. We also augment the translation data by concatenating three consecutive sentences as a document, thereby equipping the final model with both sentence-level and document-level translation capabilities. Other training hyperparameters are the same as Alpaca7B (Taori et al., 2023).
Decoding. The prompt for decoding is “Translate the following sentences from Chinese to English”. Unless otherwise specified, the beam size is 5 and the length penalty is 1.0.

heading: 4.2 Results on Synthetic Data
text: First, we conduct experiments on synthetic data to investigate the impact of enhancing the EOS penalty on under-translation and over-translation. In sentence-level NMT, we measure the proportion of sentences with word-level under-translation and over-translation errors. In document-level NMT, we do not consider word-level errors and only measure the proportion of translations with sentencelevel under-translation and over-translation errors. The experimental results are shown in Table 4.
As can be seen, the main issue faced by the model is under-translation, with far fewer cases of over-translation in comparison. After enhancing the EOS penalty, the under-translation problem significantly decreases in both sentence-level and document-level NMT. The side effect is an increase in over-translation, but this is relatively minor compared to the reduction in under-translation. In document-level NMT, we have categorized the under-translation errors in synthetic data into three tpyes: last, penultimate, and merge, as shown in Table 3. Our method primarily resolves last sentence under-translation, with 10 out of the 12 reduced under-translation errors belonging to this type.

heading: 4.3 Results on Real-world Data
text: For real-world translation, we conduct human analysis to identify under-translation and overtranslation errors. We take the first 300 sentences from the sentence-level test set and the first 100 documents from the document-level test set, and measure the proportion of under-translation and over-translation on these subsets. The results are shown in Table 5.
We can observe that under-translation is also a significant problem in real-world translation scenarios. Over-translation, also referred to as hallucination, is less common in comparision. Enhancing the EOS penalty effectively mitigates the occurrence of under-translation in both sentence-level and document-level NMT. In contrast, we have only observed one instance of over-translation resulting from the enhanced EOS penalty.
Upon further examination of the variations in translations, we find that our method is particularly prominent in terms of precision, affecting only about 10% of the translations. After enhancing the EOS penalty, the modifications in translations mainly fall into three types: adding punctuation at the end, paraphrasing of translated sentences, and retrieving missing translations, as illustrated in Table 6. In the 300 sentences we extracted, the number of modifications in these three types are 8, 6, and 9, respectively. In the 100 documents we extracted, they are 3, 1, and 6, respectively. As can be seen, the modifications primarily focus on mitigating under-translations, with minor impact on correct translations. We present the details of these translation differences in Appendix B.

heading: 4.4 Effect of Detection
text: Our method particularly prominent in terms of precision on mitigating under-translations, which can be attributed to the incorporation of an undertranslation detection module (Equation 4). The detection allows us to apply the EOS penalty selectively to sentences that exhibit a high risk of under-translation, thus having a minor impact on other correct translations.
The effect of the under-translation detection module is illustrated in Table 7. It can be observed that incorporating the EOS detection module results in only a slight reduction in the number of retrieved under-translations. The advantage lies in that it significantly reduces the impact on other originally correct translations, thereby enhancing the precision of under-translation rectification.

heading: 4.5 Types of Under-translation
text: In real-world scenarios, we found that the categories of under-translation errors are not entirely consistent with those summarized in the synthetic
Type Sentence-level Document-levelw/ det. w/o det. w/ det. w/o det.
data, but there are similarities. In both sentencelevel and document-level NMT, we divide undertranslation errors into three types according to their occurrence position: begin, middle, and end. In this context, ‘begin’ refers to the first few (1-3) words of a sentence or the document’s first sentence, and ‘end’ exhibits a similar meaning. We present the distribution of under-translation errors in Table 8.
We can see that under-translation is most likely to occur at the end, where the last sentence accounts for only a small portion of the document but contributes to half of the under-translation errors. This observation is consistent with our earlier findings in synthetic data and aligns with our theoretical analysis. Under-translation at the end is also the primary focus of our method. In sentencelevel translation, it eliminates 9 under-translated sentences, with 3 in the middle position and 6 at the end. In document-level translation, our method detects 6 out of 7 under-translations at the end and corrects 4, while the remaining two still exhibit some under-translation after modification.
A limitation of our method is its weaker ability to handle under-translation at the begin and middle stages. Our method modifies final scores and rankings of beam search candidates but cannot influence the early decoding process. Due to the limited beam size, candidates containing complete translations are already dropped during beam search, so we are unable to resolve these under-translation errors by modifying scores of final candidates.

heading: 4.6 Impact on Translation Quality
text: In the previous sections, our evaluation has been focused on the effects of our method on undertranslation and over-translation, but its overall impact on translation quality remains unclear. In this section, we further examine the impact of our method on the BLEU (Papineni et al., 2002), the most widely used evaluation metric in machine translation. We measure the sacreBLEU (Post, 2018) and report the results in Table 9.
The results indicate that addressing the undertranslation problem can lead to a positive improvement in the evaluation metric. However, this improvement is relatively moderate, which be explained by two main factors. First, the errors resulting from under-translation do not comprise a substantial portion of the entire test set, thus limiting the overall improvement from resolving under-translation. Second, as we have analyzed, sentences with increased complexity and a higher number of high-entropy words are more prone to under-translation. Therefore, the retrieved translations may not achieve the same level of accuracy as other translations in the test set, resulting in a limited improvement in the metrics.

heading: 4.7 Comparison with Other Penalties
text: In addition to the proposed EOS penalty, Wu et al. (2016) has introduced the length penalty and coverage penalty to penalize under-translation, where we have incorporated a length penalty of 1.0 in our method. In this section, we further explore the effect of length penalty and coverage penalty on under-translation, and compare them with the EOS penalty, as illustrated in Table 10.
We find that both the length penalty and coverage penalty exhibit some ability to resolve undertranslation, with the effect of the coverage penalty being more significant. However, their precision on addressing under-translation (∆Under/∆All) is relatively lower, affecting more unrelated translations. Specifically, we note that both penalties tend to convert translations into synonymous but longer forms, sometimes even causing hallucinations, which is reflected by the increased number of tokens ∆Tokens.

heading: 5 Related Work
text: Research on the under-translation problem in NMT can be roughly divided into three categories: modeling source coverage, exploring translation entropy, and integrating penalties during the decoding.
Source Coverage. Tu et al. (2016) pointed out the problems of under-translation and over-translation in neural machine translation and proposed a method for tracking source coverage vector to adjust the decoding of model. Similarly, Mi et al. (2016); Tu et al. (2017); Zheng et al. (2018) focused on modeling the coverage of source content. Mi et al. (2016) introduced a coverage embedding vector to track the coverage of source words. Tu et al. (2017) proposed to reconstruct the source sentence, thereby ensuring that all source information is included in the target side. Zheng et al. (2018) addressed under-translation by making the model aware of translated and untranslated contents.
Translation Entropy. Zhao et al. (2018, 2019) discovered that source words with a large translation entropy or those requiring reordering during translation are more likely to be ignored, and proposed pre-ordering the source sentence or converting high-entropy words into special tokens to prevent under-translation. Liu et al. (2022); Chen et al. (2023) further employed entropy to assess translation difficulty, using this information to create test sets or enhance evaluation metrics.
Decoding Penalty. Wu et al. (2016) proposed the
length penalty and coverage penalty to penalize under-translation. The length penalty adjusts the log-probability score based on sentence length, reducing bias towards short sentences. The coverage penalty encourages source-target attention to cover all source words to prevent under-translation.

heading: 6 Conclusion
text: In this paper, we conduct an in-depth study of the under-translation problem in NMT, explaining and addressing this problem from the perspective of the decoding objective. Our analysis reveals that under-translation is more likely to occur on challenging words or sentences. We explain it from the perspective of the decoding objective and propose enhancing the EOS penalty on under-translated candidates to prevent under-translation. Experimental results show that our method effectively mitigates the occurrence of under-translation, with only a minor impact on other correct translations.

heading: 7 Limitations
text: This paper investigates the underlying causes of under-translation in NMT and proposes a solution based on the End-of-Sentence (EOS) penalty. This is a preliminary attempt from the perspective of the decoding objective, and there are limitations that warrant future improvements.
First, our method modifies the final scores and rankings of beam search candidates but cannot influence the early decoding process. Therefore, when correct and complete translations are already dropped during the early search process, our method cannot retrieve the missing translation. Overcoming this limitation may be possible by identifying additional signals strongly associated with under-translation during decoding and designing corresponding penalties.
Second, the period (‘.’) plays a similar role with EOS that indicates the end of a sentence. In sentences ending with a period, the EOS probability is typically high, making it difficult to identify undertranslation. A possible future direction is to construct a vocabulary of punctuation marks, taking into account words like periods that indicate the end of a sentence when designing the penalty.
Lastly, our synthetic data construction does not fully simulate real translation data. For instance, we only translate source words sequentially when constructing the target-side translation, neglecting potential reordering and dependencies between tar-
get words. A more sophisticated construction of synthetic data may help us further reveal the characteristics of under-translation and design better decoding objectives.

heading: A Proof
text: In this section, we present the derivation process from Equation 2 to Equation 3. To begin, we redefine the notation to avoid ambiguity. In the following, we will use Ypre and Ylast to represent sentences in a linguistic context without the EOS token, and use Ypre and Ylast to denote sentences that include the EOS token. In this way, Equation 2 can be rewritten as:
logPθ(Ypre|X) + logPθ(Ylast|X,Ypre) + logPθ(eos|X,Ypre:last) (|Ypre|+ |Ylast|)α > logPθ(Ypre|X) + logPθ(eos|X,Ypre) |Ypre|α .
Let λ = |Ylast||Ypre| represent the length ratio of the last sentence to the preceding sentences. The above equation can be transformed to:
logPθ(Ypre|X) + logPθ(Ylast|X,Ypre) + logPθ(eos|X,Ypre:last) (1 + λ)α > logPθ(Ypre|X) + logPθ(eos|X,Ypre).
Since Ylast represents a single sentence and Ypre refers to all preceding sentences within a document, it is reasonable to assume that their length ratio λ is close to 0. Thus, we can approximate (1 + λ)α ≈ 1 + αλ, which is the only approximation made in the derivation. Next, we multiply both sides of the equation by 1 + αλ and cancel out common terms, yielding:
logPθ(Ylast|X,Ypre) + logPθ(eos|X,Ypre:last) > αλ · (logPθ(Ypre|X) + logPθ(eos|X,Ypre)) + logPθ(eos|X,Ypre).
Further simplifying the equation with the notation Ypre and Ylast, we obtain:
logPθ(Ylast|X,Ypre) > αλ · logPθ(Ypre|X) + logPθ(eos|X,Ypre).
Dividing each side by |Ylast| = λ · |Ypre|, we get:
logPθ(Ylast|X,Ypre) |Ylast| > α · logPθ(Ypre|X)|Ypre| + logPθ(eos|X,Ypre) |Ylast| .
Finally, by reorganizing the terms and reverting to the initial notation, we can get Equation 3 as presented in the main text.

heading: B Translation Differences

heading: C Effect of Hyperparameters
text: There are hyperparameters in the proposed EOS penalty, such as τ = 1 in Equation 4 and β = 0.4 in Equation 5. The hyperparameter τ = 1 controls the range at which the EOS penalty takes effect, and the hyperparameter β represents the weight of the EOS penalty. A larger τ or β can fix more under-translations but also introduce more other modifications.
The hyperparameters were chosen based on extensive experiments, and finally the choosen hyperparameters were found to exhibit stable performance across different datasets. Overall, our method exhibits relatively low sensitivity to these hyperparameters. Even with significant adjustments to the hyperparameters (e.g., increasing τ to 2 or decreasing β to 0.2), the impact on translations remains small. To illustrate it, we generate translations of the WMT 22 Zh-En test set under different hyperparameter settings. The BLEU scores are displayed in Table 13, showing that the impact of hyperparameters on overall translation quality is relatively minor. The proportions of under-translation and over-translation on the first 300 sentences of WMT22 Zh-En test set are displayed in Table 14, indicating that tweaking the hyperparameters can slightly affect the rates of over-translation and under-translation.
Overall, our method exhibits relatively low sensitivity to hyperparameters, affecting only a small number of translations. For instance, Table 15 illustrates the impact of reducing β to 0.2 on the translations of first 300 sentences of WMT22 Zh-En test set, resulting in changes in just two translations.

heading: D Encoder-Decoder Transformer
text: In this study, our experiments are mainly conducted on the fine-tuned LLaMA2 model, due to its superior translation capabilities and support for document-level translation, aligning more closely with real-world applications. Our method is also applicable to the traditional encoder-decoder Transformer architecture. Specifically, when applied to the Transformer-big model (Vaswani et al., 2017) trained on the WMT22 Zh-En dataset, the application of EOS penalty results in an improvement of the BLEU score from 15.25 to 15.36. After applying the EOS penalty, we observe changes in the translations within the first 300 sentences of the WMT22 Zh-En test set, with a total of 18 instances of modification. Among these, 4 instances of under-translations are corrected.

heading: E Pseudo Code for EOS Penalty
text: 1 # Initialization 2 beam_scores = initialize_zeros(batch_size , num_beams) 3 input_ids = initialize_with_start_token_ids () 4 cur_len = initial_sequence_length 5 tau = 1 6 beta = 0.4 7 L = 20 8
9 # Generation loop 10 while not all_beams_ended: 11 # Generate the probability distribution for the next token 12 next_token_logits = model(input_ids) 13 next_token_scores = F.log_softmax(next_token_logits , dim=-1).view(batch_size , num_beams , vocab_size) 14 15 # Detect under -translation by EOS probability 16 eos_score = next_token_scores [:, :, EOS_token_id] 17 next_highest_score = torch.topk(next_token_scores , 2, dim = -1)[0][:, :, 1] 18 eos_detected = (eos_score - next_highest_score) < tau 19 20 # Generate top beam candidates 21 next_token_scores = next_token_scores + beam_scores [:, None]. expand_as( next_token_scores) 22 next_token_scores = next_token_scores.view(batch_size , num_beams * vocab_size) 23 next_token_scores , next_tokens = torch.topk( 24 next_token_scores , 2 * num_beams , dim=1, largest=True , sorted=True 25 ) # (batch_size , 2 * num_beams) 26 27 # Apply EOS penalty on beam candidates 28 for batch_idx in range(batch_size): 29 for token_idx in range(num_beams * 2): 30 next_token = next_tokens[batch_idx , token_idx] 31 # if the candidate is finalized , check and apply EOS penalty 32 if next_tokens % vocab_size == EOS_token_id: 33 beam_idx = next_token // vocab_size 34 if eos_detected[batch_idx ][ beam_idx ]: 35 eos_penalty = min(cur_len - initial_len , L) * beta * eos_score[ idx][ beam_idx] 36 next_token_scores[batch_idx ][ token_idx] += eos_penalty 37 38 # Update beam scores based on next_token_scores 39 update_beam_scores(next_token_scores) 40 41 # Select the next tokens and their corresponding beams 42 next_tokens , next_beam_indices = select_topk(next_token_scores) 43 update_input_ids_and_beam_indices(next_tokens , next_beam_indices) 44 45 # Update current length 46 cur_len += 1 47 48 # Finalization 49 sequences = finalize_sequences_based_on_beam_scorer () 50 return sequences

heading: Future Work
text: This is a preliminary attempt from the perspective of the decoding objective, and there are limitations that warrant future improvements. First, our method modifies the final scores and rankings of beam search candidates but cannot influence the early decoding process. Therefore, when correct and complete translations are already dropped during the early search process, our method cannot retrieve the missing translation. Overcoming this limitation may be possible by identifying additional signals strongly associated with under-translation during decoding and designing corresponding penalties. Second, the period (‘. ’) plays a similar role with EOS that indicates the end of a sentence. In sentences ending with a period, the EOS probability is typically high, making it difficult to identify undertranslation. A possible future direction is to construct a vocabulary of punctuation marks, taking into account words like periods that indicate the end of a sentence when designing the penalty. Lastly, our synthetic data construction does not fully simulate real translation data. For instance, we only translate source words sequentially when constructing the target-side translation, neglecting potential reordering and dependencies between tar-
get words. A more sophisticated construction of synthetic data may help us further reveal the characteristics of under-translation and design better decoding objectives