Two Intermediate Translations Are Better Than One: Fine-tuning LLMs for Document-level Translation Refinement
Authors: Yichen Dong, Xinglin Lyu, Junhui Li, Daimeng Wei, Min Zhang, shimin tao, Hao Yang
Accepted at: acl25

Two Intermediate Translations Are Better Than One: Fine-tuning LLMs for Document-level Translation Refinement

Yichen Dong1, Xinglin Lyu2, Junhui Li1*, Daimeng Wei2, Min Zhang2, Shimin Tao2, Hao Yang2
1School of Computer Science and Technology, Soochow University, Suzhou, China
2Huawei Translation Services Center, Beijing, China
ycdong@stu.suda.edu.cn, lijunhui@suda.edu.cn
{lvxinglin1,weidaimeng,zhangmin186,taoshimin,yanghao30}@huawei.com

Abstract
Recent research has shown that large language models (LLMs) can enhance translation quality through self-refinement. In this paper, we build on this idea by extending the refinement from sentence-level to document-level translation, specifically focusing on document-to-document (Doc2Doc) translation refinement. Since sentence-to-sentence (Sent2Sent) and Doc2Doc translation address different aspects of the translation process, we propose fine-tuning LLMs for translation refinement using two intermediate translations, combining the strengths of both Sent2Sent and Doc2Doc. Additionally, recognizing that the quality of intermediate translations varies, we introduce an enhanced fine-tuning method with quality awareness that assigns lower weights to easier translations and higher weights to more difficult ones, enabling the model to focus on challenging translation cases. Experimental results across ten translation tasks with LLaMA-3-8B-Instruct and Mistral-Nemo-Instruct demonstrate the effectiveness of our approach.1

1 Introduction
Recent research has shown that large language models (LLMs) can improve their outputs through self-refinement (Madaan et al., 2023). In machine translation, translation refinement improves translation quality by refining intermediate results. For instance, Chen et al. (2024c) use GPT for translation refinement with simple prompts for iterative improvements. Similarly, Raunak et al. (2023) employ a chain of thought (CoT) strategy to describe suggested changes in natural language. Koneru et al. (2024) further expand this approach by using document-level context to refine current sentences.

Unlike previous studies, we extend translation refinement from sentence-level to document-level, refining the translations of all sentences in a document in one go. A document’s translation can be generated either by a sentence-to-sentence (Sent2Sent) or document-to-document (Doc2Doc) system. However, Sent2Sent translation, lacking document-level context, often faces discourse-related issues like lexical inconsistency and coherence problems. For example, as shown in Figure 1, the word “跑步机/pao_bu_ji” in the source document is translated as both running machine and treadmill in the Sent2Sent translation. Additionally, translating “但即使/dan_ji_shi” as even if disrupts coherence by ignoring the discourse relationship between sentences #2 and #3. Conversely, while Doc2Doc translation can reduce these discourse-related issues by incorporating both source- and target-side document-level context, it often suffers from under-translation, omitting phrases, clauses, or entire sentences. For example, the verb phrase “进入新领域/jin_ru_xin_ling_yu” in the source document is completely omitted in the Doc2Doc translation. Taking Chinese-to-English document-level translation as example, Table 1 compares the performance between Sent2Sent

<table>
  <tr>
    <th>Source Document</th>
    <td>#1 竞争就像是一台跑步机/pao_bu_ji.<br>#2 如果你呆在原地，就会被送上跑步机/pao_bu_ji.<br>#3 但即使/dan_ji_shi 你跑起来，你也无法真正跨出跑步机/pao_bu_ji，进入新领域/jin_ru_xin_ling_yu</td>
  </tr>
  <tr>
    <th>Sent2Sent Translation</th>
    <td>#1 Competition is like a running machine.<br>#2 If you stay where you are, you will be taken away from the treadmill.<br>#3 Even if you do run, you can’t truly step outside the treadmill, into new territory.</td>
  </tr>
  <tr>
    <th>Doc2Doc Translation</th>
    <td>#1 Competition is like a treadmill.<br>#2 If you stand still, you get thrown off.<br>#3 But even if you run, you can never really get off the treadmill.</td>
  </tr>
  <tr>
    <th>Our Translation Refinement</th>
    <td>#1 Competition is like a treadmill.<br>#2 If you stand still, you get thrown off<br>#3 But even if you run, you can’t really step off the treadmill, into new territory.</td>
  </tr>
</table>

Figure 1: An example of Sent2Sent and Doc2Doc Chinese-to-English translations.
<table>
  <tr>
    <th>System</th>
    <th>d-COMET</th>
    <th>Coh.</th>
    <th>LTCR</th>
    <th>ALTI+</th>
  </tr>
  <tr>
    <td>Sent2Sent</td>
    <td>82.18</td>
    <td>54.98</td>
    <td>46.32</td>
    <td>59.32</td>
  </tr>
  <tr>
    <td>Doc2Doc</td>
    <td>83.60</td>
    <td>56.21</td>
    <td>50.00</td>
    <td>58.66</td>
  </tr>
</table>

Table 1: Performance comparison between Sent2Sent and Doc2Doc Chinese-to-English translations.

and Doc2Doc by LLaMA3-8B-Instruct without fine-tuning. It shows that Doc2Doc achieves better performance in document-level metrics like d-COMET (Vernikos et al., 2022; Rei et al., 2022a), Coherence (Li et al., 2023) and LTCR (Lyu et al., 2021), while Sent2Sent excels in sentence level metrics like ALTI+ (Dale et al., 2023) which detects hallucination and under-translation.2

Therefore, we conjecture that refining document-level translation over two intermediate translations from both Sent2Sent and Doc2Doc systems can leverage their strengths, thereby mitigating the aforementioned issues. Given a source document, we prompt an existing LLM to generate Sent2Sent and Doc2Doc translations, denoted as sent2sent and doc2doc translations, respectively. We then construct a document-level refinement quadruple (source, sent2sent, doc2doc, reference), where reference serves as the naturally refined translation with all the elements at the document level.

Motivated by Feng et al. (2024a), who show that distinguishing between sentences with varying quality improves sentence-level translation refinement, we propose an enhanced fine-tuning with quality awareness. This enhanced fine-tuning differentiates instances based on the difficulty of refinement by expanding above quadruple into a quintuple (source, sent2sent, doc2doc, quality, reference). The goal of it is to address the varying difficulty of refining translations at sentence- and document-level. Naturally, we weight the documents at sentence level instead of instance level (Lison and Bibauw, 2017) or token level (Fang and Feng, 2023) since the quality of different sentence within one document may differ significantly. Please refer to Appendix A for more details. By incorporating a quality score as an additional factor during fine-tuning, it helps the model prioritize and output a better translation with differing refinement inputs.

Overall, our main contributions in this work can be summarized as follows:3

• We extend translation refinement from the traditional sentence-level to the document-level, and further expand it by refining two intermediate translations rather than just one.

• We introduce enhanced fine-tuning with quality awareness, which differentiates instances based on the difficulty of refinement.

• Experimental results on two popular LLMs across ten \( X \leftrightarrow \text{En} \) document-level translation tasks demonstrate that refining two intermediate translations outperforms refining from a single translation.

2 Methodology

Unlike previous studies that fine-tune LLMs for translation using sentence- or document-level parallel datasets, our approach focuses on document-level translation refinement. Specifically, to leverage the diversity between Sent2Sent and Doc2Doc translations, we introduce document-level translation refinement with two intermediates, using the reference as the target. This emphasis on document-level refinement, rather than direct translation or sentence-level refinement, distinguishes our work from prior LLM-based translation methods.

As shown in Figure 2, we develop our document-level refinement LLMs in two steps:

• Fine-Tuning Data Preparation (Section 2.1): For each source-side document in the fine-tuning set, we generate two versions of its translation: one using Sent2Sent translation and the other using Doc2Doc translation.

• Enhanced Fine-Tuning with Quality Awareness (Section 2.2): Using the prepared fine-tuning data, we fine-tune LLMs in two stages: a naïve fine-tuning stage followed by the other stage with a quality-aware strategy.

Finally, Section 2.3 describes the inference.

2.1 Fine-Tuning Data Preparation

We represent a document-level parallel in the fine-tuning data as \((\mathbf{s}, \mathbf{r})\), where \(\mathbf{s} = [s_1, \cdots, s_N]\), \(\mathbf{r} = [r_1, \cdots, r_N]\), with \(N\) denoting the number of sentences in the document pair. First, we use LLM \(\mathcal{M}_S\) to generate sentence-level translations \(\mathbf{y} = [y_1, \cdots, y_N]\) by translating sentences in \(\mathbf{s}\) individually, following the prompt template in Figure 3

2Detailed experimental settings, metrics and the results can be found in Section 3.
3See Appendix D for how our approach can be easily adapted to Doc2Doc translation, even when the source and target documents have differing numbers of sentences.
(a). Then, we generate document-level translations \( \mathbf{z} = [z_1, \cdots, z_N] \) by treating the document as a continuous sequence, as shown in Figure 3 (b). We follow Li et al. (2024) to organize the sentences within a document by inserting markers # id between neighbouring sentences, which indicate their respective positions. Typically, most references r have higher quality than y and z though some references may have lower quality (Xu et al., 2024a) which can be treated as noise. Thus, we use r as the target for refinement, as Feng et al. (2024a). This process yields the document-level refinement quadruple (s, y, z, r).

Sentence-level Quality-aware Weight. For two sentences \( s_i \) and \( s_j \) in document s, the difficulty of refining their translations can vary, depending on the quality of their respective translations \( y_i/z_i \) and \( y_j/z_j \). Based on the definition in Feng et al. (2024a), easy translations differ significantly from the reference, providing the most room for refinement, while hard translations are nearly perfect, making refinement more challenging. Thus, we assign lower weights to easy translations and higher weights to hard translations. For sentence \( s_i \) and its two translations \( y_i \) and \( z_i \), we use reference-based sentence-level COMET to evaluate the translation quality and compute the weight as follows:

\[
w_i = 1 + \lambda (\max(\mathrm{DA}(s_i, y_i, r_i), \\
\mathrm{DA}(s_i, z_i, r_i)) - \epsilon),
\]

where \( \lambda \) and \( \epsilon \) are the hyper-parameters, and DA is computed using reference-based COMET wmt22-comet-da\( ^4 \) (Rei et al., 2022a). This expands the document-level refinement quadruple into a quintuple (s, y, z, w, r), where w = \([w_1, \cdots, w_N]\) represents sentence-level quality-aware weights.

Preventing Position Bias. Figure 3 (c) shows the prompt template for document-level translation refinement. To avoid position bias, where LLMs might only attend to specific positions (Liu et al., 2024), the placeholder <hyp1> can represent either the sentence-level translation y or the document-level translation z, with the other in <hyp2>. This design creates two instances from the quintuple (s, y, z, w, r). For clarity, we refer to the quintuple as (s, h_1, h_2, w, r), where h_1 and h_2 denote the two intermediate translations in the template.

2.2 Enhanced Fine-Tuning with Quality Awareness

To better leverage the training set, we propose an enhanced fine-tuning strategy, fine-tuning LLM \( \mathcal{M}_T \) in two stages on the same dataset. In the first stage, we perform naïve fine-tuning treating all instances equally. In the second stage, we fine-tune with quality-aware weights. The prompt template for the fine-tuning in both stages is shown in Figure 3 (c).

Naïve Fine-Tuning. In this stage, the LLM \( \mathcal{M}_T \) is fine-tuned on the fine-tuning set \( \mathcal{T} \) to minimize

![Illustration of our approach.](page_186_120_1077_579.png)

Figure 2: Illustration of our approach.
the following cross-entropy loss function:

\[
\mathcal{L}_1(\mathcal{T}) = - \sum_{q \in \mathcal{T}} \log P\left(r|\mathcal{P}\left(s, h_1, h_2\right)\right)
= - \sum_{q \in \mathcal{T}} \sum_{i=1}^N \log P\left(r_i|\mathcal{P}\left(s, h_1, h_2\right), r_{<i}\right),
\]

where \( q \) denotes a quintuple \( (s, h_1, h_2, w, r) \), \( \mathcal{P}\left(s, h_1, h_2\right) \) returns the prompt defined by the template, \( r_{<i} \) represents the previous sentences before \( r_i \) in \( r \). In this stage, all sentences in the reference document \( r \) are assigned equal weights, specifically a weight of 1.

**Quality-aware Fine-Tuning.** In this stage, we continue to fine-tune \( \mathcal{M}_T \) on \( \mathcal{T} \) using a quality-aware strategy, achieved by assigning quality-aware weights to the sentences in the reference \( r \) when calculating the loss function:

\[
\mathcal{L}_2(\mathcal{T}) = - \sum_{q \in \mathcal{T}} \sum_{i=1}^n w_i \log P\left(r_i|\mathcal{P}\left(s, h_1, h_2\right), r_{<i}\right).
\]

Specifically, all tokens within a reference sentence \( r_i \) have the same weight \( w_i \). And we refer to the fine-tuned LLM as \( \mathcal{M}_T^* \).

**2.3 Inferencing**

Once fine-tuning the LLM \( \mathcal{M}_T^* \) is complete, we use it to refine translations on the test sets. As shown in Figure 2 (c), we first prompt \( \mathcal{M}_S \) to generate both Sent2Sent and Doc2Doc translations. Then, for each source document, the two intermediate translations are fed into \( \mathcal{M}_T^* \) for refinement. During inferencing, quality-aware weights are not needed.

**3 Experimentation**

**3.1 Experimental Settings**

**Datasets.** Following recent works (Li et al., 2024; Lyu et al., 2024; Alves et al., 2024; Cui et al., 2024), to avoid data leakage (Garcia et al., 2023), we utilize the latest News Commentary v18.15 in WMT24, which features parallel text with document boundaries. Our experiments cover five language pairs in both directions: English (En) \( \leftrightarrow \) {German (De), Russian (Ru), Spanish (Es), Chinese (Zh), and French (Fr)}. For each pair, we randomly select 150 documents for development and another 150 for testing. Specifically, we split long documents into chunks. Details on the dataset and handling long documents are in Appendices B and C.

**Models and Settings.** We select LLaMA-3-8B-Instruct6 (Meta, 2024) and Mistral-Nemo-Instruct7 (MistralAI, 2024) as the foundation open-source LLMs for applying prompt engineering (i.e., \( \mathcal{M}_S \)) and quality-aware fine-tuning (i.e., \( \mathcal{M}_T \)).8 For detailed fine-tuning and hyper-parameter settings, please refer to Appendices E and F. For details of computational costs, please refer to Appendix G. Moreover, fine-tuning LLMs using document-level translation refinement does not result in catastrophic forgetting of sentence-level translation capabilities. For further details, please refer to Appendix I.

**Baselines.** We compare our approach to several baselines:
• Sent2Sent: As described in Section 2.1, we prompt \( \mathcal{M}_S \) to generate sentence-level transla-

(a) Sent2Sent Translation
Translate this document from <src_lang> into <tgt_lang>.
Don’t give any explanation.
<src_lang> Source: <sent_src>
<tgt_lang> Translation:

(b) Doc2Doc Translation
Translate this document from <src_lang> into <tgt_lang>.
Don’t give any explanation.
Each sentence is separated by #id.
<src_lang> Source: <doc_src>
<tgt_lang> Translation:

(c) Translation Refinement
You are an expert in editing translations.
Given a <src_lang> source text and two <tgt_lang> translated versions, please produce an improved translated version by drawing upon the strengths of both initial translations.
Don’t give any explanation.
Each sentence is separated by #id.
<src_lang> Source: <doc_src>
<tgt_lang> Translation 1: <hyp1>
<tgt_lang> Translation 2: <hyp2>
<tgt_lang> Translation Refinement:

Figure 3: Prompt template used for translation and refinement.

6https://huggingface.co/meta-llama/Llama-3-8B-Instruct
7https://huggingface.co/mistralai/Mistral-Nemo-Instruct-2407
8We consider \( \mathcal{M}_S \) and \( \mathcal{M}_T \) to be the same LLM. For further discussion on cases where \( \mathcal{M}_S \) and \( \mathcal{M}_T \) differ, please refer to Appendix J.
tion. In a contrastive setting, we first fine-tune \( \mathcal{M}_S \) at sentence-level translation and then obtain sentence-level translation, referred as Sent2Sent\textsubscript{tuned}.

• Doc2Doc: As described in Section 2.1, we prompt \( \mathcal{M}_S \) to generate document-level translation. Similarly, Doc2Doc\textsubscript{tuned} refers to document-level translation from fine-tuned \( \mathcal{M}_S \) at document-level translation.

• SentRefine\textsubscript{sent}: It is sentence-level translation refinement by fine-tuning \( \mathcal{M}_T \) on Sent2Sent, similar to Chen et al. (2024c).

• DocRefine\textsubscript{sent}: It is document-level translation refinement by fine-tuning \( \mathcal{M}_T \) on Sent2Sent, similar to Koneru et al. (2024).

• DocRefine\textsubscript{doc}: It is also document-level translation refinement by fine-tuning \( \mathcal{M}_T \) on Doc2Doc.

Note that SentRefine\textsubscript{sent}, DocRefine\textsubscript{sent} and DocRefine\textsubscript{doc} all use one intermediate translation. Please refer to Figure 7 in Appendix K for detailed prompts. Differently, our approach uses both Sent2Sent and Doc2Doc as intermediate translations.

Evaluation Metrics. We report document-level COMET (d-COMET) scores proposed by Vernikos et al. (2022). Specifically, we apply reference-based metric wmt22-comet-da (Rei et al., 2022a). For other traditional evaluation metrics, including sentence-level COMET (s-COMET), document-level BLEU (d-BLEU), please refer to Appendix L.

Besides, we also report several additional metrics. 1) We follow Li et al. (2023) and Su et al. (2022) to compute coherence score (Coh.) using cosine similarity between the sentence embeddings of SimCSE (Gao et al., 2021) of the neighbouring sentences. 2) We report ALTI+ score (Ferrando et al., 2022; Dale et al., 2023; Wu et al., 2024b) to detect under-translation and hallucination issues in translation. 3) We follow Lyu et al. (2021) and compute LTCR score to measure lexical translation consistency. 4) We compute document-level perplexity (PPL) using GPT-2\footnote{https://huggingface.co/openai-community/gpt2} (Radford et al., 2019). 5) We report BlonDe (Jiang et al., 2022), which evaluates discourse phenomena via a set of automatically extracted features (Deutsch et al., 2023). Except for ALTI+, these metrics are document-level metrics. LTCR, BlonDe, and PPL are computed only for the \( X \rightarrow \mathrm{En} \) translation direction, while the other two metrics are applicable to all translation directions.

3.2 Main Results

Table 2 presents the performance comparison in d-COMET. From it, we observe:

• Extending the translation unit from sentence-level to document-level improves overall performance, as Doc2Doc outperforms Sent2Sent. This aligns with findings from related studies (Karpinska and Iyer, 2023). However, fine-tuned LLMs exhibit different performance trends. LLaMA-3-8B-Instruct shows similar performance for both Sent2Sent\textsubscript{tuned} and Doc2Doc\textsubscript{tuned}, while Mistral-Nemo-Instruct performs better with Doc2Doc\textsubscript{tuned} compared to Sent2Sent\textsubscript{tuned}.

• Refining with a single input, whether from Sent2Sent or Doc2Doc, leads to higher COMET scores. However, this refinement shows little to no improvement over the performance of directly fine-tuned LLMs.

• Our refinement approach, based on the two intermediate translations Sent2Sent and Doc2Doc, significantly improves translation performance across all language pairs. It achieves COMET score improvements of 2.73 and 1.80 on LLaMA-3-8B-Instruct, and 2.21 and 1.79 on Mistral-Nemo-Instruct. Our approach also outperforms other baselines, including both refining with single translations and directly fine-tuning, demonstrating the effectiveness of our proposed approach.

• Lastly, disabling the quality-aware fine-tuning stage results in a performance drop, highlighting the effectiveness of our fine-tuning strategy. Additionally, compared to SentRefine\textsubscript{sent}, DocRefine\textsubscript{sent}, and DocRefine\textsubscript{doc}, refinement using two intermediate translations outperforms refinements with just one.

Table 3 presents the performance on several additional metrics when LLaMA-3-8B-Instruct is used. The results show that, except for ALTI+, document-level translation and refinement systems outperform their sentence-level counterparts. By combining the strengths of Sent2Sent and Doc2Doc translations, our approach achieves the best performance across all five metrics.
<table>
  <tr>
    <th>#</th>
    <th>System</th>
    <th colspan="5">X→En</th>
    <th colspan="5">En→X</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th>De→</th>
    <th>Es→</th>
    <th>Ru→</th>
    <th>Fr→</th>
    <th>Zh→</th>
    <th>De→</th>
    <th>Es→</th>
    <th>Ru→</th>
    <th>Fr→</th>
    <th>Zh→</th>
    <th></th>
  </tr>
  <tr>
    <td colspan="13">LLaMA-3-8B-Instruct</td>
  </tr>
  <tr>
    <td>1</td>
    <td>Sent2Sent</td>
    <td>85.97</td>
    <td>86.62</td>
    <td>81.63</td>
    <td>84.43</td>
    <td>82.18</td>
    <td>82.50</td>
    <td>85.02</td>
    <td>80.97</td>
    <td>82.89</td>
    <td>76.80</td>
    <td>82.90</td>
  </tr>
  <tr>
    <td>2</td>
    <td>Sent2Sent<sub>tuned</sub></td>
    <td>87.94</td>
    <td>87.46</td>
    <td>81.98</td>
    <td>86.46</td>
    <td>84.18</td>
    <td>85.42</td>
    <td>86.11</td>
    <td>80.88</td>
    <td>84.30</td>
    <td>82.84</td>
    <td>84.76</td>
  </tr>
  <tr>
    <td>3</td>
    <td>Doc2Doc</td>
    <td>87.05</td>
    <td>87.21</td>
    <td>81.07</td>
    <td>85.40</td>
    <td>83.60</td>
    <td>83.35</td>
    <td>85.36</td>
    <td>80.18</td>
    <td>83.14</td>
    <td>81.89</td>
    <td>83.83</td>
  </tr>
  <tr>
    <td>4</td>
    <td>Doc2Doc<sub>tuned</sub></td>
    <td>87.82</td>
    <td>88.04</td>
    <td>81.25</td>
    <td>86.37</td>
    <td>84.88</td>
    <td>85.45</td>
    <td>85.61</td>
    <td>81.06</td>
    <td>84.63</td>
    <td>82.18</td>
    <td>84.73</td>
  </tr>
  <tr>
    <td>5</td>
    <td>SentRefine<sub>sent</sub></td>
    <td>83.70</td>
    <td>87.94</td>
    <td>82.64</td>
    <td>85.98</td>
    <td>84.78</td>
    <td>85.21</td>
    <td>86.34</td>
    <td>83.74</td>
    <td>84.57</td>
    <td>82.95</td>
    <td>84.72</td>
  </tr>
  <tr>
    <td>6</td>
    <td>DocRefine<sub>sent</sub></td>
    <td>87.42</td>
    <td>87.98</td>
    <td>81.16</td>
    <td>86.56</td>
    <td>85.06</td>
    <td>85.38</td>
    <td>86.32</td>
    <td>80.39</td>
    <td>84.43</td>
    <td>82.61</td>
    <td>84.73</td>
  </tr>
  <tr>
    <td>7</td>
    <td>DocRefine<sub>doc</sub></td>
    <td>87.71</td>
    <td>88.06</td>
    <td>82.73</td>
    <td>86.32</td>
    <td>84.99</td>
    <td>85.07</td>
    <td>86.49</td>
    <td>83.16</td>
    <td>84.73</td>
    <td>82.70</td>
    <td>85.19</td>
  </tr>
  <tr>
    <td>8</td>
    <td><b>Ours</b></td>
    <td><b>88.14</b></td>
    <td><b>88.42</b></td>
    <td><b>82.75</b></td>
    <td><b>86.69</b></td>
    <td><b>85.39</b></td>
    <td><b>86.05</b></td>
    <td><b>86.86</b></td>
    <td><b>83.85</b></td>
    <td><b>84.84</b></td>
    <td><b>83.35</b></td>
    <td><b>85.63</b></td>
  </tr>
  <tr>
    <td>9</td>
    <td>- QA</td>
    <td>88.02</td>
    <td>88.35</td>
    <td>82.63</td>
    <td>86.53</td>
    <td>85.09</td>
    <td>85.70</td>
    <td>86.60</td>
    <td>83.17</td>
    <td>84.48</td>
    <td>82.98</td>
    <td>85.36</td>
  </tr>
</table>

<table>
  <tr>
    <th>#</th>
    <th>System</th>
    <th>Coh.</th>
    <th>ALTI+↑</th>
    <th>LTCR↑</th>
    <th>PPL↓</th>
    <th>BlonDe↑</th>
  </tr>
  <tr>
    <td>1</td>
    <td>Sent2Sent</td>
    <td>56.17</td>
    <td>42.57</td>
    <td>57.23</td>
    <td>32.86</td>
    <td>48.49</td>
  </tr>
  <tr>
    <td>2</td>
    <td>Sent2Sent<sub>tuned</sub></td>
    <td>56.23</td>
    <td>42.94</td>
    <td>60.45</td>
    <td>30.34</td>
    <td>58.61</td>
  </tr>
  <tr>
    <td>3</td>
    <td>Doc2Doc</td>
    <td>62.28</td>
    <td>40.04</td>
    <td>61.25</td>
    <td>31.85</td>
    <td>51.30</td>
  </tr>
  <tr>
    <td>4</td>
    <td>Doc2Doc<sub>tuned</sub></td>
    <td>63.42</td>
    <td>42.99</td>
    <td>64.99</td>
    <td>31.58</td>
    <td>57.86</td>
  </tr>
  <tr>
    <td>5</td>
    <td>SentRefine<sub>sent</sub></td>
    <td>64.27</td>
    <td>43.09</td>
    <td>60.08</td>
    <td>32.14</td>
    <td>57.47</td>
  </tr>
  <tr>
    <td>6</td>
    <td>DocRefine<sub>sent</sub></td>
    <td>64.95</td>
    <td>43.00</td>
    <td>63.62</td>
    <td>30.13</td>
    <td>58.69</td>
  </tr>
  <tr>
    <td>7</td>
    <td>DocRefine<sub>doc</sub></td>
    <td>65.09</td>
    <td>42.80</td>
    <td>63.68</td>
    <td>31.62</td>
    <td>59.01</td>
  </tr>
  <tr>
    <td>8</td>
    <td><b>Ours</b></td>
    <td><b>67.12</b></td>
    <td><b>43.53</b></td>
    <td><b>66.57</b></td>
    <td><b>26.51</b></td>
    <td><b>59.86</b></td>
  </tr>
  <tr>
    <td>9</td>
    <td>- QA</td>
    <td>66.07</td>
    <td>43.06</td>
    <td>65.98</td>
    <td>31.64</td>
    <td>59.57</td>
  </tr>
</table>

Table 2: Performance in document-level COMET (d-COMET) score. Bold scores represent the highest performance, while underlined scores indicate the second-best performance. -QA indicates disabling the quality-aware fine-tuning stage. Scores of our approach (System #8 and #9) that exceed the highest value in the baselines (System #1 ~ #7) by ≥ 0.4 points are highlighted with dark red boxes, while those that are positive but < 0.4 points higher are highlighted with shallow red boxes.

Table 3: Averaged performance of LLaMA-3-8B-Instruct in additional metrics.

4 Discussion

4.1 Refining Translations by GPT and NLLB

To further evaluate our approach, we use our fine-tuned LLMs (i.e., Ours in Table 2) to refine translations from GPT-4o-mini (OpenAI, 2024) and NLLB (NLLB Team et al., 2024), the state-of-the-art NMT system. As shown in the upper part of Table 4, refining GPT-4o-mini’s output with one intermediate translation yields limited improvement (#4/#5 vs. #2). In contrast, using two intermediate translations increase the COMET score by 0.22 (#6 vs. #2), suggesting that using two intermediate translations is more effective. Our two fine-tuned LLMs behave differently: LLaMA-3-8B-Instruct experiences a slight drop (85.62 to 85.46), while Mistral-Nemo-Instruct successfully improves performance (85.62 to 86.31). For detailed s-COMET scores, please refer to Appendix L.

Additionally, we refine NLLB-generated translations using our fine-tuned LLMs. Since NLLB does not support Doc2Doc translation, we simplify the process by treating Doc2Doc translation as equivalent to Sent2Sent translation (i.e., the two intermediate translations are identical) for refinement with our fine-tuned LLMs. As shown in the lower part of Table 4, even though the two intermediate translations are identical, LLaMA-3-8B-Instruct still shows a slight improvement (85.17 to 85.29), while Mistral-Nemo-Instruct demonstrates a more substantial improvement, (85.17 to 86.13).

Furthermore, we prompt LLMs to refine translations produced by other LLMs. For detailed experimental results, please refer to Appendix J.
<table>
  <tr>
    <th>#</th>
    <th>System</th>
    <th colspan="5">X→En</th>
    <th colspan="5">En→X</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th>De→</th>
    <th>Es→</th>
    <th>Ru→</th>
    <th>Fr→</th>
    <th>Zh→</th>
    <th>→ De</th>
    <th>→ Es</th>
    <th>→ Ru</th>
    <th>→ Fr</th>
    <th>→ Zh</th>
    <th></th>
  </tr>
  <tr>
    <td colspan="13">GPT Translation & Refining GPT Translation</td>
  </tr>
  <tr>
    <td>1</td>
    <td>GPT Sent2Sent</td>
    <td>86.49</td>
    <td>86.53</td>
    <td>82.43</td>
    <td>84.73</td>
    <td>83.98</td>
    <td>85.96</td>
    <td>86.52</td>
    <td>85.28</td>
    <td>84.97</td>
    <td>83.70</td>
    <td>85.06</td>
  </tr>
  <tr>
    <td>2</td>
    <td>GPT Doc2Doc</td>
    <td>87.00</td>
    <td>87.12</td>
    <td>83.71</td>
    <td>85.64</td>
    <td>84.75</td>
    <td>86.30</td>
    <td>86.76</td>
    <td>85.59</td>
    <td>85.23</td>
    <td>84.07</td>
    <td>85.62</td>
  </tr>
  <tr>
    <td>3</td>
    <td>GPT SentRefine<sup>sent</sup></td>
    <td>86.86</td>
    <td>86.89</td>
    <td>83.37</td>
    <td>85.70</td>
    <td>83.37</td>
    <td>85.32</td>
    <td>86.43</td>
    <td>85.42</td>
    <td>84.30</td>
    <td>83.97</td>
    <td>84.96</td>
  </tr>
  <tr>
    <td>4</td>
    <td>GPT DocRefine<sup>sent</sup></td>
    <td>87.03</td>
    <td>87.26</td>
    <td>83.23</td>
    <td>85.77</td>
    <td>84.29</td>
    <td>86.57</td>
    <td>87.04</td>
    <td>86.04</td>
    <td>85.40</td>
    <td>84.07</td>
    <td>85.67</td>
  </tr>
  <tr>
    <td>5</td>
    <td>GPT DocRefine<sup>doc</sup></td>
    <td>87.04</td>
    <td>87.29</td>
    <td>83.27</td>
    <td>85.63</td>
    <td>84.41</td>
    <td>86.37</td>
    <td>87.03</td>
    <td>86.14</td>
    <td>85.43</td>
    <td>83.93</td>
    <td>85.62</td>
  </tr>
  <tr>
    <td>6</td>
    <td>GPT DocRefine<sup>doc+sent</sup></td>
    <td>87.39</td>
    <td>87.65</td>
    <td>83.44</td>
    <td>85.77</td>
    <td>84.78</td>
    <td>86.61</td>
    <td>86.96</td>
    <td>86.16</td>
    <td>85.46</td>
    <td>84.13</td>
    <td>85.84</td>
  </tr>
  <tr>
    <td>7</td>
    <td>L-DocRefine<sup>doc+sent</sup></td>
    <td><b>87.88</b></td>
    <td><b>88.15</b></td>
    <td><b>82.07</b></td>
    <td><b>86.57</b></td>
    <td><b>85.22</b></td>
    <td><b>86.31</b></td>
    <td><b>86.09</b></td>
    <td><b>83.66</b></td>
    <td><b>85.28</b></td>
    <td><b>83.32</b></td>
    <td>85.46</td>
  </tr>
  <tr>
    <td>8</td>
    <td>M-DocRefine<sup>doc+sent</sup></td>
    <td><b>88.14</b></td>
    <td><b>88.22</b></td>
    <td><b>84.39</b></td>
    <td><b>86.73</b></td>
    <td><b>85.48</b></td>
    <td><b>86.88</b></td>
    <td><b>87.20</b></td>
    <td><b>86.20</b></td>
    <td><b>85.69</b></td>
    <td><b>84.12</b></td>
    <td><b>86.31</b></td>
  </tr>
  <tr>
    <td colspan="13">NL1B Translation & Refining NL1B Translation</td>
  </tr>
  <tr>
    <td>9</td>
    <td>NLLB Sent2Sent</td>
    <td>86.79</td>
    <td>87.55</td>
    <td>83.22</td>
    <td>85.62</td>
    <td>83.17</td>
    <td>84.93</td>
    <td>86.22</td>
    <td>85.47</td>
    <td>84.60</td>
    <td>84.17</td>
    <td>85.17</td>
  </tr>
  <tr>
    <td>10</td>
    <td>L-DocRefine<sup>doc+sent</sup></td>
    <td><b>87.85</b></td>
    <td><b>88.41</b></td>
    <td><b>81.65</b></td>
    <td><b>86.51</b></td>
    <td><b>85.00</b></td>
    <td><b>86.20</b></td>
    <td><b>86.45</b></td>
    <td><b>83.03</b></td>
    <td><b>83.97</b></td>
    <td><b>82.80</b></td>
    <td>85.29</td>
  </tr>
  <tr>
    <td>11</td>
    <td>M-DocRefine<sup>doc+sent</sup></td>
    <td><b>88.10</b></td>
    <td><b>88.66</b></td>
    <td><b>84.44</b></td>
    <td><b>86.17</b></td>
    <td><b>85.48</b></td>
    <td><b>86.81</b></td>
    <td><b>86.74</b></td>
    <td><b>85.49</b></td>
    <td><b>85.34</b></td>
    <td><b>84.08</b></td>
    <td><b>86.13</b></td>
  </tr>
</table>

Table 4: Performance in d-COMET when refining translations from GPT-4o-mini (upper) and NLLB (lower). For the GPT-based refinement systems, we use the same prompt templates as those used in our approach, but without fine-tuning (System #3~#6). L-* and M-* denote our fine-tuned LLAMA-3-8B-Instrcut and Mistral-Nemo-Instruct (i.e., Ours in Table 2), respectively.

<table>
  <tr>
    <th>Stage1</th>
    <th>Stage2</th>
    <th>De→En</th>
    <th>En→De</th>
    <th>Zh→En</th>
    <th>En→Zh</th>
  </tr>
  <tr>
    <td>naïve</td>
    <td>QA</td>
    <td>88.14</td>
    <td>86.05</td>
    <td>85.39</td>
    <td>83.35</td>
  </tr>
  <tr>
    <td>naïve</td>
    <td>-</td>
    <td>88.02</td>
    <td>85.70</td>
    <td>85.09</td>
    <td>82.98</td>
  </tr>
  <tr>
    <td>QA</td>
    <td>-</td>
    <td>87.76</td>
    <td>85.60</td>
    <td>84.88</td>
    <td>83.05</td>
  </tr>
  <tr>
    <td>naïve</td>
    <td>naïve</td>
    <td>87.75</td>
    <td>85.91</td>
    <td>83.98</td>
    <td>82.14</td>
  </tr>
</table>

Table 5: Performance comparison when using different fine-tuning strategies. QA indicates quality-aware fine-tuning.

<table>
  <tr>
    <th>Our Approach</th>
    <th>De→En</th>
    <th>En→De</th>
  </tr>
  <tr>
    <td>w/ preventing position bias</td>
    <td>88.14</td>
    <td>86.05</td>
  </tr>
  <tr>
    <td>w/o preventing position bias</td>
    <td>87.60</td>
    <td>85.55</td>
  </tr>
</table>

Table 6: Performance comparison with and without preventing position bias.

ates translation quality with high accuracy. As an alternative, we consider using BLEU for computing these weights due to its lower computational overhead. Specifically, we define the weight \( w_i \) as:

\[
w_i = 1 + \lambda (\max(\text{BLEU}(y_i, r_i), \\
\text{BLEU}(z_i, r_i)) - \epsilon),
\]

where BLEU denotes the BLEU score between a generated output and its corresponding reference.

Table 7 compares the performance of LLAMA-3-8B-Instruct using COMET-based versus BLEU-based sentence-level weighting. While BLEU-based weighting yields slightly lower performance compared to COMET, it remains competitive and offers a more resource-efficient alternative.

For more details in comparison with other weighting variants, including instance-level weighting, please refer to Appendix H.

4.3 Effect of Preventing Position Bias

To prevent introducing position bias, \( <hyp1> \) in the prompt template can be either Sent2Sent or Doc2Doc translation. To examine its effect, we compare it with a version where \( <hyp1> \) is always set to Sent2Sent and \( <hyp2> \) is set to Doc2Doc. As shown in Table 6, preventing position bias leads to a significant boost in performance.

4.4 Effect of Sentence-Level Weighting Using BLEU Scores

In Eq. 1, sentence-level weights are computed using COMET, a reference-based metric that evaluates translation quality with high accuracy. As an alternative, we consider using BLEU for computing these weights due to its lower computational overhead. Specifically, we define the weight \( w_i \) as:

\[
w_i = 1 + \lambda (\max(\text{BLEU}(y_i, r_i), \\
\text{BLEU}(z_i, r_i)) - \epsilon),
\]

where BLEU denotes the BLEU score between a generated output and its corresponding reference.

Table 7 compares the performance of LLAMA-3-8B-Instruct using COMET-based versus BLEU-based sentence-level weighting. While BLEU-based weighting yields slightly lower performance compared to COMET, it remains competitive and offers a more resource-efficient alternative.

For more details in comparison with other weighting variants, including instance-level weighting, please refer to Appendix H.

4.5 Comparison to Reranking

To demonstrate the effectiveness of our approach in combining Sent2Sent and Doc2Doc translations, we compare it with two other strategies:

1) Reranking, which chooses the translation with the higher reference-free COMETKiwi score\( ^{10} \) (Rei et al., 2022b) for each source sentence (He et al., 2024; Farinhas et al., 2023);

\( ^{10}\)wmt22-cometkiwi-da : https://huggingface.co/Unbabel/wmt22-cometkiwi-da
<table>
  <tr>
    <th rowspan="2">Weighting</th>
    <th colspan="5">X→En</th>
    <th colspan="5">En→X</th>
    <th rowspan="2">Avg.</th>
  </tr>
  <tr>
    <th>De→</th><th>Es→</th><th>Ru→</th><th>Fr→</th><th>Zh→</th>
    <th>→ De</th><th>→ Es</th><th>→ Ru</th><th>→ Fr</th><th>→ Zh</th>
  </tr>
  <tr>
    <td>COMET</td>
    <td>88.14</td><td>88.42</td><td>82.75</td><td>86.69</td><td>85.39</td>
    <td>86.05</td><td>86.86</td><td>83.85</td><td>84.84</td><td>83.35</td>
    <td>85.63</td>
  </tr>
  <tr>
    <td>BLEU</td>
    <td>88.09</td><td>88.17</td><td>82.65</td><td>86.43</td><td>85.07</td>
    <td>86.03</td><td>86.67</td><td>83.56</td><td>84.83</td><td>83.19</td>
    <td>85.47</td>
  </tr>
</table>

Table 7: Comparison of d-COMET scores using COMET-based and BLEU-based sentence-level weighting methods.

<table>
  <tr>
    <th>T1</th><th>T2</th><th>Strategy</th><th>De→En</th><th>En → De</th>
  </tr>
  <tr>
    <td>S2S</td><td>D2D</td><td>Rerank</td><td>86.96</td><td>84.20</td>
  </tr>
  <tr>
    <td>S2S</td><td>D2D</td><td>Rerank + Refine</td><td>87.74</td><td>85.56</td>
  </tr>
  <tr>
    <td>S2S</td><td>D2D</td><td>Ours</td><td>88.02</td><td>86.05</td>
  </tr>
  <tr>
    <td>S2S</td><td>S2S</td><td>Rerank</td><td>86.16</td><td>83.07</td>
  </tr>
  <tr>
    <td>S2S</td><td>S2S</td><td>Rerank + Refine</td><td>87.63</td><td>85.58</td>
  </tr>
  <tr>
    <td>S2S</td><td>S2S</td><td>Ours</td><td>87.76</td><td>86.04</td>
  </tr>
  <tr>
    <td>D2D</td><td>D2D</td><td>Rerank</td><td>86.99</td><td>83.30</td>
  </tr>
  <tr>
    <td>D2D</td><td>D2D</td><td>Rerank + Refine</td><td>87.50</td><td>85.65</td>
  </tr>
  <tr>
    <td>D2D</td><td>D2D</td><td>Ours</td><td>87.61</td><td>85.69</td>
  </tr>
</table>

Table 8: Comparison with reranking and reranking + refining. T1/T2 refers to intermediate translation 1/2.

2) Reranking + Refining, which firstly selects better translation (i.e., Strategy 1) and further refines the selected translation using DocRefine_{doc} and DocRefine_{sent}, similar to Vernikos and Popescu-Belis (2024).

As shown in Table 8, our approach outperforms the other two strategies in combining intermediate translations. Furthermore, our approach benefits from the diversity of intermediate translations, achieving the best performance when T1 and T2 originate from Sent2Sent and Doc2Doc 11, respectively. This illustrates that our approach effectively integrates the advantages of both translations. For more details, please refer to Appendix M.

4.6 GPT-based Error Annotating

Following Wu et al. (2024a), we identify translation errors at both sentence- and document-level. Please refer to Appendix N for detailed prompts. Specifically, we use GPT-4o-mini to detect sentence-level issues such as mistranslation, over-translation (including additions), and under-translation (including omissions). Additionally, we address document-level issues related to cohesion, coherence and inconsistent style (including the use of multiple terms for the same concept). Figure 4 shows the results for De→En translation. It highlights that: 1) our approach addresses all major issues observed in Doc2Doc translation; and 2) it improves most of the issues in Sent2Sent translation, with a trade-off in performance related to under-translation (including omissions). The two highlights suggest that our approach effectively combines the strengths of both Sent2Sent and Doc2Doc translations.

Figure 4: Counts of error types on De→En translation.

![Counts of error types on De→En translation.](page_1012_370_420_246.png)

11To generate diverse S2S and D2D translations, we set do_sample to true, temperature to 0.3 and top_p to 0.7.

5 Related Work

5.1 LLM-based Translation Refinement

Current approaches to LLM-based translation refinement can be categorized into two types: prompt engineering and supervised fine-tuning (SFT).

In prompt engineering, Chen et al. (2024c) propose a method where ChatGPT iteratively self-corrects translations. Raunak et al. (2023) explore using GPT-4 for automatically post-editing (APE) of neural machine translation (NMT) outputs. Farinhas et al. (2023) generate multiple hypotheses and experiment with various ensemble methods. Feng et al. (2024b) introduce Translate-Estimate-Refine framework, leveraging LLMs for self-refinement. Xu et al. (2023, 2024b) prompt LLMs to generate intermediate translations, and then provide self-feedback to optimize the final output. Yang et al. (2023) examine human intervention in LLM inference for MT tasks. Chen et al. (2024b,a) explore LLMs’ self-reflective and contextual understanding abilities. Berger et al. (2024) prompt LLMs to edit translations with human error markings. Chen et al. (2024d) apply retrieval-augmented generation (RAG) to enhance translation faithfulness. All of these studies focus on sentence-level refinement.

In SFT, Ki and Carpuat (2024) train LLMs using source sentences, intermediate translations and error annotations. Alves et al. (2024) fine-tune LLMs for translation-related tasks including APE,
and train a model called Tower-Instruct. Feng et al. (2024a) propose hierarchical fine-tuning, grouping instances by refinement difficulty for multi-stage training. While these studies focus on sentence-level refinement, Koneru et al. (2024) extend refinement by incorporating document-level context. Building on this, our work further extends refinement to the entire document level.

5.2 LLM-based Document-level Machine Translation

Current LLM-based document-level machine translation (DMT) approaches can also be categorized into two types: prompt engineering and SFT.

In prompt engineering, Wang et al. (2023) firstly prompt GPTs for DMT. Karpinska and Iyyer (2023) evaluate GPT-3.5 on novel translation tasks. Cui et al. (2024) apply RAG to select relevant contextual examples. Wang et al. (2025) and Guo et al. (2025) introduce agents with memory mechanism to capture long-range dependencies to enhance consistency and accuracy. Briakou et al. (2024) frame DMT as a multi-turn process with a step for refinement. Sun et al. (2024) employ instruction-tuned LLMs and use GPT-4 for document assessment.

On the other hand, SFT approaches enhance LLMs ability for DMT by leveraging tailored training strategies. Li et al. (2024) integrate sentence- and document-level instructions. Wu et al. (2024a) introduce a multi-stage fine-tuning approach, first fine-tuning on monolingual documents, then on parallel documents. Stap et al. (2024) fine-tune LLMs on sentence-level instances and evaluate DMT. Lyu et al. (2024) present a decoding-enhanced, multi-phase prompt tuning method.

6 Conclusion

In this paper, we have proposed a novel approach to refine Doc2Doc translation by combining the strengths of both sentence-level and document-level translations. Our approach employs an enhanced fine-tuning with quality awareness to improve the performance of large language models (LLMs). Experimental results across ten document-level translation tasks show substantial improvements in translation quality, coherence, and consistency for a variety of language pairs.

Acknowledgments

The authors would like to thank the anonymous reviewers for their constructive feedback. This work was supported by the National Natural Science Foundation of China (Grant No. 62036004, 62261160648) and the Priority Academic Program Development of Jiangsu Higher Education Institutions.

Limitations

Our experiments are primarily conducted on a news dataset, which may not fully represent LLMs’ performance in other specific domains and other non-English translation directions. Moreover, we train one model for one specific translation direction, leading to huge computational cost. The model may be biased to refining texts of a specific style and may perform worse when refining texts in other styles. Further research may enhance the multilingual performance of LLMs or apply pairwise preference-based optimization tuning.

References

Duarte M. Alves, José Pombal, Nuno M. Guerreiro, Pedro H. Martins, João Alves, Amin Farajian, Ben Peters, Ricardo Rei, Patrick Fernandes, Sweta Agrawal, Pierre Colombo, José G. C. de Souza, and André F. T. Martins. 2024. Tower: An open multilingual large language model for translation-related tasks. CoRR, abs/2402.17733.

Nathaniel Berger, Stefan Riezler, Miriam Exel, and Matthias Huck. 2024. Prompting large language models with human error markings for self-correcting machine translation. In Proceedings of EAMT, pages 636–646.

Eleftheria Briakou, Jiaming Luo, Colin Cherry, and Markus Freitag. 2024. Translating step-by-step: Decomposing the translation process for improved translation quality of long-form texts. In Proceedings of WMT, pages 1301–1317.

Andong Chen, Kehai Chen, Yang Xiang, Xuefeng Bai, Muyun Yang, Yang Feng, Tiejun Zhao, and Min zhang. 2024a. Lim-based translation inference with iterative bilingual understanding. CoRR, abs/2410.12543.

Andong Chen, Lianzhang Lou, Kehai Chen, Xuefeng Bai, Yang Xiang, Muyun Yang, Tiejun Zhao, and Min Zhang. 2024b. DUAL-REFLECT: Enhancing large language models for reflective translation through dual learning feedback mechanisms. In Proceedings of ACL (Short Papers), pages 693–704.

Pinzhen Chen, Zhicheng Guo, Barry Haddow, and Kenneth Heafield. 2024c. Iterative translation refinement with large language models. In Proceedings of EACL, pages 181–190.
Shangfeng Chen, Xiayang Shi, Pu Li, Yinyin Li, and Jingjing Liu. 2024d. Refining translations with llms: A constraint-aware iterative prompting approach. CoRR, abs/2411.08348.

Menglong Cui, Jiangcun Du, Shaolin Zhu, and Deyi Xiong. 2024. Efficiently exploring large language models for document-level machine translation with in-context learning. In Findings of ACL, pages 10885–10897.

David Dale, Elena Voita, Loic Barrault, and Marta R.-Costa-jussà. 2023. Detecting and mitigating hallucinations in machine translation: Model internal workings alone do well, sentence similarity Even better. In Proceedings of ACL, pages 36–50.

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized llms. In Proceedings of NeurIPS, pages 10088–10115.

Daniel Deutsch, Juraj Juraska, Mara Finkelstein, and Markus Freitag. 2023. Training and meta-evaluating machine translation evaluation metrics at the paragraph level. In Proceedings of WMT, pages 996–1013.

Qingkai Fang and Yang Feng. 2023. Understanding and bridging the modality gap for speech translation. In Proceedings of ACL, pages 15864–15881.

António Farinhas, José de Souza, and Andre Martins. 2023. An empirical study of translation hypothesis ensembling with large language models. In Proceedings of EMNLP, pages 11956–11970.

Zhaopeng Feng, Ruizhe Chen, Yan Zhang, Zijie Meng, and Zuozhu Liu. 2024a. Ladder: A model-agnostic framework boosting LLM-based machine translation to the next level. In Proceedings of EMNLP, pages 15377–15393.

Zhaopeng Feng, Yan Zhang, Hao Li, Wenqiang Liu, Jun Lang, Yang Feng, Jian Wu, and Zuozhu Liu. 2024b. Tear: Improving llm-based machine translation with systematic self-refinement. CoRR, abs/2402.16379.

Javier Ferrando, Gerard I. Gállego, Belen Alastrauey, Carlos Escolano, and Marta R. Costa-jussà. 2022. Towards opening the black box of neural machine translation: Source and target interpretations of the transformer. In Proceedings of the EMNLP, pages 8756–8769.

Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence embeddings. In Proceedings of EMNLP, pages 6894–6910.

Xavier Garcia, Yamini Bansal, Colin Cherry, George Foster, Maxim Krikun, Melvin Johnson, and Orhan Firat. 2023. The unreasonable effectiveness of few-shot learning for machine translation. In Proceedings of ICML, pages 10867–10878.

Jiaxin Guo, Yuanchang Luo, Daimeng Wei, Ling Zhang, Zongyao Li, Hengchao Shang, Zhiqiang Rao, Shaojun Li, Jinlong Yang, Zhanglin Wu, and Hao Yang. 2025. Doc-guided sent2sent++: A sent2sent++ agent with doc-guided memory for document-level machine translation. CoRR, abs/2501.08523.

Zhiwei He, Tian Liang, Wenxiang Jiao, Zhuosheng Zhang, Yujiu Yang, Rui Wang, Zhaopeng Tu, Shuming Shi, and Xing Wang. 2024. Exploring human-like translation strategy with large language models. Transactions of the Association for Computational Linguistics, 12:229–246.

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. In Proceedings of ICLR.

Yuchen Jiang, Tianyu Liu, Shuming Ma, Dongdong Zhang, Jian Yang, Haoyang Huang, Rico Sennrich, Ryan Cotterell, Mrinmaya Sachan, and Ming Zhou. 2022. BlonDe: An automatic evaluation metric for document-level machine translation. In Proceedings of NAACL: HLT, pages 1550–1565.

Marzena Karpińska and Mohit Iyyer. 2023. Large language models effectively leverage document-level context for literary translation, but critical errors persist. In Proceedings of WMT, pages 419–451.

Dayeon Ki and Marine Carpuat. 2024. Guiding large language models to post-edit machine translation with error annotations. In Findings of NAACL, pages 4253–4273.

Sai Koneru, Miriam Exel, Matthias Huck, and Jan Niehues. 2024. Contextual refinement of translations: Large language models for sentence and document-level post-editing. In Proceedings of NAACL: HLT, pages 2711–2725.

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of SOSP, page 611–626.

Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. 2023. Contrastive decoding: Open-ended text generation as optimization. In Proceedings of ACL, pages 12286–12312.

Yachao Li, Junhui Li, Jing Jiang, and Min Zhang. 2024. Enhancing document-level translation of large language model via translation mixed-instructions. CoRR, abs/2401.08088.

Pierre Lison and Serge Bibauw. 2017. Not all dialogues are created equal: Instance weighting for neural conversational models. In Proceedings of SIGDIAL, pages 384–394.
Lei Liu and Min Zhu. 2023. Bertalign: Improved word embedding-based sentence alignment for chinese–english parallel corpora of literary texts. Digital Scholarship in the Humanities, 38:621–634.

Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173.

Xinglin Lyu, Junhui Li, Zhengxian Gong, and Min Zhang. 2021. Encouraging lexical translation consistency for document-level neural machine translation. In Proceedings of EMNLP, pages 3265–3277.

Xinglin Lyu, Junhui Li, Yanqing Zhao, Min Zhang, Daimeng Wei, Shimin Tao, Hao Yang, and Min Zhang. 2024. DeMPT: Decoding-enhanced multi-phase prompt tuning for making LLMs be better context-aware translators. In Proceedings of EMNLP, pages 20280–20295.

Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-refine: Iterative refinement with self-feedback. In Proceedings of NeurIPS, pages 46534–46594.

Meta. 2024. Introducing meta llama 3: The most capable openly available llm to date. https://ai.meta.com/blog/meta-llama-3/.

MistralAI. 2024. Mistral nemo. https://mistral.ai/news/mistral-nemo/.

NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loïc Barrault, Gabriel Mejía-González, Prangthip Hansanti, John Hoffman, Semañar Jarret, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Rogers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2024. No language left behind: Scaling human-centered machine translation. CoRR, abs/2207.04672.

OpenAI. 2024. Gpt-4o mini: advancing cost-efficient intelligence. https://openai.com/research/gpt-4.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog.

Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Xuxiong He. 2020. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of ACM SIGKDD, page 3505–3506.

Vikas Raunak, Amr Sharaf, Yiren Wang, Hany Awadalla, and Arul Menezes. 2023. Leveraging GPT-4 for automatic translation post-editing. In Findings of EMNLP, pages 12009–12024.

Ricardo Rei, José GC De Souza, Duarte Alves, Chrysoúla Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André FT Martins. 2022a. Comet-22: Unbabel-ist 2022 submission for the metrics shared task. In Proceedings of WMT, pages 578–585.

Ricardo Rei, Marcos Treviso, Nuno M. Guerreiro, Chrysoúla Zerva, Ana C Farinha, Christine Maroti, José G. C. de Souza, Taisiya Glushkova, Duarte Alves, Luisa Coheur, Alon Lavie, and André F. T. Martins. 2022b. CometKiwi: IST-unbabel 2022 submission for the quality estimation shared task. In Proceedings of WMT, pages 634–645.

David Stap, Eva Hasler, Bill Byrne, Christof Monz, and Ke Tran. 2024. The fine-tuning paradox: Boosting translation quality without sacrificing LLM abilities. In Proceedings of ACL, pages 6189–6206.

Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Lingpeng Kong, and Nigel Collier. 2022. A contrastive framework for neural text generation. In Proceedings of NeurIPS, pages 21548–21561.

Yirong Sun, Dawei Zhu, Yanjun Chen, Erjia Xiao, Xinghao Chen, and Xiaoyu Shen. 2024. Instruction-tuned llms succeed in document-level mt without fine-tuning – but bleu turns a blind eye. CoRR, abs/2410.20941.

Giorgos Vernikos and Andrei Popescu-Belis. 2024. Don’t rank, combine! combining machine translation hypotheses using quality estimation. In Proceedings of ACL, pages 12087–12105.

Giorgos Vernikos, Brian Thompson, Prashant Mathur, and Marcello Federico. 2022. Embarrassingly easy document-level MT metrics: How to convert any pretrained metric into a document-level metric. In Proceedings of WMT, pages 118–128.

Longyue Wang, Chenyang Lyu, Tianbo Ji, Zhirui Zhang, Dian Yu, Shuming Shi, and Zhaopeng Tu. 2023. Document-level machine translation with large language models. In Proceedings of EMNLP, pages 16646–16661.

Yutong Wang, Jiali Zeng, Xuebo Liu, Derek F. Wong, Fandong Meng, Jie Zhou, and Min Zhang. 2025. Delta: An online document-level translation agent based on multi-level memory. In Proceedings of ICLR.
Minghao Wu, Thuy-Trang Vu, Lizhen Qu, George Foster, and Gholamreza Haffari. 2024a. Adapting large language models for document-level machine translation. CoRR, abs/2401.06468.

Qiyu Wu, Masaaki Nagata, Zhongtao Miao, and Yoshimasa Tsuruoka. 2024b. Word alignment as preference for machine translation. In Proceedings of EMNLP, pages 3223–3239.

Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. 2024a. Contrastive preference optimization: Pushing the boundaries of LLM performance in machine translation. In Proceedings of ICML, pages 55204–55224.

Wenda Xu, Daniel Deutsch, Mara Finkelstein, Juraj Juraska, Biao Zhang, Zhongtao Liu, William Yang Wang, Lei Li, and Markus Freitag. 2024b. LLMR-fine: Pinpointing and refining large language models via fine-grained actionable feedback. In Findings of NAACL: HLT, pages 1429–1445.

Wenda Xu, Danqing Wang, Liangming Pan, Zhenqiao Song, Markus Freitag, William Wang, and Lei Li. 2023. INSTRUCTSCORE: Towards explainable text generation evaluation with automatic feedback. In Proceedings of EMNLP, pages 5967–5994.

Xinyi Yang, Runzhe Zhan, Derek F. Wong, Junchao Wu, and Lidia S. Chao. 2023. Human-in-the-loop machine translation with large language model. In Proceedings of MTSummit, pages 88–98.

Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, and Zheyuan Luo. 2024. LlamaFactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of ACL, pages 400–410.

A S-COMET Score Distribution

Figure 5 (a) shows the distribution of COMET scores in Chinese (Zh) → English (En) dataset produced by LLaMA-3-SB-Instruct. Over 30% of sentences achieve a s-COMET score above 90.0, while more than 30% score below 85.0 .

Figure 5 (b) illustrates the distribution of COMET score differences between Sent2Sent and Doc2Doc translations in the same dataset. While some instances exhibit a score difference of zero, the majority follow a normal-like distribution within the range of -15 to 15, with the mean around -1.5 rather than zero.

B Data Statistics

Table 9 shows the detailed statistics of our training, validation and test datasets for the ten translation directions.

<table>
  <tr>
    <th>Dataset</th>
    <th>#Document Train/Valid/Test</th>
    <th>#Sentence Train/Valid/Test</th>
  </tr>
  <tr>
    <td>De ↔ En</td>
    <td>8.4K/150/150</td>
    <td>333K/5.9K/6.0K</td>
  </tr>
  <tr>
    <td>Fr ↔ En</td>
    <td>7.9K/150/150</td>
    <td>310K/5.9K/5.8K</td>
  </tr>
  <tr>
    <td>Es ↔ En</td>
    <td>9.7K/150/150</td>
    <td>378K/5.8K/5.8K</td>
  </tr>
  <tr>
    <td>Ru ↔ En</td>
    <td>7.3K/150/150</td>
    <td>279K/5.7K/5.6K</td>
  </tr>
  <tr>
    <td>Zh ↔ En</td>
    <td>8.6K/150/150</td>
    <td>342K/6.0K/5.9K</td>
  </tr>
</table>

Table 9: Statistics of the datasets.

![Distribution of s-COMET scores.](page_1012_120_388_388.png)

(a) s-COMET scores

![Difference of s-COMET scores between Sent2Sent and Doc2Doc](page_1012_504_388_388.png)

(b) difference of s-COMET scores between Sent2Sent and Doc2Doc

Figure 5: Distribution of s-COMET scores.

C Details in Splitting Long Documents

Similar to Li et al. (2024) and Koneru et al. (2024), we split long documents with more than 512 tokens into smaller chunks. Algorithm 1 denotes the detailed algorithm we use, where \( \mathcal{M}_T \) denotes the LLM, \( N \) denotes the number of the sentences in the document pair, \( s \) denotes the source document, \( s_i \) denotes the \( i \)-th sentence in the document, \( L \) denotes the maximum length of the chunk, \( C \) denotes the list of the chunks in the document, \( c \) denotes the chunk, \( l_c \) denotes the length of the chunk, respectively. Thus, each document is divided into multiple chunks, each containing no more than 512 tokens while ensuring sentence integrity.

During evaluating document-level metrics, we reassemble the chunks into complete documents.
Algorithm 1 Algorithm for Splitting Documents

Input: \( \mathcal{M}_T, N, s = [s_1, \cdots, s_N] \)
Output: C
\( L \leftarrow 512 \)
\( C \leftarrow [] \)
\( c \leftarrow [] \)
\( l_c \leftarrow 0 \)
for \( i \leftarrow 1 \) to \( N \) do
    \( l_i \leftarrow \) the tokenized length of \( s_i \) by \( \mathcal{M}_T \)
    if \( l_c + l_i > L \) then
        C.append(c)
        \( l_c \leftarrow 0, c \leftarrow [] \)  ▷ Starting a new chunk
        \( l_c \leftarrow l_i \)
        c.append(s_i)
    else
        \( l_c \leftarrow l_c + l_i \)
        c.append(s_i)
    end if
end for
C.append(c)

D Discussion on Doc2Doc Translation with Mismatched Source Sentence Boundaries

We observe that natural document translations often have mismatched sentence counts between the source and target. Our fine-tuned LLMs handle these cases effectively, as sentence-level alignment is not strictly required during inference. In a small number of cases, this may result in the refined translation having a different number of sentences.

During fine-tuning, only the quality-aware fine-tuning process requires sentence-level alignment between the source and target documents. However, in practical scenarios, this alignment can be relaxed by shifting to segment-level alignment. A segment may consist of one or more sentences, allowing aligned segment pairs to differ in sentence count. For instance, in a parallel document pair \((S, T)\) that is not sentence-aligned, an alignment tool like BertAlign (Liu and Zhu, 2023) can be used to generate sentence-level alignments, which can then be grouped into segment-level alignments.

E Fine-Tuning and Inferencing Settings

During fine-tuning, we adopt QLoRA (Dettmers et al., 2023), a quantized version of LoRA (Hu et al., 2021). For the hyper-parameters in Eq. 1, we set \( \lambda \) to 3.75 and \( \epsilon \) to 0.7, respectively. we set LoRA rank to 8 and LoRA alpha to 16. We apply LoRA target modules to both the query and the value components. We use the AdamW optimizer and learning rate scheduler of cosine, with an initial learning rate to 1e-4, warmup ratio of 0.1, batch size of 2, gradient accumulation over 8 steps. In both stages of quality-aware enhanced fine-tuning, we train 1 epoch. During inference, to ensure reproducibility, we set do_sample to false and temperature to 0. Following Alves et al. (2024) and Koneru et al. (2024), we set num_beams to 3. Our implementation is based on LLaMA-Factory Framework\(^{12}\) (Zheng et al., 2024).

F Effects of Hyper-Parameters

We use the combined En ↔ De validation sets to tune two hyper-parameters: \( \lambda \) and \( \epsilon \). First, we explore values of \( \epsilon \) in the range from 0.5 to 0.9 with a step size of 0.1. Our experiments reveal that \( \epsilon \) has a minimal effect on performance, and we ultimately set \( \epsilon \) to 0.7.

Next, we search for an optimal value of \( \lambda \) within the range of 1.0 to 5.0, using a step size of 0.5. We observe that \( \lambda \) values between 2.5 and 4.0 yield better performance than other values. As a result, we narrow the search for \( \lambda \) to the range of 2.5 to 4.0 with a finer step size of 0.25. Figure 6 illustrates the learning curve for \( \lambda \) values between 1.0 and 5.0, showing that \( \lambda = 3.75 \) achives the best performance.

Based on these findings, we set \( \lambda = 3.75 \) and \( \epsilon = 0.7 \) for all experiments.

![Performance of d-COMET scores curve on the En ↔ De validation sets for λ values ranging from 1.0 to 5.0. The optimal performance is achieved when λ = 3.75.](page_1012_180_388_246.png)

Figure 6: Performance of d-COMET scores curve on the En ↔ De validation sets for \( \lambda \) values ranging from 1.0 to 5.0. The optimal performance is achieved when \( \lambda = 3.75 \).

12https://github.com/hiyouga/LLaMA-Factory
<table>
  <tr>
    <th></th>
    <th>De→En</th>
    <th>En→De</th>
    <th>Zh→En</th>
    <th>En→Zh</th>
  </tr>
  <tr>
    <th>Ours</th>
    <td><b>88.14</b></td>
    <td><b>86.05</b></td>
    <td><b>85.39</b></td>
    <td><b>83.35</b></td>
  </tr>
  <tr>
    <th>Variant 1</th>
    <td>87.12</td>
    <td>85.31</td>
    <td>84.79</td>
    <td>83.17</td>
  </tr>
  <tr>
    <th>Variant 2</th>
    <td>87.60</td>
    <td>85.52</td>
    <td>84.72</td>
    <td>83.03</td>
  </tr>
</table>

Table 10: Performance comparison of d-COMET scores when using different equations to calculate weights.

G Analyse of Computational Costs

All experiments are conducted using four NVIDIA V100 GPUs. We use DeepSpeed (Rasley et al., 2020) to accelerate training and vLLM (Kwon et al., 2023) to speed up generation. On average, for each language direction, generating the Sent2Sent translations in the training set (i.e., Figure 2 (a) ) takes approximately 20 hours, while the Doc2Doc translations require around 15 hours. Computing COMET weights takes about 1 hour. Each fine-tuning stage in Figure 2 (b) takes approximately 6 hours. Generating Sent2Sent and Doc2Doc translations on the test set (i.e., Figure 2 (c) ) takes only about 30 minutes each. The final translation refinement generation requires roughly 45 minutes.

H Comparison to Other Two Weighting Variants

In addition to using Eq. 1 to compute the sentence-level weighting, we also compare it with two alternative weighting variants:

• Variant 1: Instead of using the maximum DA score, we compute the weight based on \( h_i \), which is the first translation in the prompt template (either \( y_i \) or \( z_i \)):

\[
w_i = 1 + \lambda (\mathrm{DA}(s_i, h_i, r_i) - \epsilon).
\]

(5)

• Variant 2: Rather than assigning a weight to each sentence, we assign a weight to each instance. This instance-level weight is computed as:

\[
w = 1 + \lambda (\max(\mathrm{avgDA}(s, y, r), \mathrm{avgDA}(s, z, r)) - \epsilon),
\]

(6)

where \( \mathrm{avgDA}(s, y, r) \) returns the averaged reference-based COMET score.

Table 10 compares the performance. It shows that our weighting method outperforms the other several weighting variants.

(a) SentRefine_{sent}
You are an expert in editing translations.
Given a <src_lang> source sentence and a <tgt_lang> translated version, please produce an improved translated version.
Don’t give any explanations.
<src_lang> Source: <sent_src>
<tgt_lang> Translation: <hyp>
<tgt_lang> Translation Refinement:

(b) DocRefine_{sent}/DocRefine_{doc}
You are an expert in editing translations.
Given a <src_lang> source document and a <tgt_lang> translated version, please produce an improved translated version.
Don’t give any explanations.
Each sentence is separated by #id.
<src_lang> Source: <doc_src>
<tgt_lang> Translation: <hyp>
<tgt_lang> Translation Refinement:

Figure 7: Prompt templates used in our baselines.

I Analyses of Catastrophic Forgetting

Training models on sentence-level datasets for document-level translation refinement often causes models to generate only the first sentence of the document, leading to catastrophic forgetting. However, our proposed enhanced document-level fine-tuning, incorporating sentence-level quality-aware fine-tuning, preserves the model’s sentence-level translation refinement ability. Specifically, for a given source sentence \( s_i \), the model refines two intermediate translations, \( y_i \) and \( z_i \). As shown in the last row of Table 11, our approach maintains strong performance in sentence-level refinement, confirming that catastrophic forgetting is not an issue.

J Analyses of Model-Agnostic

It is not necessary using the same LLM during training and inference. Fine-tuned LLMs can effectively refine translations from other systems, such as GPT-4o-mini and NLLB (Section 4.1). Additionally, LLaMA-3-8B-Instruct can refine Mistral-Nemo-Instruct translations and vice versa. Table 12 presents the results, where Model 1 generates sentence- and document-level translations, and Model 2 performs refinement.

K Translation Refinement Prompts

Figure 7 presents the prompt we use for baselines, including SentRefine_{sent}, DocRefine_{sent} and DocRefine_{doc}. Note that we use the same prompt when we conduct DocRefine_{sent} and DocRefine_{doc}.
<table>
  <tr>
    <th rowspan="2">System</th>
    <th colspan="5">X→En</th>
    <th colspan="5">En→X</th>
    <th rowspan="2">Avg.</th>
  </tr>
  <tr>
    <th>De→</th><th>Es→</th><th>Ru→</th><th>Fr→</th><th>Zh→</th>
    <th>→ De</th><th>→ Es</th><th>→ Ru</th><th>→ Fr</th><th>→ Zh</th>
  </tr>
  <tr>
    <td>Sent2Sent</td>
    <td>85.97</td><td>86.62</td><td>81.63</td><td>84.43</td><td>82.18</td>
    <td>82.50</td><td>85.02</td><td>80.97</td><td>82.89</td><td>76.80</td>
    <td>82.90</td>
  </tr>
  <tr>
    <td>Doc2Doc</td>
    <td>87.05</td><td>87.21</td><td>81.07</td><td>85.40</td><td>83.60</td>
    <td>83.35</td><td>85.36</td><td>80.18</td><td>83.14</td><td>81.89</td>
    <td>83.83</td>
  </tr>
  <tr>
    <td>Ours (document-level)</td>
    <td>88.14</td><td>88.42</td><td>82.75</td><td>86.69</td><td>86.69</td>
    <td>86.05</td><td>86.86</td><td>83.85</td><td>84.84</td><td>83.35</td>
    <td>85.63</td>
  </tr>
  <tr>
    <td>Ours (sentence-level)</td>
    <td>87.83</td><td>88.06</td><td>82.74</td><td>86.29</td><td>82.20</td>
    <td>86.13</td><td>84.71</td><td>83.60</td><td>84.48</td><td>82.52</td>
    <td>84.86</td>
  </tr>
</table>

Table 11: Performance of d-COMET scores when we use LLaMA-3-8B-Instruct to conduct sentence-level refinement with multiple inputs.

<table>
  <tr>
    <th>#</th>
    <th>Model 1</th>
    <th>Model 2</th>
    <th colspan="5">X→En</th>
    <th colspan="5">En→X</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td></td>
    <td>De→</td><td>Es→</td><td>Ru→</td><td>Fr→</td><td>Zh→</td>
    <td>→ De</td><td>→ Es</td><td>→ Ru</td><td>→ Fr</td><td>→ Zh</td>
    <td></td>
  </tr>
  <tr>
    <td>1</td>
    <td>LLaMA</td>
    <td>Mistral</td>
    <td>87.79</td><td>88.40</td><td>82.34</td><td>86.36</td><td>85.08</td>
    <td>86.16</td><td>86.35</td><td>83.31</td><td>84.96</td><td>82.79</td>
    <td>85.35</td>
  </tr>
  <tr>
    <td>2</td>
    <td>Mistral</td>
    <td>LLaMA</td>
    <td>88.01</td><td>88.38</td><td>84.25</td><td>86.61</td><td>85.28</td>
    <td>86.79</td><td>86.95</td><td>85.95</td><td>85.50</td><td>84.12</td>
    <td>86.18</td>
  </tr>
</table>

Table 12: Performance of d-COMET scores when we use different models in translation and refinement. LLaMA refers to LLaMA-3-8B-Instruct, and Mistral refers to Mistral-Nemo-Instruct.

![Bar chart comparing our approach with the reranking variant](page_104_1042_627_312.png)

Figure 8: Comparison of our approach with the reranking variant.

L Experimental Results in s-COMET and d-BLEU

Table 13 shows the detailed d-BLEU scores of our main experiments. Table 14 shows the detailed s-COMET scores of our main experiments. Table 15 shows the detailed s-COMET scores of our experiments in refining GPT translations.

M Comparison of Our Approach with Reranking Variant

Since our approach uses two intermediate translations, we compare it to a reranking variant that selects the better sentence-level translation from our two baselines, ensuring a fair comparison. Specifically, we calculate the percentage of sentences, based on the reference-based COMET score, where our approach either outperforms, underperforms, or ties\(^{13}\) with the reranking variant.

Figure 8 presents the comparison results for De ↔ En translation. It demonstrates that our approach outperforms the reranking variant by winning more sentences, even when the latter reranks several different two baselines.

N Prompt for Analysing Translation Errors

We present the prompt used for analysing translation errors in Table 16. "Mistranslation", "Overtranslation", "Undertranslation", "Addition" and "Omission" are sentence-level translation error types, while "Cohesion", "Coherence", "Inconsistent style" and "Multiple terms in translation" are document-level translation error types.

\(^{13}\)If the difference in their COMET scores is 0.1 or smaller, the two translations are considered a tie.
<table>
  <tr>
    <th rowspan="2">System</th>
    <th colspan="5">X→En</th>
    <th colspan="5">En→X</th>
    <th rowspan="2">Avg.</th>
  </tr>
  <tr>
    <th>De→</th><th>Es→</th><th>Ru→</th><th>Fr→</th><th>Zh→</th>
    <th>→ De</th><th>→ Es</th><th>→ Ru</th><th>→ Fr</th><th>→ Zh</th>
  </tr>
  <tr>
    <td colspan="11">LLaMA-3-8B-Instruct</td>
  </tr>
  <tr>
    <td>Sent2Sent</td><td>34.73</td><td>40.81</td><td>31.16</td><td>33.30</td><td>22.35</td><td>25.07</td><td>39.33</td><td>22.25</td><td>31.85</td><td>29.12</td><td>30.99</td>
  </tr>
  <tr>
    <td>Sent2Sent_tuned</td><td>48.26</td><td>53.44</td><td>41.58</td><td>45.09</td><td>34.02</td><td>31.93</td><td>43.92</td><td>27.24</td><td>34.29</td><td>36.07</td><td>39.58</td>
  </tr>
  <tr>
    <td>Doc2Doc</td><td>37.02</td><td>43.01</td><td>32.92</td><td>34.52</td><td>26.33</td><td>25.68</td><td>40.04</td><td>23.09</td><td>30.32</td><td>33.41</td><td>32.63</td>
  </tr>
  <tr>
    <td>Doc2Doc_tuned</td><td>47.04</td><td>53.50</td><td>42.80</td><td>43.35</td><td>35.95</td><td>30.11</td><td>44.59</td><td>27.37</td><td>34.96</td><td>38.65</td><td>39.83</td>
  </tr>
  <tr>
    <td>SentiRefine_sent</td><td>46.11</td><td>52.54</td><td>42.20</td><td>43.58</td><td>32.88</td><td>30.22</td><td>44.84</td><td>27.38</td><td>35.05</td><td>38.07</td><td>39.29</td>
  </tr>
  <tr>
    <td>DocRefine_sent</td><td>45.16</td><td>53.77</td><td>44.33</td><td>45.44</td><td>35.92</td><td>30.02</td><td>43.93</td><td>26.68</td><td>34.90</td><td>37.79</td><td>39.79</td>
  </tr>
  <tr>
    <td>DocRefine_doc</td><td>46.16</td><td>53.90</td><td>44.32</td><td>45.07</td><td>36.14</td><td>29.50</td><td>44.65</td><td>28.34</td><td>34.73</td><td>37.65</td><td>40.05</td>
  </tr>
  <tr>
    <td>Ours</td><td>48.51</td><td>54.70</td><td>45.59</td><td>45.57</td><td>37.66</td><td>32.23</td><td>45.78</td><td>28.74</td><td>35.26</td><td>38.96</td><td>41.30</td>
  </tr>
  <tr>
    <td>- QA Fine-tuning</td><td>47.86</td><td>54.07</td><td>44.81</td><td>45.02</td><td>37.07</td><td>31.47</td><td>44.87</td><td>28.43</td><td>34.42</td><td>38.77</td><td>40.68</td>
  </tr>
  <tr>
    <td colspan="11">Mistral-Nemo-Instruct</td>
  </tr>
  <tr>
    <td>Sent2Sent</td><td>38.18</td><td>43.20</td><td>34.45</td><td>35.87</td><td>27.51</td><td>29.02</td><td>41.88</td><td>25.44</td><td>33.17</td><td>34.37</td><td>34.31</td>
  </tr>
  <tr>
    <td>Sent2Sent_tuned</td><td>40.62</td><td>45.67</td><td>39.29</td><td>38.93</td><td>31.90</td><td>30.00</td><td>42.77</td><td>27.15</td><td>33.73</td><td>35.07</td><td>36.51</td>
  </tr>
  <tr>
    <td>Doc2Doc</td><td>40.92</td><td>45.20</td><td>37.51</td><td>37.98</td><td>29.74</td><td>29.70</td><td>42.10</td><td>27.88</td><td>34.10</td><td>37.09</td><td>36.22</td>
  </tr>
  <tr>
    <td>Doc2Doc_tuned</td><td>49.17</td><td>55.10</td><td>43.35</td><td>46.01</td><td>38.25</td><td>31.65</td><td>45.75</td><td>22.15</td><td>37.10</td><td>42.24</td><td>41.08</td>
  </tr>
  <tr>
    <td>SentiRefine_sent</td><td>46.11</td><td>52.54</td><td>47.90</td><td>43.25</td><td>32.65</td><td>30.22</td><td>44.84</td><td>30.40</td><td>36.05</td><td>35.10</td><td>40.11</td>
  </tr>
  <tr>
    <td>DocRefine_sent</td><td>48.75</td><td>55.56</td><td>46.45</td><td>46.49</td><td>36.76</td><td>34.13</td><td>46.12</td><td>31.13</td><td>37.45</td><td>41.44</td><td>42.43</td>
  </tr>
  <tr>
    <td>DocRefine_doc</td><td>49.77</td><td>55.70</td><td>46.29</td><td>46.52</td><td>37.09</td><td>33.82</td><td>46.33</td><td>31.02</td><td>37.29</td><td>42.68</td><td>42.65</td>
  </tr>
  <tr>
    <td>Ours</td><td>51.17</td><td>56.20</td><td>48.58</td><td>47.97</td><td>41.00</td><td>35.44</td><td>47.01</td><td>32.79</td><td>38.43</td><td>43.13</td><td>44.17</td>
  </tr>
  <tr>
    <td>- QA Fine-tuning</td><td>50.43</td><td>55.37</td><td>47.97</td><td>45.92</td><td>37.89</td><td>35.28</td><td>46.64</td><td>31.62</td><td>37.87</td><td>42.41</td><td>43.14</td>
  </tr>
</table>

Table 13: Performance in document-level (d-BLEU) score.

<table>
  <tr>
    <th rowspan="2">System</th>
    <th colspan="5">X→En</th>
    <th colspan="5">En→X</th>
    <th rowspan="2">Avg.</th>
  </tr>
  <tr>
    <th>De→</th><th>Es→</th><th>Ru→</th><th>Fr→</th><th>Zh→</th>
    <th>→ De</th><th>→ Es</th><th>→ Ru</th><th>→ Fr</th><th>→ Zh</th>
  </tr>
  <tr>
    <td colspan="11">LLaMA-3-8B-Instruct</td>
  </tr>
  <tr>
    <td>Sent2Sent</td><td>87.71</td><td>88.32</td><td>83.74</td><td>86.63</td><td>84.60</td><td>84.47</td><td>86.82</td><td>83.23</td><td>84.55</td><td>79.76</td><td>84.98</td>
  </tr>
  <tr>
    <td>Sent2Sent_tuned</td><td>88.93</td><td>88.91</td><td>86.38</td><td>88.33</td><td>86.27</td><td>86.28</td><td>87.12</td><td>86.25</td><td>86.43</td><td>86.49</td><td>87.14</td>
  </tr>
  <tr>
    <td>Doc2Doc</td><td>88.62</td><td>88.76</td><td>84.47</td><td>87.36</td><td>85.84</td><td>83.87</td><td>87.07</td><td>82.61</td><td>84.79</td><td>83.85</td><td>85.72</td>
  </tr>
  <tr>
    <td>Doc2Doc_tuned</td><td>89.35</td><td>89.91</td><td>80.51</td><td>88.29</td><td>86.38</td><td>87.20</td><td>88.20</td><td>83.76</td><td>86.26</td><td>85.51</td><td>86.54</td>
  </tr>
  <tr>
    <td>SentiRefine_sent</td><td>89.12</td><td>89.65</td><td>85.29</td><td>88.08</td><td>86.53</td><td>87.10</td><td>88.17</td><td>87.16</td><td>86.38</td><td>86.70</td><td>87.42</td>
  </tr>
  <tr>
    <td>DocRefine_sent</td><td>88.96</td><td>89.08</td><td>83.09</td><td>88.45</td><td>87.19</td><td>87.18</td><td>88.17</td><td>83.21</td><td>86.08</td><td>86.42</td><td>86.78</td>
  </tr>
  <tr>
    <td>DocRefine_doc</td><td>89.22</td><td>89.51</td><td>84.45</td><td>88.24</td><td>87.25</td><td>86.86</td><td>88.34</td><td>86.12</td><td>86.39</td><td>86.70</td><td>87.31</td>
  </tr>
  <tr>
    <td>Ours</td><td>89.63</td><td>89.95</td><td>84.58</td><td>88.58</td><td>87.26</td><td>87.76</td><td>88.61</td><td>86.34</td><td>86.50</td><td>86.88</td><td>87.61</td>
  </tr>
  <tr>
    <td>- QA Fine-tuning</td><td>89.41</td><td>89.88</td><td>84.44</td><td>88.43</td><td>87.19</td><td>87.43</td><td>88.37</td><td>85.63</td><td>86.14</td><td>86.69</td><td>87.36</td>
  </tr>
  <tr>
    <td colspan="11">Mistral-Nemo-Instruct</td>
  </tr>
  <tr>
    <td>Sent2Sent</td><td>88.52</td><td>88.40</td><td>84.24</td><td>87.00</td><td>86.18</td><td>86.64</td><td>87.32</td><td>86.25</td><td>85.52</td><td>85.41</td><td>86.54</td>
  </tr>
  <tr>
    <td>Sent2Sent_tuned</td><td>88.49</td><td>88.55</td><td>85.03</td><td>87.78</td><td>86.42</td><td>87.24</td><td>87.22</td><td>87.17</td><td>86.52</td><td>85.85</td><td>87.03</td>
  </tr>
  <tr>
    <td>Doc2Doc</td><td>89.15</td><td>89.29</td><td>85.16</td><td>87.90</td><td>86.81</td><td>86.56</td><td>87.30</td><td>86.66</td><td>85.65</td><td>85.74</td><td>87.02</td>
  </tr>
  <tr>
    <td>Doc2Doc_tuned</td><td>89.70</td><td>90.20</td><td>85.01</td><td>88.61</td><td>87.70</td><td>85.91</td><td>88.66</td><td>85.19</td><td>86.99</td><td>87.56</td><td>87.53</td>
  </tr>
  <tr>
    <td>SentiRefine_sent</td><td>89.33</td><td>89.80</td><td>85.51</td><td>88.24</td><td>86.71</td><td>88.04</td><td>88.30</td><td>87.89</td><td>86.77</td><td>86.71</td><td>87.73</td>
  </tr>
  <tr>
    <td>DocRefine_sent</td><td>89.63</td><td>90.03</td><td>84.21</td><td>88.02</td><td>87.64</td><td>88.24</td><td>88.68</td><td>86.93</td><td>86.90</td><td>87.55</td><td>87.78</td>
  </tr>
  <tr>
    <td>DocRefine_doc</td><td>89.74</td><td>90.06</td><td>83.50</td><td>88.21</td><td>87.49</td><td>88.21</td><td>88.69</td><td>86.33</td><td>86.85</td><td>87.44</td><td>87.65</td>
  </tr>
  <tr>
    <td>Ours</td><td>89.94</td><td>90.45</td><td>86.10</td><td>87.53</td><td>87.96</td><td>88.53</td><td>89.02</td><td>88.31</td><td>87.16</td><td>88.04</td><td>88.40</td>
  </tr>
  <tr>
    <td>- QA Fine-tuning</td><td>89.90</td><td>90.12</td><td>85.82</td><td>88.65</td><td>87.87</td><td>88.49</td><td>88.87</td><td>87.81</td><td>87.07</td><td>87.71</td><td>88.23</td>
  </tr>
</table>

Table 14: Performance in sentence-level COMET (s-COMET) score.
<table>
  <tr>
    <th>#</th>
    <th>System</th>
    <th colspan="5">X→En</th>
    <th colspan="5">En→X</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th>De→</th>
    <th>Es→</th>
    <th>Ru→</th>
    <th>Fr→</th>
    <th>Zh→</th>
    <th>→ De</th>
    <th>→ Es</th>
    <th>→ Ru</th>
    <th>→ Fr</th>
    <th>→ Zh</th>
    <th></th>
  </tr>
  <tr>
    <td colspan="13">GPT Translation & Refining GPT Translation</td>
  </tr>
  <tr>
    <td>1</td>
    <td>GPT Sent2Sent</td>
    <td>88.39</td>
    <td>88.51</td>
    <td>83.76</td>
    <td>87.05</td>
    <td>86.34</td>
    <td>87.43</td>
    <td>87.63</td>
    <td>87.55</td>
    <td>86.35</td>
    <td>87.09</td>
    <td>87.01</td>
  </tr>
  <tr>
    <td>2</td>
    <td>GPT Doc2Doc</td>
    <td>88.12</td>
    <td>89.10</td>
    <td>85.24</td>
    <td>87.02</td>
    <td>86.96</td>
    <td>87.98</td>
    <td>88.41</td>
    <td>87.88</td>
    <td>86.83</td>
    <td>87.70</td>
    <td>87.52</td>
  </tr>
  <tr>
    <td>3</td>
    <td>GPT SentRefine<sub>sent</sub></td>
    <td>88.51</td>
    <td>88.56</td>
    <td>84.42</td>
    <td>87.63</td>
    <td>86.60</td>
    <td>87.72</td>
    <td>88.28</td>
    <td>87.16</td>
    <td>86.72</td>
    <td>87.44</td>
    <td>87.30</td>
  </tr>
  <tr>
    <td>4</td>
    <td>GPT DocRefine<sub>sent</sub></td>
    <td>88.65</td>
    <td>88.69</td>
    <td>84.82</td>
    <td>87.61</td>
    <td>86.55</td>
    <td>88.41</td>
    <td>88.78</td>
    <td>88.45</td>
    <td>87.10</td>
    <td>87.24</td>
    <td>87.63</td>
  </tr>
  <tr>
    <td>5</td>
    <td>GPT DocRefine<sub>doc</sub></td>
    <td>88.64</td>
    <td>88.90</td>
    <td>84.83</td>
    <td>87.70</td>
    <td>86.65</td>
    <td>88.38</td>
    <td>88.71</td>
    <td>88.47</td>
    <td>87.16</td>
    <td>87.42</td>
    <td>87.69</td>
  </tr>
  <tr>
    <td>6</td>
    <td>GPT DocRefine<sub>doc+sent</sub></td>
    <td><b>88.99</b></td>
    <td><b>89.25</b></td>
    <td>85.09</td>
    <td>87.79</td>
    <td><b>86.98</b></td>
    <td>88.28</td>
    <td>88.59</td>
    <td>88.41</td>
    <td>87.03</td>
    <td><b>87.79</b></td>
    <td><b>87.82</b></td>
  </tr>
  <tr>
    <td>7</td>
    <td>L-DocRefine<sub>doc+sent</sub></td>
    <td>88.78</td>
    <td>88.98</td>
    <td>84.28</td>
    <td>87.81</td>
    <td>86.73</td>
    <td>88.16</td>
    <td><b>89.11</b></td>
    <td>86.45</td>
    <td>86.95</td>
    <td>87.32</td>
    <td>87.46</td>
  </tr>
  <tr>
    <td>8</td>
    <td>M-DocRefine<sub>doc+sent</sub></td>
    <td><b>90.02</b></td>
    <td>89.05</td>
    <td><b>86.29</b></td>
    <td><b>87.92</b></td>
    <td><b>86.99</b></td>
    <td><b>88.67</b></td>
    <td>89.08</td>
    <td><b>88.57</b></td>
    <td><b>87.32</b></td>
    <td><b>87.95</b></td>
    <td><b>88.19</b></td>
  </tr>
  <tr>
    <td colspan="13">NLLB Translation & Refining NLLB Translation</td>
  </tr>
  <tr>
    <td>9</td>
    <td>NLLB Sent2Sent</td>
    <td>88.45</td>
    <td>89.16</td>
    <td>84.89</td>
    <td>87.66</td>
    <td>85.65</td>
    <td>86.74</td>
    <td>88.30</td>
    <td>87.76</td>
    <td>86.57</td>
    <td>77.82</td>
    <td>86.29</td>
  </tr>
  <tr>
    <td>10</td>
    <td>L-DocRefine<sub>doc+sent</sub></td>
    <td>88.76</td>
    <td>89.27</td>
    <td>83.92</td>
    <td>87.74</td>
    <td><b>87.35</b></td>
    <td>88.14</td>
    <td><b>88.44</b></td>
    <td>85.79</td>
    <td>86.99</td>
    <td><b>86.80</b></td>
    <td><b>87.32</b></td>
  </tr>
  <tr>
    <td>11</td>
    <td>M-DocRefine<sub>doc+sent</sub></td>
    <td><b>88.98</b></td>
    <td><b>89.50</b></td>
    <td><b>85.33</b></td>
    <td><b>87.92</b></td>
    <td>86.92</td>
    <td><b>88.60</b></td>
    <td>88.08</td>
    <td><b>88.31</b></td>
    <td><b>87.32</b></td>
    <td><b>86.96</b></td>
    <td><b>87.79</b></td>
  </tr>
</table>

Table 15: Performance in s-COMET when refining translations from GPT-4o-mini. For the GPT-based refinement systems, we use the same prompt templates as those used in our approach, but without fine-tuning. L-* and M-* denote our fine-tuned LLaMA-3-8B-Instrcut and Mistral-Nemo-Instruct, respectively.

[Source]:
<src_doc>
[Reference]:
<ref_doc>
[Hypothesis]:
<hyp_doc>

[Error Types]:
- Mistranslation: Error occurring when the target content does not accurately represent the source.
- Overtranslation: Error occurring in the target content that is inappropriately more specific than the source.
- Undertranslation: Error occurring in the target content that is inappropriately less specific than the source.
- Addition: Error occurring in the target content that includes content not present in the source.
- Omission: Error where content present in the source is missing in the target.
- Cohesion: Portions of the text needed to connect it into an understandable whole (e.g., reference, substitution, ellipsis, conjunction, and lexical cohesion) missing or incorrect.
- Coherence: Text lacking a clear semantic relationship between its parts, i.e., the different parts don’t hang together, don’t follow the discourse conventions of the target language, or don’t “make sense.”
- Inconsistent style: Style that varies inconsistently throughout the text, e.g., One part of a text is written in a clear, "terse" style, while other sections are written in a more wordy style.
- Multiple terms in translation: Error where source content terminology is correct, but target content terms are not used consistently.

Considering the provided context, please identify the errors of the translation from the source to the target in the current sentence based on a subset of Multidimensional Quality Metrics (MQM) error typology. You should pay extra attention to the error types related to the relationship between the current sentence and its context, such as "Unclear reference", "Cohesion", "Coherence", "Inconsistent style", and "Multiple terms in translation".
For each sentence in machine translation, please give the error types and brief explanation for errors.The returned format is as follows:
Sentence #id :
Error types: ...
Explanation for errors: ...

Table 16: Prompt used for analyzing translation errors.

Reviewer paper summaries: This paper addresses the challenge of document-level machine translation by combining two separate intermediate translations—one produced at the sentence level (Sent2Sent) and one at the document level (Doc2Doc)—and then refining them via large language models (LLMs). By constructing a (source, sent2sent, doc2doc, reference) quadruple and adding a difficulty-based weighting mechanism, the authors aim to unite the best aspects of sentence-level and document-level translations while mitigating their respective shortcomings. Experiments on two popular open-source LLMs (LLaMA-3-8B-Instruct and Mistral-Nemo-Instruct) across five language pairs (in both directions) demonstrate gains in multiple metrics, especially COMET and discourse-related scores.
Summary of strengths: 1. The proposed approach achieves notable performance gains without requiring a large increase in computational resources.

2. The paper leverages the contextual richness of document-level translations while also drawing on the literal precision of sentence-level translations. 

3. By introducing a quality-aware weighting scheme, the model learns to focus more heavily on “hard” sentences and adjust the refinement accordingly. This is particularly beneficial for long or intricate sentences, ultimately improving translation quality on challenging inputs.
Summary of weaknesses: 1. The paper should discuss whether simultaneously fine-tuning on sentence-level and document-level data risks catastrophic forgetting. In other words, could the strengths of one type of translation be partially overwritten by the other?

2. The method for processing very long documents remains somewhat under-explained. If the entire text is translated and refined in one pass, hardware requirements may be high. Additional details on practical handling—such as splitting documents, using chunked inference, or other memory strategies—would be valuable for readers who want to scale this approach.
Comments, suggestions and typos: See weaknesses
Overall assessment scores: 3
Metareview score: 4
Metareview: The author propose to refine automatic translations using an LLMs. Thereby, they use a sentence level MT system and a document level MT system as input and then combine them into a hopefully better translation.
Summary of reasons to publish: Good performance of the proposed approach and well-set scenario
Authors propose an interesting method