LLM-based Translation Inference with Iterative Bilingual Understanding
Authors: Andong Chen, Kehai Chen, Yang Xiang, Xuefeng Bai, Muyun Yang, Yang Feng, Tiejun Zhao, Min zhang
Accepted at: acl25

LLM-based Translation Inference with Iterative Bilingual Understanding

Andong Chen 1, Kehai Chen*2, Yang Xiang3, Xuefeng Bai2,
Muyun Yang1, Yang Feng4, Tiejun Zhao1, Min Zhang2
1 Harbin Institute of Technology, Harbin, China
2 Harbin Institute of Technology, Shenzhen, China
3 Pengcheng Laboratory, Shenzhen, China
4 Key Laboratory of Intelligent Information Processing,
Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS)
ands691119@gmail.com, xiangy@pcl.ac.cn, fengyang@ict.ac.cn
{baixuefeng, chenkeha, yangmuyun, tzhao, zhangmin2021}@hit.edu.cn

Abstract

The remarkable understanding and generation capabilities of large language models (LLMs) have greatly improved translation performance. However, incorrect understanding of the sentence to be translated can degrade translation quality. To address this issue, we proposed a novel Iterative Bilingual Understanding Translation (IBUT) method based on the cross-lingual capabilities of LLMs and the dual characteristics of translation tasks. The cross-lingual capability of LLMs enables the generation of contextual understanding for both the source and target languages separately. Furthermore, the dual characteristics allow IBUT to generate effective cross-lingual feedback, iteratively refining contextual understanding, thereby reducing errors and improving translation performance. Experimental results showed that the proposed IBUT outperforms several strong comparison methods, especially being generalized to multiple domains (e.g., news, commonsense, and cultural translation benchmarks). 1

1 Introduction

In the field of machine translation (MT), translations based on LLMs (LLM-MT) have become a research focus (Tyen et al., 2023; Liang et al., 2023; Guerreiro et al., 2023; Ranaldi et al., 2023; Zhang et al., 2024). Numerous studies have shown that the remarkable understanding and generation capabilities of LLMs significantly improve translation performance (Hendy et al., 2023; Jiao et al., 2023; Le Scao et al., 2023; Iyer et al., 2023; Zeng et al., 2023; Zhao et al., 2024). One type of translation paradigm can be summarized as follows: the LLM first generates a contextual understanding of the sentence to be translated, and then, based on this understanding,

![Illustration of the LLMs translation paradigm based on contextual understanding (Fig a). A commonsense domain example of LLM (gpt-3.5-turbo) translation from Chinese to English (Fig b).](page_728_370_484_312.png)

Figure 1: Illustration of the LLMs translation paradigm based on contextual understanding (Fig a). A commonsense domain example of LLM (gpt-3.5-turbo) translation from Chinese to English (Fig b).

performs a target translation (as shown in Figure 1(a)) (He et al., 2024; Chen et al., 2024a; Liang et al., 2023; Wu et al., 2024).

However, our study found that when the LLM generates incorrect contextual understanding of the sentence to be translated, it negatively affects translation quality (as shown in Figure 1(b)). These generated understanding errors lead to the introduction of misleading information during the translation process, particularly when dealing with complex concepts such as commonsense and cultural domains. We refer to this issue as Understanding Distortion of LLMs. This study manually evaluated the Chinese-English test set in the commonsense domain (randomly sampled 200 sentences). It found that Understanding Distortion makes up 40% of translation errors, highlighting the importance of this issue (§ 5.5).

To address this issue, we propose a new method called Iterative Bilingual Understanding Translation (IBUT). The IBUT method consists of four parts: Understanding Generation, Alignment Judgment, Iterative Refinement, and Understanding-Based Translation. For Understanding Generation, IBUT leverages the cross-linguistic capabilities of LLMs to generate
contextual understanding for both the source and target languages. For the alignment judgment part, IBUT uses the generated bilingual contextual understanding and employs dual learning from the translation task (He et al., 2016; Qin, 2020; Chen et al., 2024a) as supervisory signals to produce explicit verbal feedback. For the Iterative Refinement part, the verbal feedback act as a semantic gradient (Wang et al., 2024; Shinn et al., 2024), providing LLM with a clear direction for refinement, thereby iteratively refining the bilingual contextual understanding. For the Understanding-Based Translation part, LLM-MT finally translates using the source sentence and refined bilingual contextual understanding.

Empirical evaluations were conducted on closed-source and open-source LLMs including ChatGPT, GPT-4, Alpaca, and Vicuna, covering multiple domains (e.g., news, commonsense, and cultural translation benchmarks). The results show an average improvement of +1.3, +4.2, and +2.3 COMET scores compared to the baseline, confirming the effectiveness of the IBUT strategy. Additionally, analysis experiments show that IBUT positively enhances the quality of bilingual contextual understanding with each iteration, thereby improving translation performance.

2 Related Work

Machine Translation Based on Large Language Models (LLM-MT). Large language models, such as GPT-3 (Brown et al., 2020), have demonstrated their effectiveness in machine translation across various language pairs (Jiao et al., 2023; Chen et al., 2024b; Iyer et al., 2023; Zeng et al., 2023; Karpinska and Iyyer, 2023; Moslem et al., 2023c; Huang et al., 2024). Recent studies delve into the performance of LLM in machine translation, including control over formality in translation outputs (Garcia and Firat, 2022), in-context translation abilities during pre-training (Shin et al., 2022), and the impact of LLM-based machine translation on culturally sensitive texts (Yao et al., 2023). Additionally, a study has explored the bilingual capabilities of LLMs to enhance translation performance (Huang et al., 2024). For translation tasks requiring reasoning, multi-agent debates can effectively enhance the reasoning abilities of LLM-MT (Liang et al., 2023). These investigations further validate the research value of LLM-MT, offering diverse research directions for scholars.

Knowledge-based Machine Translation. Extensive research indicates that incorporating knowledge enhances translation performance. This external knowledge includes bilingual dictionaries (Arthur et al., 2016), probabilistic interpolation of dictionaries (Khandelwal et al., 2020), data augmentation through back-translation (Hu et al., 2019), and entity-based denoising pre-training (Hu et al., 2021). Additionally, researchers introduced domain (Gao et al., 2023) and part-of-speech information during the inference phase and obtained multilingual translations of key terms through the NLLB translator (Lu et al., 2023), thereby enhancing the translation quality for low-resource languages. LLMs enhance MT through internal knowledge and sentence-level understanding (He et al., 2023; Huang et al., 2024), while we focus on their bilingual understanding inconsistency, unlike Huang et al., 2024, which examines misalignment in difficult word translation.

3 Iterative Bilingual Understanding Translation

The poor understanding of translated sentences generated by LLMs leads to a decline in translation quality. To address this issue, we propose a new method called Iterative Bilingual Understanding Translation (IBUT). IBUT utilizes LLMs to generate bilingual contextual understanding of the source input and utilizes the dual learning of translation tasks to establish verbal feedback for iteratively refining this understanding. Finally, the iterative refinement reduces errors in bilingual contextual understanding, thereby enhancing translation performance. The IBUT consists of four parts: 1) Understanding Generation; 2) Alignment Judgment; 3) Iterative Refinement; 4) Understanding-Based Translation. We use \( MT \) to denote a translation model based on LLM, and lowercase letters \( s \) and \( t \) to represent sentences in the source language (\( L^s \)) and target language (\( L^t \)), respectively. That is, \( s = (s[1], \cdots, s[n]) \) and \( t = (t[1], \cdots, t[m]) \), where each \( s[i] \) and \( t[i] \) is a token.

Understanding Generation. For the first part of the IBUT method, as shown in Figure 2, LLMs generate contextual understanding in both the source and target languages from the source sentence, represented as \( C_s \) and \( C_t \) respectively.
Figure 2: IBUT translation framework. The process involves first generating a bilingual understanding of the translation input sentence using an LLM. Next, verbal feedback is obtained via LLM, informed by the translation input and the bilingual understanding. This feedback is then used to further refine the bilingual understanding. The final step involves using LLM to perform the translation, leveraging both the bilingual understanding and the original input sentence. Gray text indicates English annotations for the Chinese.

This understanding includes key concepts, terms, term explanations, and examples. Detailed prompts are provided in Appendix A.1.

Alignment Judgment. The second part of IBUT introduces an LLM-based agent, denoted as \( JA \), which evaluates the consistency of bilingual contextual understanding and supervises the entire translation process. Based on the dual learning (He et al., 2016; Qin, 2020; Chen et al., 2024a), bilingual contextual understanding is generated from the same source sentence, and both should be consistent in form and semantics. Based on this assumption, \( JA \) initially identifies whether there are differences in the bilingual contextual understanding (\( C_s \) and \( C_t \)) generated based on the source sentence \( s \). If \( JA(C_s, C_t, s) = \) True, as shown in Figure 2, \( JA \) generates explicit verbal feedback in both the source and target languages (\( F_s, F_t \leftarrow \text{JA}(C_s, C_t, s) \)). The verbal feedback specifies the content of the differences between \( C_s \) and \( C_t \) and provides suggestions for refinement. If \( JA(C_s, C_t, s) = \) False, the process moves to Understanding-Based Translation (Appendix A.2 for prompts).

Iterative Refinement. In the third part of IBUT, the max number of iterations (\( \text{max\_iter} \)) is initially defined. As shown in Figure 2, the previously generated bilingual contextual understanding is refined based on the verbal feedback signals \( F_s \) and \( F_t \) (\( C'_s \leftarrow M(s, C_s, F_s) \) and \( C'_t \leftarrow M(s, C_t, F_t) \), where \( M \) is an LLM). If the number of iterations exceeds \( \text{max\_iter} \), the process will directly enter the Understanding-Based translation part. If the number of iterations does not exceed \( \text{max\_iter} \), the process will continue into the Alignment Judgment part
to iteratively refine the bilingual contextual understanding. Specific prompts are displayed in Appendix A.3.

Understanding-Based Translation. In the final part of IBUT, the refined bilingual contextual understanding (\( C_s' \) and \( C_t' \)) and the sentence to be translated are taken as inputs, and the translation is directly carried out through LLM-MT (\( t = MT(s, C_s', C_t') \)). See Appendix A.4 for prompts.

4 Experimental Setup

Dataset: We conduct experiments on four MT benchmarks: WMT22, WMT23 (general news MT benchmarks), commonsense MT, and cultural MT. Dataset details are in Appendix A.5.

Comparative Methods. In our evaluation, IBUT is compared with a range of translation methods, including Zero-shot (Wei et al., 2022), 5-shot (Brown et al., 2020), Rerank (Moslem et al., 2023a), Refine (Chen et al., 2023), MAD (Liang et al., 2023), TEaR (Feng et al., 2024), Dual-Reflect (Chen et al., 2024a), DUAT (Huang et al., 2024), and MAPS (He et al., 2023). To validate its generalizability, we utilize multiple LLMs, which include closed-source models such as ChatGPT (Ouyang et al., 2022) and GPT-4 (Achiam et al., 2023) 2, as well as open-source models like Alpaca-7B (Taori et al., 2023) 3, Vicuna-7B (Chiang et al., 2023) 4, and Qwen2.5-7B (Team, 2024) 5. Details on comparative methods are in Appendix A.6.

Evaluation Metrics. In evaluating our translation methodology, we initially employ COMET 6 (Rei et al., 2022a) and BLEURT 7 (Sellam et al., 2020) as automatic metrics, aligning with the established standards in LLM-based translation literature (He et al., 2023; Huang et al., 2024). For traditional translation evaluation, we use BLEU 8 (Papineni et al., 2002). To further evaluate our translation method, we employ human evaluations to verify translation performance. Details on human evaluations are in Appendix B.7.

2The ChatGPT and GPT-4 models used in this work are accessed through the gpt-3.5-turbo and gpt-4 APIs, respectively.
3https://huggingface.co/tatsu-lab/alpaca-7b-wd1f/tree/main
4https://huggingface.co/limsys/vicuna-7b-v1.5
5https://modelscope.cn/models/Qwen/Qwen2.5-7B-Instruct/summary
6https://huggingface.co/Unbabel/wmt22-comet-da
7https://github.com/lucadiliello/bleurt-pytorch
8https://github.com/mjpost/sacrebleu

5 Experimental Results

5.1 Main Results

The effectiveness of IBUT in general news translation tasks. In the WMT22 general news tasks, as shown in Table 1 (WMT23 results in the Appendix B.4), IBUT outperforms other methods across 13 language pairs and 3 evaluation metrics. Specifically, in the news domain, the IBUT method outperforms translations directly based on contextual understanding by +1.5 COMET and +1.4 BLEURT. This indicates that the IBUT method alleviates the issue of Understanding Distortion in the news domain.

The effectiveness of IBUT in low-resource tasks. We selected all low-resource tasks (Uk↔Cs, Ru↔Sah, Liv↔En, En→Hr) from WMT22. As observed in Table 1, current low-resource tasks still pose challenges to LLMs. However, compared to baseline methods, IBUT achieved an average improvement of +2.6 COMET in these low-resource tasks, with increases of +4 and +6.5 COMET for Liv↔En, respectively.

IBUT is effective across different language similarities. In WMT22, we validated the IBUT model using tasks with different language similarities. Specifically, Uk↔Cs represents closely related languages; En→De and En→Hr are from the same language family; Liv↔En, Ru↔Sah, and En→Ja are categorized as distant language families. The experimental results, as shown in Table 1, demonstrate significant improvements across different language similarities due to IBUT. Notably, for the selected distant family languages, there was an average increase of +3.4 COMT, highlighting IBUT’s potential to enhance translation tasks in distant language families.

5.2 Cross-domain generalizability of IBUT

IBUT Adapts to Cultural MT. As shown in Table 2, IBUT outperforms other methods across all 6 language pairs. For translation corpora containing cultural-specific items, the IBUT method achieved an average increase of +2.02 and +1.6 COMET compared to the ChatGPT and MAPS methods. Notably, in the En→Ta translation task, IBUT outperformed ChatGPT by +5.5 COMET. The experimental results above indicate that IBUT is suitable for translation tasks in the cultural domain.

IBUT performed well in commonsense translation tasks. As shown in Table 3, IBUT significantly outperformed other methods
<table>
  <tr>
    <th>WMT22</th>
    <th>En→De</th>
    <th>En→Ja</th>
    <th>Cs→Uk</th>
    <th>Uk→Cs</th>
    <th>En→Hr</th>
    <th>Sah→Ru</th>
    <th>Ru→Sah</th>
    <th>En→Liv</th>
    <th>Liv→En</th>
  </tr>
  <tr>
    <th colspan="10">COMET †/BLEURT †</th>
  </tr>
  <tr>
    <td>ChatGPT</td>
    <td>85.8/75.6</td>
    <td>88.3/66.3</td>
    <td>89.7/79.0</td>
    <td>88.7/79.0</td>
    <td>86.6/76.8</td>
    <td>57.5/36.0</td>
    <td>52.8/37.2</td>
    <td>52.7/41.8</td>
    <td>40.6/39.0</td>
  </tr>
  <tr>
    <td>+5shot</td>
    <td>86.5/76.3</td>
    <td>88.2/71.1</td>
    <td>88.3/75.6</td>
    <td>89.6/79.1</td>
    <td>86.4/75.6</td>
    <td>58.3/36.0</td>
    <td>53.1/75.4</td>
    <td>55.3/42.1</td>
    <td>42.7/40.9</td>
  </tr>
  <tr>
    <td>+Rerank</td>
    <td>86.2/75.3</td>
    <td>88.0/66.6</td>
    <td>88.3/75.3</td>
    <td>89.7/79.5</td>
    <td>86.3/75.4</td>
    <td>58.6/36.3</td>
    <td>53.8/75.9</td>
    <td>55.5/42.7</td>
    <td>42.9/41.0</td>
  </tr>
  <tr>
    <td>+MAD</td>
    <td>86.5/76.4</td>
    <td>88.4/67.9</td>
    <td>90.2/79.3</td>
    <td>89.6/79.3</td>
    <td>87.0/76.9</td>
    <td>58.1/37.1</td>
    <td>53.5/76.4</td>
    <td>55.5/42.5</td>
    <td>43.2/41.3</td>
  </tr>
  <tr>
    <td>+MAPS</td>
    <td>86.4/76.3</td>
    <td>88.5/67.4</td>
    <td>88.8/76.1</td>
    <td>89.8/79.6</td>
    <td>86.5/76.0</td>
    <td>58.7/37.3</td>
    <td>53.3/76.1</td>
    <td>54.1/42.0</td>
    <td>43.6/39.7</td>
  </tr>
  <tr>
    <td>+Refine</td>
    <td>86.0/75.9</td>
    <td>88.6/67.9</td>
    <td>89.8/79.0</td>
    <td>89.3/79.8</td>
    <td>87.0/76.9</td>
    <td>58.3/37.4</td>
    <td>53.8/76.5</td>
    <td>55.5/42.7</td>
    <td>43.9/40.1</td>
  </tr>
  <tr>
    <td>+TEaR</td>
    <td>86.2/76.2</td>
    <td>88.0/67.3</td>
    <td>88.7/77.3</td>
    <td>89.3/79.2</td>
    <td>87.2/76.2</td>
    <td>58.3/37.2</td>
    <td>53.4/75.3</td>
    <td>54.7/42.9</td>
    <td>43.5/39.8</td>
  </tr>
  <tr>
    <td>+DUAT</td>
    <td>86.7/74.1</td>
    <td>88.5/66.9</td>
    <td>88.1/76.5</td>
    <td>88.5/78.9</td>
    <td>87.2/76.7</td>
    <td>58.8/37.4</td>
    <td>53.8/75.6</td>
    <td>55.2/42.7</td>
    <td>42.8/40.5</td>
  </tr>
  <tr>
    <td>+Dual-Reflect</td>
    <td>85.8/75.1</td>
    <td>88.3/67.2</td>
    <td>88.9/76.3</td>
    <td>87.1/79.0</td>
    <td>87.2/76.9</td>
    <td>58.0/37.1</td>
    <td>58.2/47.2</td>
    <td>53.7/43.0</td>
    <td>43.1/38.1</td>
  </tr>
  <tr>
    <td>+IBUT</td>
    <td><b>87.0/77.0</b></td>
    <td><b>89.5/69.9</b></td>
    <td><b>91.2/80.1</b></td>
    <td><b>90.0/80.1</b></td>
    <td><b>87.8/77.1</b></td>
    <td><b>59.5/37.9</b></td>
    <td><b>54.5/76.9</b></td>
    <td><b>56.7/44.2</b></td>
    <td><b>47.1/40.5</b></td>
  </tr>
</table>

<table>
  <tr>
    <th></th>
    <th colspan="10">BLEU †</th>
  </tr>
  <tr>
    <td>ChatGPT</td>
    <td>32.3</td>
    <td>17.3</td>
    <td>29.9</td>
    <td>30.6</td>
    <td>26.9</td>
    <td>5.9</td>
    <td>1.9</td>
    <td>2.4</td>
    <td>8.5</td>
  </tr>
  <tr>
    <td>+5shot</td>
    <td>32.9</td>
    <td>17.9</td>
    <td>29.3</td>
    <td>31.2</td>
    <td>25.8</td>
    <td>6.4</td>
    <td>2.3</td>
    <td>2.7</td>
    <td>8.8</td>
  </tr>
  <tr>
    <td>+Rerank</td>
    <td>33.6</td>
    <td>21.2</td>
    <td>29.5</td>
    <td>31.9</td>
    <td>26.9</td>
    <td>6.5</td>
    <td>2.6</td>
    <td>2.9</td>
    <td>8.9</td>
  </tr>
  <tr>
    <td>+MAD</td>
    <td>32.9</td>
    <td>19.7</td>
    <td>31.6</td>
    <td>31.6</td>
    <td>26.5</td>
    <td>6.7</td>
    <td>2.6</td>
    <td>3.1</td>
    <td>9.7</td>
  </tr>
  <tr>
    <td>+MAPS</td>
    <td>33.1</td>
    <td>21.2</td>
    <td>29.5</td>
    <td>31.4</td>
    <td>27.0</td>
    <td>6.7</td>
    <td>2.2</td>
    <td>2.9</td>
    <td>9.7</td>
  </tr>
  <tr>
    <td>+Refine</td>
    <td>33.8</td>
    <td>23.4</td>
    <td>30.3</td>
    <td>32.8</td>
    <td>27.5</td>
    <td>6.7</td>
    <td>2.5</td>
    <td>3.3</td>
    <td>9.5</td>
  </tr>
  <tr>
    <td>+TEaR</td>
    <td>33.8</td>
    <td>23.4</td>
    <td>30.3</td>
    <td>32.8</td>
    <td>27.5</td>
    <td>6.7</td>
    <td>2.5</td>
    <td>3.3</td>
    <td>9.5</td>
  </tr>
  <tr>
    <td>+DUAT</td>
    <td>32.9</td>
    <td>21.7</td>
    <td>29.6</td>
    <td>32.4</td>
    <td>26.9</td>
    <td>6.6</td>
    <td>2.7</td>
    <td>3.4</td>
    <td>9.6</td>
  </tr>
  <tr>
    <td>+Dual-Reflect</td>
    <td>32.4</td>
    <td>20.2</td>
    <td>29.4</td>
    <td>31.9</td>
    <td>26.4</td>
    <td>6.5</td>
    <td>2.6</td>
    <td>3.2</td>
    <td>9.4</td>
  </tr>
  <tr>
    <td>+IBUT</td>
    <td><b>34.5</b></td>
    <td><b>24.3</b></td>
    <td><b>31.9</b></td>
    <td><b>34.3</b></td>
    <td><b>28.5</b></td>
    <td><b>6.9</b></td>
    <td><b>4.9</b></td>
    <td><b>4.7</b></td>
    <td><b>10.1</b></td>
  </tr>
</table>

Table 1: The main results from the WMT22 news benchmark are presented. ChatGPT mean to perform translation directly through Zero-Shot. The bold indicates the highest scores that are statistically significant, with p-values less than 0.01 in the paired t-test against all compared methods.

<table>
  <tr>
    <th>Culture</th>
    <th>En→Es</th>
    <th>En→Fr</th>
    <th>En→Hi</th>
    <th>En→Ta</th>
    <th>En→Te</th>
    <th>En→Zh</th>
  </tr>
  <tr>
    <th colspan="7">COMET †/BLEURT †/BLEU↑</th>
  </tr>
  <tr>
    <td>ChatGPT</td>
    <td>83.0 / 69.3 / 35.7</td>
    <td>77.9 / 58.3 / 31.1</td>
    <td>73.6 / 76.1 / 18.8</td>
    <td>67.9 / 57.4 / 11.3</td>
    <td>69.9 / 52.0 / 13.2</td>
    <td>83.3 / 64.5 / 35.0</td>
  </tr>
  <tr>
    <td>+5-shot</td>
    <td>83.2 / 70.3 / 44.0</td>
    <td>78.0 / 58.5 / 24.0</td>
    <td>74.3 / 63.7 / 18.7</td>
    <td>71.8 / 60.2 / 11.2</td>
    <td>70.6 / 53.6 / 13.3</td>
    <td>83.2 / 64.9 / 35.3</td>
  </tr>
  <tr>
    <td>+Rerank</td>
    <td>82.7 / 70.5 / 43.9</td>
    <td>78.1 / 58.2 / 24.5</td>
    <td>73.9 / 62.4 / 18.8</td>
    <td>70.5 / 59.4 / 11.2</td>
    <td>70.4 / 52.7 / 13.0</td>
    <td>83.0 / 64.6 / 34.4</td>
  </tr>
  <tr>
    <td>+MAD</td>
    <td>83.4 / <b>70.8</b> / 43.8</td>
    <td>78.5 / <b>59.0</b> / 31.0</td>
    <td>71.6 / 60.5 / 18.1</td>
    <td>71.1 / 60.3 / 11.5</td>
    <td>71.0 / 53.6 / 13.3</td>
    <td>83.6 / 64.7 / 34.5</td>
  </tr>
  <tr>
    <td>+MAPS</td>
    <td>82.9 / 70.0 / 42.1</td>
    <td>78.2 / 58.7 / 30.6</td>
    <td>71.8 / 60.4 / 11.9</td>
    <td>72.1 / 60.7 / 11.2</td>
    <td>72.0 / 54.8 / 13.6</td>
    <td>83.5 / 64.1 / 34.6</td>
  </tr>
  <tr>
    <td>+Refine</td>
    <td>83.0 / 70.1 / 42.1</td>
    <td>78.0 / 58.3 / 30.4</td>
    <td>74.3 / 63.2 / 18.8</td>
    <td>71.8 / 60.9 / 11.7</td>
    <td>71.7 / 54.6 / 13.7</td>
    <td>83.0 / 65.1 / 34.7</td>
  </tr>
  <tr>
    <td>+TEaR</td>
    <td>82.6 / 70.3 / 43.3</td>
    <td>77.1 / 58.7 / 30.2</td>
    <td>71.4 / 61.2 / 15.3</td>
    <td>71.9 / 59.3 / 10.5</td>
    <td>71.7 / 53.4 / 12.8</td>
    <td>83.2 / 64.3 / 35.0</td>
  </tr>
  <tr>
    <td>+DUAT</td>
    <td>84.0 / 70.3 / 43.7</td>
    <td>78.8 / 57.7 / 31.0</td>
    <td>73.6 / 62.7 / 16.5</td>
    <td>70.8 / 58.8 / 11.3</td>
    <td>70.9 / 54.6 / 13.6</td>
    <td>83.0 / 65.1 / 34.6</td>
  </tr>
  <tr>
    <td>+Dual-Reflect</td>
    <td>83.5 / 70.4 / 44.2</td>
    <td>77.9 / 57.1 / 31.3</td>
    <td>74.0 / 62.0 / 14.2</td>
    <td>70.3 / 58.6 / 10.4</td>
    <td>71.5 / 54.2 / 13.4</td>
    <td>83.2 / 65.3 / 35.1</td>
  </tr>
  <tr>
    <td>+IBUT</td>
    <td><b>84.0</b> / <b>70.7</b> / <b>44.6</b></td>
    <td><b>79.2</b> / <b>58.9</b> / <b>31.8</b></td>
    <td><b>75.0</b> / <b>64.3</b> / <b>19.3</b></td>
    <td><b>73.4</b> / <b>60.9</b> / <b>12.2</b></td>
    <td><b>73.4</b> / <b>55.4</b> / <b>14.6</b></td>
    <td><b>84.2</b> / <b>66.2</b> / <b>35.7</b></td>
  </tr>
</table>

Table 2: The main results from the cultural MT dataset are presented. The bold indicates the highest values that are statistically significant, with p-values less than 0.01 in the paired t-test against all compared methods.

<table>
  <tr>
    <th>Commonsense</th>
    <th>Zh→En</th>
  </tr>
  <tr>
    <th colspan="2">COMET †/BLEURT †/BLEU↑</th>
  </tr>
  <tr>
    <td>GPT4</td>
    <td>82.0 / 71.0 / 32.6</td>
  </tr>
  <tr>
    <td>ChatGPT</td>
    <td>79.7 / 68.2 / 29.8</td>
  </tr>
  <tr>
    <td>+5-shot</td>
    <td>79.6 / 68.5 / 28.7</td>
  </tr>
  <tr>
    <td>+Rerank</td>
    <td>80.9 / 69.1 / 29.9</td>
  </tr>
  <tr>
    <td>+MAPS</td>
    <td>81.9 / 69.4 / 27.2</td>
  </tr>
  <tr>
    <td>+Refine</td>
    <td>81.3 / 69.0 / 28.1</td>
  </tr>
  <tr>
    <td>+MAD</td>
    <td>82.0 / 70.8 / 29.1</td>
  </tr>
  <tr>
    <td>+DUAT</td>
    <td>82.6 / 71.8 / 31.9</td>
  </tr>
  <tr>
    <td>+Dual-Reflect</td>
    <td>82.2 / 71.8 / 28.4</td>
  </tr>
  <tr>
    <td>+TEaR</td>
    <td>81.5 / 68.3 / 28.4</td>
  </tr>
  <tr>
    <td>+IBUT</td>
    <td><b>83.9</b> / <b>72.7</b> / <b>32.6</b></td>
  </tr>
</table>

Table 3: The main results from the Commonsense MT benchmark are presented. The bold indicates the highest values, statistically significant with p-values less than 0.01 in the paired t-test against compared methods.

in commonsense MT tasks, achieving the best translation performance. Compared to the MAPS method, IBUT improved by +2 in the COMET metric, demonstrating an enhanced ability to generate higher-quality contextual understanding. Moreover, IBUT surpassed the MAD method, which relies on multi-agent debate feedback, showing its outstanding feedback quality. Notably, in translation tasks involving logical reasoning, IBUT’s performance was even better than GPT-4, fully showcasing its exceptional reasoning ability.

5.3 Automated Evaluation of Understanding Distortion and Translation Performance

This study explored the positive impact of reducing understanding distortion issues in bilin-
Figure 3: The experiment measures the relationship between the improvement in contextual understanding quality and translation performance during iterative refinement.

gual contextual understanding on translation performance using IBUT. We randomly selected a set of 200 Chinese→English translation sentence pairs from the Commonsense MT dataset, which provides a test subset for lexical ambiguity. Based on the subset, IBUT iterated 8 times (\( max\_iter = 8 \)), saving the results of bilingual contextual understanding and translation COMET scores after each iteration.

As shown in Figure 3, the vertical axis represents the translation performance, measured as the COMET score. The horizontal axis represents the scores evaluated by GPT-4 for the quality of bilingual contextual understanding affected by understanding distortion issues, with a maximum score of 10. The score for the source language is \( v_s \) and for the target language is \( v_t \), while the overall score \( v \) is the average of the two (\( v = \frac{v_s + v_t}{2} \)). Details on the evaluation prompt can be found in Appendix B.3.

The experimental results, as shown in Figure 3, demonstrate a positive correlation between the quality of contextual understanding and translation performance. Additionally, as the number of iterations increases, the quality of contextual understanding progressively improves, indicating that the IBUT method effectively reduces understanding distortion issues.

5.4 Impact of Iterative Refinement on Translation Performance

To further verify the impact of the Iterative Refinement part on overall translation performance, we conducted experiments on Cultural MT (En→Zh) and Commonsense MT (Zh→En), comparing methods like MAD and Refine to iteratively enhance translation quality. We set the maximum number of iterations at 9 and required that each iteration in the Iterative Refinement part obtain a new translation COMET score, rather than allowing adaptive termination in the Alignment Judgment part.

The experimental results, as shown in Figure 4, first indicate that IBUT surpasses the comparative methods in translation performance in most iterations, further proving the effectiveness of the method. Secondly, compared to the comparative methods, IBUT progressively enhances its performance in each iteration, demonstrating that the dual learning of translation can provide positive supervision signals in each iteration.

Figure 4: Analysis of the experimental setup for assessing the impact of the Iterative Refinement part on translation performance.

To illustrate this iterative refinement more clearly, Table 14 (in Appendix B.10) presents three
cases where translations were correctly refined after a single iteration. These examples highlight how bilingual supervision signals contribute to enhancing translation quality through iteration.

5.5 Human Evaluation

Human Evaluation of Understanding Distortion Issue. In the human evaluation of understanding distortion issue, this study follows the method of Huang et al., 2024 and Chen et al., 2024a to assess translation outcomes from two main dimensions: accuracy in ambiguity resolution (commonsense domain) and the statistical results of understanding distortion issue (see Appendix B.7 for experimental setup details).

The experimental results are shown in Table 4. Understanding distortion issues accounts for a significant proportion (40%). Our method (IBUT) significantly addressed these failures, with a success rate of approximately 89%, demonstrating the effectiveness of our method. Additionally, in terms of ambiguity resolution accuracy, IBUT outperformed the baseline by 13 acc points, indicating that bilingual understanding and iterative refinement contribute to enhancing ambiguity resolution capabilities in MT tasks.

<table>
  <tr>
    <th>Methods</th>
    <th colspan="2">Human Evaluation</th>
  </tr>
  <tr>
    <th></th>
    <th>Nums</th>
    <th>ACC †</th>
  </tr>
  <tr>
    <td>Understanding Distortion of Baseline</td>
    <td>28 (40%)</td>
    <td>65.5</td>
  </tr>
  <tr>
    <td>Understanding Distortion of IBUT</td>
    <td>3 (-89%)</td>
    <td><b>79.0</b></td>
  </tr>
</table>

Table 4: The human-annotated results of the Commonsense MT benchmark. Baseline refers to the MAPS method modified into the form shown in Figure 1(a). In the baseline method, there are 70 sentences with translation errors.

To better understand the limitations of the IBUT methods, Table 5 presents three sentences where IBUT still made translation errors in this experiment and analyzes them through human-annotated. These negative examples show that accurate translation depends on the source and target language achieving correct understanding through multiple iterations. If the LLMs misunderstand complex sentences during these iterations, translation errors will occur.

Transaltion Quality. In human evaluation of translation quality, this study adopted the method (Liang et al., 2023) to validate translation quality on both the En→Zh and Zh→En test sets of the Cultural MT and the Commonsense MT dataset (Appendix B.7 for experimental setup details).

<table>
  <tr>
    <th>No.</th>
    <th>Human-annotated</th>
    <th>Examples: Source/Error Result/Reference</th>
  </tr>
  <tr>
    <td>1</td>
    <td>Nuanced translation errors arise from lack of deep cultural understanding, leading to the loss of core meaning.</td>
    <td>Source: 加油不要松手，就快到家了<br>Error: If you don’t put in the effort, you won’t be able to save poor learning.<br>Right: If you don’t study hard, you can’t do scholarly research.</td>
  </tr>
  <tr>
    <td>2</td>
    <td>Although LLM grasp that “觀光” implies “intuitively,” textual noise hinders correction of mistranslations.</td>
    <td>Source: 要注意戶外活動的各種風險<br>Error: Peddling the bourgeoisie’s spiritual opium.<br>Right: Inculcate the spiritual opium of the bourgeoisie.</td>
  </tr>
  <tr>
    <td>3</td>
    <td>Iterative translation struggles to understand the meaning of “能人” in Chinese, leading to mistakes.</td>
    <td>Source: 你家別亂火了，到我家吃飯吧<br>Error: The young gallants are new-born bucks in chase of the hunt.<br>Right: Young ones are like rabbits, new to the hunt, Born in a patch of grass, on sandy ground</td>
  </tr>
</table>

Table 5: Translation Errors with Examples.

The experimental results are displayed in Figure 5. Within the Commonsense MT Dataset, IBUT performed best in terms of ambiguity resolution accuracy, thereby achieving higher human evaluation scores compared to other methods. In the Cultural MT Dataset, IBUT received higher human evaluation scores, indicating that its generated contextual understanding effectively enhances the performance of culturally translation tasks.

![Human preference study comparing ChatGPT, Refine, MAPS, and MAD.](page_1012_1042_388_180.png)

Figure 5: Human preference study comparing ChatGPT, Refine, MAPS, and MAD.

5.6 Effectiveness of Bilingual Contextual Understanding and Ablation Experiments

The IBUT introduced bilingual contextual understanding based on the source sentence to improve translation performance. To evaluate the effects of bilingual contextual understanding, we designed 5 control methods: (a) LLM-MT directly translating (ChatGPT); (b) LLM generating contextual understanding based on the source language, translated by LLM-MT (SRC); (c) LLM generating contextual understanding based on the target language, translated by LLM-MT (TGT); (d) LLM generating contextual understanding for both source and target languages, translated by LLM-MT (SRC+TGT); (e) using the IBUT method described in section 3.

The effectiveness of Bilingual Contextual Understanding. Figure 6 shows that on WMT22 and cultural MT datasets, translation based on contextual understanding outperforms baseline methods, validating our research direction. Bilingual (SRC+TGT) contextual understanding
Figure 6: Effectiveness of Bilingual Contextual Understanding and Ablation Experiments. On the left are results for four language pairs from WMT22, and on the right are results for five language pairs from cultural MT. ChatGPT for direct translation; SRC for translation based on source language understanding; TGT for translation based on target language understanding; SRC+TGT for translation based on both source and target language understanding; IBUT as proposed method in section 3.

notably improves performance over monolingual (SRC or TGT) understanding. Furthermore, target language (TGT) understanding has a greater impact on translation quality than source language (SRC) understanding.

Ablation Experiments on IBUT Components. Figure 6 shows that using only the Understanding Generation component ("SRC or TGT") or skipping iterative refinement ("SRC+TGT") leads to inferior performance. These results validate the design rationale and effectiveness of the IBUT.

5.7 IBUT Demonstrates Generalizability in Model Selection

<table>
  <tr>
    <th rowspan="2">WMT22</th>
    <th colspan="2">En→De</th>
    <th colspan="2">En→Ja</th>
  </tr>
  <tr>
    <th>COMET↑/BLEURT↑</th>
    <th>/BLEURT↑</th>
    <th>COMET↑/BLEURT↑</th>
    <th>/BLEURT↑</th>
  </tr>
  <tr>
    <td>Alpaca-7B</td>
    <td>75.5 / 62.2 / 11.3</td>
    <td>56.6 / 31.4 / 6.6</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>+5shot</td>
    <td>76.3 / 62.8 / 12.1</td>
    <td>57.9 / 31.9 / 7.0</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>+MAPS</td>
    <td>76.7 / 63.5 / 12.6</td>
    <td>58.3 / 33.9 / 7.5</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>+IBUT</td>
    <td><b>78.4 / 64.9 / 13.1</b></td>
    <td><b>61.3 / 34.8 / 8.2</b></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Vicuna-7B</td>
    <td>79.8 / 67.4 / 15.2</td>
    <td>82.3 / 58.7 / 9.4</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>+5shot</td>
    <td>80.3 / 67.8 / 15.3</td>
    <td>83.3 / 59.7 / 9.5</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>+MAPS</td>
    <td>81.1 / 68.0 / 16.0</td>
    <td>84.1 / 60.3 / 9.8</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>+IBUT</td>
    <td><b>82.0 / 69.1 / 17.3</b></td>
    <td><b>85.1 / 61.1 / 11.0</b></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Qwen2.5-7B</td>
    <td>62.5 / 43.4 / 15.2</td>
    <td>64.5 / 31.7 / 7.1</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>+5shot</td>
    <td>62.6 / 43.6 / 15.3</td>
    <td>64.0 / 31.7 / 7.3</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>+MAPS</td>
    <td>62.3 / 43.3 / 15.2</td>
    <td>64.5 / 31.8 / 7.3</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>+IBUT</td>
    <td><b>63.2 / 44.7 / 16.0</b></td>
    <td><b>66.1 / 33.0 / 9.2</b></td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 6: The experimental results of IBUT on open-source models. The bold indicates the highest values that are statistically significant, with p-values less than 0.01 in the paired t-test against all compared methods.

To validate the generalizability of the IBUT method on open-source models, we selected two open-source models (Alpaca and Vicuna) for experimental verification. The experimental results, as shown in Table 6, indicate that the overall performance trends of the two open-source models are consistent with those observed using the GPT3.5 model. This demonstrates the generalizability of the IBUT method in open-source models. Additionally, we further validated the effectiveness of the IBUT method in GPT-4.

The results are shown in Appendix B.6.

5.8 Computational Resource Analysis

Since the IBUT method requires multiple iterative steps, it is necessary to discuss and analyze its resource consumption. For token consumption, we used the gpt-3.5-turbo tokenizer9 to tokenize and then calculated the token consumption of the comparative methods requiring iteration on the commonsense dataset.

<table>
  <tr>
    <th>Methods</th>
    <th>Avg Output ↓</th>
    <th>COMET ↑/BLEURT ↑</th>
    <th>/BLEURT ↑</th>
  </tr>
  <tr>
    <td>ChatGPT</td>
    <td>24.4</td>
    <td>79.7 / 68.2 / 29.8</td>
    <td></td>
  </tr>
  <tr>
    <td>+5-shot</td>
    <td>35.6</td>
    <td>79.9 / 68.6 / 30.2</td>
    <td></td>
  </tr>
  <tr>
    <td>+MAPS</td>
    <td>172.2</td>
    <td>81.9 / 69.4 / 27.2</td>
    <td></td>
  </tr>
  <tr>
    <td>+MAD</td>
    <td>224.4</td>
    <td>82.0 / 70.8 / 29.1</td>
    <td></td>
  </tr>
  <tr>
    <td>+IBUT</td>
    <td><b>209.3</b></td>
    <td><b>83.9 / 72.7 / 32.6</b></td>
    <td></td>
  </tr>
</table>

Table 7: The statistics of methods inference cost on the commonsense dataset.

Table 7 shows that the IBUT method increases token consumption by 5 times compared to the 5-shot method, yet achieves substantial performance gains in COMET/BLEURT/BLEU metrics (+4.0/+4.1/+2.4). IBUT performs comparably to strong methods like MAD and MAPS, with an average improvement of 2 points. The computational trade-offs of long-context processing and inference time are detailed in Appendix B.1 and Appendix B.2, respectively. This limitation is discussed in the Limitations section as a future research direction for MT.

6 Conclusion

This paper presents Iterative Bilingual Understanding Translation (IBUT), a method for improving LLM-based machine translation (LLM-MT) by addressing Understanding Distortion issue. IBUT generates bilingual contextual understanding, uses dual learning to create a

9https://github.com/openai/tiktoken
supervisory signal, and iteratively refines the understanding to enhance translation performance. The method shows strong results across general news, commonsense, and cultural MT tasks, with human evaluations validating its effectiveness.

Limitations

The IBUT method has several limitations. Firstly, models with stronger understanding and generation capabilities will obtain better contextual understanding, thereby enhancing translation performance. Additionally, since our method requires multiple steps, it necessitates a significant amount of computational resources.

7 Acknowledgements

We want to thank all the anonymous reviewers for their valuable comments. The work was supported by the National Natural Science Foundation of China under Grant (62376075 and 62276077), Guangdong Basic and Applied Basic Research Foundation (2024A1515011205), and Shenzhen Science and Technology Program (GXWD20220811170358002, GXWD20220817123150002, KQTD20240729102154066, and ZDSYS20230626091203008).

References

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shaymal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.

Philip Arthur, Graham Neubig, and Satoshi Nakamura. 2016. Incorporating discrete translation lexicons into neural machine translation. arXiv preprint arXiv:1606.02006.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.

Andong Chen, Lianzhang Lou, Kehai Chen, Xuefeng Bai, Yang Xiang, Muyun Yang, Tiejun Zhao, and Min Zhang. 2024a. DUAL-REFLECT: Enhancing large language models for reflective translation through dual learning feedback mechanisms. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 693–704, Bangkok, Thailand. Association for Computational Linguistics.

Andong Chen, Yuchen Song, Kehai Chen, Muyun Yang, Tiejun Zhao, and Min Zhang. 2024b. Make imagination clearer! stable diffusion-based visual imagination for multimodal machine translation. CoRR, abs/2412.12627.

Pinzhen Chen, Zhicheng Guo, Barry Haddow, and Kenneth Heafield. 2023. Iterative translation refinement with large language models. arXiv preprint arXiv:2306.03856.

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.

Zhaopeng Feng, Yan Zhang, Hao Li, Wenqiang Liu, Jun Lang, Yang Feng, Jian Wu, and Zuozhu Liu. 2024. Improving llm-based machine translation with systematic self-correction. CoRR, abs/2402.16379.

Yuan Gao, Ruili Wang, and Feng Hou. 2023. How to design translation prompts for chatgpt: An empirical study. arXiv e-prints, pages arXiv–2304.

Xavier Garcia and Orhan Firat. 2022. Using natural language prompts for machine translation. arXiv preprint arXiv:2202.11822.

Nuno Miguel Guerreiro, Duarte M. Alves, Jonas Waldendorf, Barry Haddow, Alexandra Birch, Pierre Colombo, and André F. T. Martins. 2023. Hallucinations in large multilingual translation models. CoRR, abs/2303.16104.

Di He, Yingce Xia, Tao Qin, Liwei Wang, Nenghai Yu, Tie-Yan Liu, and Wei-Ying Ma. 2016. Dual learning for machine translation. Advances in neural information processing systems, 29.

Jie He, Tao Wang, Deyi Xiong, and Qun Liu. 2020. The box is in the pen: Evaluating commonsense reasoning in neural machine translation. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3662–3672, Online. Association for Computational Linguistics.

Zhiwei He, Tian Liang, Wenxiang Jiao, Zhuosheng Zhang, Yuju Yang, Rui Wang, Zhaopeng Tu, Shuming Shi, and Xing Wang. 2023. Exploring human-like translation strategy with large language models. ArXiv, abs/2305.04118.

Zhiwei He, Tian Liang, Wenxiang Jiao, Zhuosheng Zhang, Yuju Yang, Rui Wang, Zhaopeng Tu, Shuming Shi, and Xing Wang. 2024. Exploring human-like translation strategy with large language models. Transactions of the Association for Computational Linguistics, 12:229–246.

Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. 2023. How good are gpt models at
machine translation? a comprehensive evaluation. arXiv preprint arXiv:2302.09210.

Junjie Hu, Hiroaki Hayashi, Kyunghyun Cho, and Graham Neubig. 2021. Deep: denoising entity pre-training for neural machine translation. arXiv preprint arXiv:2111.07393.

Junjie Hu, Mengzhou Xia, Graham Neubig, and Jaime Carbonell. 2019. Domain adaptation of neural machine translation by lexicon induction. arXiv preprint arXiv:1906.00376.

Yichong Huang, Xiaocheng Feng, Baohang Li, Chengpeng Fu, Wenshuai Huo, Ting Liu, and Bing Qin. 2024. Aligning translation-specific understanding to general understanding in large language models. arXiv preprint arXiv:2401.05072.

Vivek Iyer, Pinzhen Chen, and Alexandra Birch. 2023. Towards effective disambiguation for machine translation with large language models. In Proceedings of the Eighth Conference on Machine Translation, WMT 2023, Singapore, December 6-7, 2023, pages 482–495. Association for Computational Linguistics.

Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, and Zhaopeng Tu. 2023. Is chatgpt a good translator? a preliminary study. arXiv preprint arXiv:2301.08745, 1(10).

Marzena Karpinska and Mohit Iyyer. 2023. Large language models effectively leverage document-level context for literary translation, but critical errors persist. In Proceedings of the Eighth Conference on Machine Translation, WMT 2023, Singapore, December 6-7, 2023, pages 419–451. Association for Computational Linguistics.

Urvashi Khandelwal, Angela Fan, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020. Nearest neighbor machine translation. arXiv preprint arXiv:2010.00710.

Tom Kocmi, Eleftherios Avramidis, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, Barry Haddow, Philipp Koehn, Benjamin Marie, Christof Monz, Makoto Morishita, Kenton Murray, Makoto Nagata, Yoshiaki Nakazawa, Martin Popel, Maja Popović, and Mariya Shmatova. 2023. Findings of the 2023 conference on machine translation (WMT23): LLMs are here but not quite there yet. In Proceedings of the Eighth Conference on Machine Translation, pages 1–42, Singapore. Association for Computational Linguistics.

Tom Kocmi, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Thamme Gowda, Yvette Graham, Roman Grundkiewicz, Barry Haddow, et al. 2022. Findings of the 2022 conference on machine translation (wmt22). In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 1–45.

Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2023. Bloom: A 176b-parameter open-access multilingual language model.

Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Zhaopeng Tu, and Shuming Shi. 2023. Encouraging divergent thinking in large language models through multi-agent debate. arXiv preprint arXiv:2305.19118.

Hongyuan Lu, Haoyang Huang, Dongdong Zhang, Haoran Yang, Wai Lam, and Furui Wei. 2023. Chain-of-dictionary prompting elicits translation in large language models. arXiv preprint arXiv:2305.06575.

Yasmin Moslem, Rejwanul Haque, John D. Kelleher, and Andy Way. 2023a. Adaptive machine translation with large language models. In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, EAMT 2023, Tampere, Finland, 12-15 June 2023, pages 227–237. European Association for Machine Translation.

Yasmin Moslem, Rejwanul Haque, and Andy Way. 2023b. Adaptive machine translation with large language models. arXiv preprint arXiv:2301.13294.

Yasmin Moslem, Gianfranco Romani, Mahdi Molaei, John D. Kelleher, Rejwanul Haque, and Andy Way. 2023c. Domain terminology integration into machine translation: Leveraging large language models. In Proceedings of the Eighth Conference on Machine Translation, WMT 2023, Singapore, December 6-7, 2023, pages 902–911. Association for Computational Linguistics.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.

Tao Qin. 2020. Dual learning. Springer.

Leonardo Ranaldi, Giulia Pucci, and André Freitas. 2023. Empowering cross-lingual abilities of instruction-tuned large language models by translation-following demonstrations. CoRR, abs/2308.14186.

Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. 2022a. COMET-22: Unbabel-IST 2022 submission for the metrics shared task. In Proceedings of the Seventh Conference on Machine Translation
(WMT), pages 578–585. Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.

Ricardo Rei, Marcos Treviso, Nuno M. Guerreiro, Chrysoula Zerva, Ana C Farinha, Christine Maroti, José G. C. de Souza, Taisiya Glushkova, Duarte Alves, Luisa Coheur, Alon Lavie, and André F. T. Martins. 2022b. CometKiwi: IST-unbabel 2022 submission for the quality estimation shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 634–645. Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.

Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881–7892, Online. Association for Computational Linguistics.

Seongjin Shin, Sang-Woo Lee, Hwijeen Ahn, Sungdong Kim, HyoungSeok Kim, Boseop Kim, Kyunghyun Cho, Gichang Lee, Woomyoung Park, Jung-Woo Ha, et al. 2022. On the effect of pretraining corpora on in-context learning by a large-scale language model. arXiv preprint arXiv:2204.13509.

Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2024. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36.

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca.

Qwen Team. 2024. Qwen2.5: A party of foundation models.

Gladys Tyen, Hassan Mansoor, Peter Chen, Tony Mak, and Victor Carbune. 2023. Llms cannot find reasoning errors, but can correct them! arXiv preprint arXiv:2311.08516.

Wenyi Wang, Hisham A. Alyahya, Dylan R. Ashley, Oleg Serikov, Dmitri Khizhulin, Francesco Faccio, and Jürgen Schmidhuber. 2024. How to correctly do semantic backpropagation on language-based agentic systems. CoRR, abs/2412.03624.

Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022. Finetuned language models are zero-shot learners. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.

Minghao Wu, Yulin Yuan, Gholamreza Haffari, and Longyue Wang. 2024. beyond human translation: Harnessing multi-agent collaboration for translating ultra-long literary texts. arXiv preprint arXiv:2405.11804.

Binwei Yao, Ming Jiang, Diyi Yang, and Junjie Hu. 2023. Empowering llm-based machine translation with cultural awareness. abs/2305.14328.

Jiali Zeng, Fandong Meng, Yongjing Yin, and Jie Zhou. 2023. Improving machine translation with large language models: A preliminary study with cooperative decoding. CoRR, abs/2311.02851.

Hongbin Zhang, Kehai Chen, Xuefeng Bai, Yang Xiang, and Min Zhang. 2024. Paying more attention to source context: Mitigating unfaithful translations from large language model. In Findings of the Association for Computational Linguistics ACL 2024, pages 13816–13836, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.

Tiejun Zhao, Muven Xu, and Antony Chen. 2024. A review of natural language processing research. Journal of Xinjiang Normal University (Philosophy and Social Sciences), pages 1–23.
A  Experiment Setup

A.1 Detailed prompt for part-1

Part-1: Understanding Generation: Please fully understand the meaning of the following \( L^s \) text from your memory and describe your understanding of key concepts, definitions, examples, and explanations of specific terms related to the translation task in \( L^s/L^t \):s

Input Text:
Source Sentence s

Output Text:
\( C_s \) or \( C_t \)

A.2 Detailed prompt for part-2

Part-2: Alignment Judgment-1: If you are a \( L^s \) and \( L^t \) linguist, determine whether provided source contextual understanding \( C_s \) and target contextual understanding \( C_t \), based on the source sentence \( s \), convey different key concepts, definitions, examples, and explanations of specific terms related to the translation task. If so, provide a 'True' response; otherwise, give a 'False' response.

Input Text:
Source Sentence s and source/target contextual understanding \( C_s/C_t \)

Output Text:
True or False

Part-2: Alignment Judgment-2: If you are a linguist proficient in both \( L^s \) and \( L^t \), based on the core meaning of the source sentence \( s \), analyze the source contextual understanding \( C_s \) / the target contextual understanding \( C_t \). Generate verbal feedback in the language of \( C_s/C_t \) to correct any current errors in \( C_s/C_t \).

Input Text:
Source Sentence s, source/target language understanding \( C_s/C_t \)

Output Text:
\( F_s \) or \( F_t \)

A.3 Detailed prompt for part-3

Part-3: Iterative Refinement: If you are a linguist proficient in both \( L^s \) and \( L^t \), based on the core meaning of the source sentence s and the opinions from \( F_s/F_t \), further modify the current \( C_s/C_t \).

Input Text:
Source Sentence s, source/target contextual understanding \( C_s/C_t \) and source/target verbal feedback \( F_s/F_t \)

Output Text:
\( C_s \) or \( C_t \)

A.4 Detailed prompt for part-4

Part-4: Understanding-Based Translation: Based on \( C_s \) and \( C_t \), translate the following text from \( L^s \) to \( L^t \).

Input Text:
Source Sentence s, source/target contextual understanding \( C_s/C_t \)

Output Text:
Target Sentence t

A.5 Dataset Detail

For the WMT22 test set (Kocmi et al., 2022), the experimental analysis covers 9 language pairs. We used the full test dataset. Among these languages, Sah↔Ru, Uk↔Cs, En→Hr and En↔Liv are classified as low-resource languages, respectively.

For the WMT23 test set (Kocmi et al., 2023), the experimental analysis covers 4 language pairs. We used the full test dataset. Among them, En→De and En→Ja are identified as high and medium-resource languages, with the former belonging to the same language family and the latter exhibiting significant differences.

The Commonsense MT dataset (He et al., 2020) encompasses vocabulary that requires common knowledge for resolution, along with instances of ambiguity in Zh→En translation data. Each translation data includes a source sentence and two contrasting translations, involving seven different types of common knowledge. Although these sentences appear suitable for direct translation, they often lead to misleading interpretations.

The cultural MT dataset (Yao et al., 2023) introduces a culturally relevant parallel corpus, enriched with annotations of cultural-specific items. This dataset encompasses 6 language pairs: En→Es, En→Fr, En→Hr, En→Ta, En→Te, and En→Zh. It also encompasses over 7,000 cultural-
specific items from 18 concept categories across more than 140 countries and regions.

A.6 Comparative Methods
The following content will provide detailed descriptions of these comparative methods:

• Baseline is standard zero-shot translation performed in ChatGPT (Ouyang et al., 2022) and GPT-4 (Achiam et al., 2023). The temperature parameter set to 0, which is the default value for our experiments.

• 5-Shot (Hendy et al., 2023) involves prepending five high-quality labelled examples from the training data to the test input.

• Rerank (Moslem et al., 2023a) was conducted with the identical prompt as the baseline, employing a temperature of 0.3 (Moslem et al., 2023b). Three random samples were generated and combined with the baseline to yield four candidates. The best candidate was chosen through GPT-4.

• Refine (Chen et al., 2023) first requests a translation from ChatGPT, then provides the source text and translation results, and obtains a refined translation through multiple rounds of modifications.

• MAPS (He et al., 2023) incorporate the knowledge of keywords, topic words, and demonstrations similar to the given source sentence to enhance the translation process.

• Dual-Reflect (Chen et al., 2024a) provide supervisory signals for large models to reflect on translation results through dual learning, hereby iteratively improving translation performance (the maximum number of iterations is set to 5).

• TEaR (Feng et al., 2024) propose the first systematic and effective LLM-based self-refinement translation framework.

• DUAT (Huang et al., 2024) optimizes the translation of difficult-to-translate words through cross-lingual interpretation and enhances its performance by integrating external tools for better word detection and interpretation generation.

• MAD (Liang et al., 2023) enhance the capabilities of large language models (LLMs) by encouraging divergent thinking. In this method, multiple agents engage in a debate, while an agent oversees the process to derive a final solution.

• IBUT is proposed method in Sec.3. The method uses only ChatGPT with a max number of iterations set to 8 (\(max\_iter = 8\)).

B Experiment Results

B.1 Performance and Overhead of Long-Context Processing

In the commonsense test datasets, the benchmark includes only one bilingual meaning word per sentence to better evaluate performance. To further analyze the performance and computational overhead of complex long-context processing, we concatenated \(N\) sentences from the commonsense test datasets to create longer sentences. For instance, \(N = 3\) means three source sentences are combined into one longer sentence. We then evaluated this modified dataset, and the results are shown in Table 8.

<table>
  <tr>
    <th>Method</th>
    <th>Avg I/O ↓</th>
    <th>COMET ↑/BLEURT ↑/BLEU ↑</th>
  </tr>
  <tr>
    <td colspan="3">\(N = 2\)</td>
  </tr>
  <tr>
    <td>ChatGPT</td>
    <td>28.7 / 72.0</td>
    <td>72.2 / 61.4 / 23.8</td>
  </tr>
  <tr>
    <td>+MAPS</td>
    <td>351.5 / 407.1</td>
    <td>76.9 / 66.3 / 26.1</td>
  </tr>
  <tr>
    <td>+MAD</td>
    <td>433.4 / 624.1</td>
    <td>78.4 / 67.1 / 25.6</td>
  </tr>
  <tr>
    <td>+IBUT</td>
    <td>456.8 / 613.0</td>
    <td>80.4 / 68.9 / 27.4</td>
  </tr>
  <tr>
    <td colspan="3">\(N = 3\)</td>
  </tr>
  <tr>
    <td>ChatGPT</td>
    <td>37.9 / 57.7</td>
    <td>72.1 / 60.2 / 22.7</td>
  </tr>
  <tr>
    <td>+MAPS</td>
    <td>481.4 / 499.7</td>
    <td>75.1 / 66.2 / 25.4</td>
  </tr>
  <tr>
    <td>+MAD</td>
    <td>610.9 / 675.4</td>
    <td>77.2 / 66.2 / 25.8</td>
  </tr>
  <tr>
    <td>+IBUT</td>
    <td>510.3 / 609.2</td>
    <td>78.6 / 67.8 / 27.2</td>
  </tr>
</table>

Table 8: Evaluation Results for Different Methods with \(N = 2\) and \(N = 3\)

The experimental results demonstrate that IBUT outperforms both direct translation by LLMs and other multi-step LLM-MT methods, even when handling longer sentences containing multiple bilingual meaning words. For more complex and lengthy sentences, IBUT’s computational overhead increases significantly due to the need to generate more concepts or terms. However, its translation performance remains superior. Therefore, developing more efficient and resource-efficient methods is an important direction for future research.
B.2 Computational Costs

We illustrate with our method based on Vicuna-7B, using a single A100 GPU with 80G. Our proposed IBUT method has an inference speed of 6.71s/sample with a batch size of 2 and memory usage of 17657MiB. If using Vicuna-7B for zero-shot inference, under the same batch size settings, the inference speed is 4.72s/sample with memory usage of 14965MiB.

B.3 The Experiment Setting of Error Reduction and Translation Enhancement

For the Commonsense MT lexical ambiguity subset, first manually annotate the correct understanding of ambiguous words. The annotated data includes the source language Chinese and the target language English. Details of the scoring prompt for GPT-4, focusing on the reduction of error in bilingual contextual understanding after iterative refinement, are as follows:

Prompt for GPT-4 Evaluation Please evaluate the source input s, contextual understanding \( C_s/C_t \), and the manually annotated meanings of lexical ambiguities to assess if the contextual understanding includes error content to the translation.
Scoring Guide:
1-2 points: The contextual understanding completely deviates from the source input, leading to generated content that is severely incorrect or irrelevant.
3-4 points: The contextual understanding partially deviates from the source input, resulting in partially relevant content with evident issues.
5-6 points: Although the contextual understanding does not completely deviate, there are errors in the interpretation of the source input, leading to content that is partially correct but flawed.
7-8 points: The contextual understanding is fundamentally accurate, correctly handles the source input, and the generated content is largely correct with only minor errors.
9-10 points: The contextual understanding is completely accurate, perfectly handles the source input and lexical ambiguities, and the generated content fully meets the requirements, successfully avoiding irrelevant content.
Based on these guidelines, score the model response from 0 to 10. Provide only the total score (just a number), without scores or explanations for each aspect. The score is ___.

Input Text:
Source Sentence s, source/target context understanding \( C_s/C_t \)

Output Text:
The score is __

B.4 Results on WMT23

To further validate the generalizability of the method, we conducted experiments on the WMT23 test set. The experimental results are shown in Table 9.

B.5 Results on Reference-free metric

To further clarify the robustness of our evaluation, we incorporated COMET-KIWI\(^{10}\) (Rei et al., 2022b), a reference-free metric in the COMET series. The experimental results are shown in Table 10.

These results demonstrate that our method still outperforms comparison methods in terms of COMET-KIWI scores, thereby further confirming the robustness of our evaluation.

B.6 General Performance

To demonstrate the generalizability of the method, we conducted experiments in Section 5.7, verifying that IBUT is effective not only on closed-source models but also on open-source models. Finally, since GPT-4 is an updated model of GPT-3.5, our method’s effectiveness on GPT-3.5 theoretically implies effectiveness on GPT-4. To further illustrate this point, we conducted experiments on GPT-4 for commonsense MT. The experimental results are shown in Table 11.

The experimental results demonstrate that our method achieves significant improvements when applied to GPT-4, thereby indicating the generalizability of our approach.

B.7 Human Evaluations

Human Evaluation of Understanding Distortion Issue. In this section, we conduct a human evaluation to assess translation quality, focusing on understanding distortion issues and ambiguity resolution. To ensure a rigorous evaluation, we invited three professional translators with extensive experience in machine translation evaluation and at least two years of practical experience in translation studies as an annotator. The test data consists of 200 randomly selected sentences from CommonsenseMT. The evaluation process follows a structured approach to ensure consistency and reduce subjective biases. Before the assessment, the annotator participates in a calibration session where they review the scoring criteria, examine representative examples, and discuss the key

10https://github.com/Unbabel/COMET
<table>
  <tr>
    <th>Metrics</th>
    <th>WMT23</th>
    <th>En→De</th>
    <th>En→Ja</th>
    <th>En→He</th>
    <th>Cs→Uk</th>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>COMET ↑/BLEURT ↑</td>
    <td>COMET ↑/BLEURT ↑</td>
    <td>COMET ↑/BLEURT ↑</td>
    <td>COMET ↑/BLEURT ↑</td>
  </tr>
  <tr>
    <td>ChatGPT</td>
    <td>83.5/69.1/39.7</td>
    <td>87.3/60.2/9.7</td>
    <td>82.1/69.3/22.3</td>
    <td>86.7/74.1/27.2</td>
  </tr>
  <tr>
    <td>+5shot</td>
    <td>83.7/69.4/40.1</td>
    <td>87.8/61.5/10.1</td>
    <td>82.5/69.8/22.5</td>
    <td>87.3/74.5/27.5</td>
  </tr>
  <tr>
    <td>+MAD</td>
    <td>83.9/70.3/41.6</td>
    <td>88.0/63.1/9.4</td>
    <td>82.9/70.0/24.0</td>
    <td>87.5/74.9/28.5</td>
  </tr>
  <tr>
    <td>+MAPS</td>
    <td>83.6/69.9/42.1</td>
    <td>87.9/62.6/9.8</td>
    <td>82.5/69.3/23.1</td>
    <td>87.8/74.6/28.0</td>
  </tr>
  <tr>
    <td>+Refine</td>
    <td>83.5/68.9/41.8</td>
    <td>87.6/62.4/10.8</td>
    <td>82.3/68.8/23.7</td>
    <td>87.3/74.1/28.3</td>
  </tr>
  <tr>
    <td>+IBUT</td>
    <td><b>84.3/71.8/42.6</b></td>
    <td><b>88.5/63.8/14.0</b></td>
    <td><b>83.1/72.1/24.9</b></td>
    <td><b>88.1/77.9/30.4</b></td>
  </tr>
</table>

Table 9: The main results from WMT23 are shown. The highest values are in bold, with p-values less than 0.01.

<table>
  <tr>
    <th>Methods</th>
    <th>En-De</th>
    <th>En-Ja</th>
    <th>Cs-Uk</th>
    <th>En-Hr</th>
  </tr>
  <tr>
    <td>ChatGPT</td>
    <td>82.1</td>
    <td>84.4</td>
    <td>83.6</td>
    <td>83.4</td>
  </tr>
  <tr>
    <td>+Rerank</td>
    <td>82.4</td>
    <td>84.2</td>
    <td>83.0</td>
    <td>83.4</td>
  </tr>
  <tr>
    <td>+MAPS</td>
    <td>82.0</td>
    <td>83.7</td>
    <td>83.6</td>
    <td>83.3</td>
  </tr>
  <tr>
    <td>+MAD</td>
    <td>83.6</td>
    <td>84.7</td>
    <td>84.2</td>
    <td>83.8</td>
  </tr>
  <tr>
    <td>+IBUT</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 10: WMT22 evaluation results on COMET-KIWI metric.

<table>
  <tr>
    <th>Methods</th>
    <th>COMET ↑/BLEURT ↑ /BLEUE ↑</th>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>82.0/71.0/32.6</td>
  </tr>
  <tr>
    <td>+5 shot</td>
    <td>82.3/71.5/32.9</td>
  </tr>
  <tr>
    <td>+Rerank</td>
    <td>82.9/72.0/32.9</td>
  </tr>
  <tr>
    <td>+IBUT</td>
    <td>84.3/73.6/32.8</td>
  </tr>
</table>

Table 11: General Performance of general performance on commonsense MT

aspects of ambiguity resolution and understanding distortion. This alignment session ensures a shared understanding of the evaluation dimensions and minimizes discrepancies in judgment. The evaluation proceeds as follows: First, the annotator carefully examines each sentence in the Baseline translation to identify and count cases with ambiguity errors, recording a total of 70 such sentences. Among these, the annotator further analyzes and filters those where the errors stem from understanding distortion, identifying a total of 28 sentences. Next, the annotator compares these erroneous cases against IBUT translations, counting instances where the IBUT translation successfully corrects the Baseline errors based on contextual understanding, totaling 25 sentences.

Additionally, to further ensure robustness in ambiguity resolution assessment, five domain experts each with a strong background in computational linguistics and translation studies evaluate each sample against the reference translation. They score ambiguity resolution effectiveness on a binary scale, awarding 1 point for successfully resolved ambiguities and 0 points for unresolved cases. The final annotation for each sample is determined based on majority agreement among the five experts. In cases where a clear majority is not reached, the median score is adopted to mitigate the impact of outlier ratings.

**Human Evaluation of Translation Quality.**
We conducted a human preference study on both the English-Chinese and Chinese-English test sets of the Cultural MT Datasets and the Commonsense MT Dataset. We invited one annotator to participate (a professional translator), and we randomly selected 100 translation results of the same source sentences generated by methods such as ChatGPT, Refine, MAPS, MAD, and IBUT. In terms of translation quality, the annotators compared the translation results of IBUT against other comparative methods. For the same source sentences, if IBUT’s translation quality is superior, it is marked as **IBUT Win**; if the translation qualities are comparable, it is marked as **Tie**; if the translation quality of other methods is better, it is marked as **IBUT Loss**. We conducted three rounds of revisions on all evaluation results to increase the fairness of the assessments as much as possible. For the content with Chinese ambiguity in the commonsense MT dataset, we ensured the correctness of the source side understanding by confirming it with classmates whose native language is Chinese.

**B.8 IBUT Demonstrates Generalizability on Low-Resource Languages**

To further explore whether the IBUT method can be effective in low-resource translation tasks using open-source models, we conducted experiments on the low-resource directions of WMT23\footnote{https://www.statmt.org/wmt22/translation-task.html}. The experimental results are shown in Table 12, demonstrating that our method significantly improves the performance of open-source models in low-resource translation, thereby further validating the generalizability of IBUT.
<table>
  <tr>
    <th>Metrics</th>
    <th>WMT22 Cs→Uk COMET ↑ /BLEURT ↑</th>
    <th>En→Hr COMET ↑ /BLEURT ↑</th>
  </tr>
  <tr>
    <td>Alpaca-7B</td>
    <td>74.1/52.4/8.31</td>
    <td>65.9/53.2/8.1</td>
  </tr>
  <tr>
    <td>+Sshot</td>
    <td>75.9/53.1/8.3</td>
    <td>67.9/53.6/8.3</td>
  </tr>
  <tr>
    <td>+MAPS</td>
    <td>76.3/53.7/9.2</td>
    <td>68.1/54.2/8.9</td>
  </tr>
  <tr>
    <td>+IBUT</td>
    <td><b>77.9/54.3/9.5</b></td>
    <td><b>69.2/55.1/9.0</b></td>
  </tr>
  <tr>
    <td>Vicuna-7B</td>
    <td>74.9/57.8/10.5</td>
    <td>69.3/57.7/9.9</td>
  </tr>
  <tr>
    <td>+Sshot</td>
    <td>76.3/58.3/10.9</td>
    <td>70.2/58.1/10.7</td>
  </tr>
  <tr>
    <td>+MAPS</td>
    <td>77.2/59.6/11.1</td>
    <td>71.1/58.8/11.6</td>
  </tr>
  <tr>
    <td>+IBUT</td>
    <td><b>78.3/60.7/11.5</b></td>
    <td><b>72.9/60.4/13.1</b></td>
  </tr>
</table>

Table 12: The experimental low-resource results of IBUT on open-source models. Alpaca-7B and Vicuna-7B mean to perform translation directly through Zero-Shot. The bold indicates the highest values that are statistically significant, with p-values less than 0.01 in the paired t-test against all compared methods.

B.9 Introduce the Full Names of Languages.

To better understand the experimental setup, we present the language codes and their corresponding full language names in Table 13.

<table>
  <tr>
    <th>Language Codes</th>
    <th>Full Name of Language Code</th>
  </tr>
  <tr>
    <td>En</td>
    <td>English</td>
  </tr>
  <tr>
    <td>JA</td>
    <td>Japanese</td>
  </tr>
  <tr>
    <td>Cs</td>
    <td>Czech</td>
  </tr>
  <tr>
    <td>Uk</td>
    <td>Ukrainian</td>
  </tr>
  <tr>
    <td>De</td>
    <td>German</td>
  </tr>
  <tr>
    <td>Hr</td>
    <td>Croatian</td>
  </tr>
  <tr>
    <td>Ru</td>
    <td>Russian</td>
  </tr>
  <tr>
    <td>Hi</td>
    <td>Hindi</td>
  </tr>
  <tr>
    <td>Ta</td>
    <td>Tamil</td>
  </tr>
  <tr>
    <td>Te</td>
    <td>Telugu</td>
  </tr>
  <tr>
    <td>Fr</td>
    <td>French</td>
  </tr>
  <tr>
    <td>Es</td>
    <td>Spain</td>
  </tr>
  <tr>
    <td>Sah</td>
    <td>Yakut</td>
  </tr>
  <tr>
    <td>Liv</td>
    <td>Livonian</td>
  </tr>
</table>

Table 13: The language names corresponding to the language codes.

B.10 Iterative Translation Refinement Case Studies

The table 14 illustrates how errors in source-language contextual understanding are rectified through the integration of target-language contextual understanding. Experimental results demonstrate that target-language contextual understanding provides valuable target-language insights, serving as supervisory signals to address previous misunderstandings. This iterative process refines bilingual contextual understanding, leading to enhanced translation performance.
<table>
  <tr>
    <th>Case</th>
    <th>Source Sentence</th>
    <th>Iteration 0 Source Understanding</th>
    <th>Translated Result</th>
    <th>Iteration 0 Target Understanding</th>
    <th>Verbal Source Feedback</th>
    <th>New Source Understanding</th>
    <th>New Translated Result</th>
    <th>Reference</th>
  </tr>
  <tr>
    <td>1</td>
    <td>“一箭雙鵰殺死一個敵人。”（Reimbursement is understood as taking something from a certain place.)</td>
    <td>—报酬理解为取出、实列。报酬理解为取物。（Reimbursement is understood as taking something from a certain place.)</td>
    <td>One bullet takes out one enemy.</td>
    <td>...In military or colloquial contexts, “one bullet” is used as a euphemism for eliminating, neutralizing or killing an enemy.</td>
    <td>“报酬”不能被笼统地理解为取物，应根据具体语境对词语进行解释。例如，“一箭双雕”中“一箭”含义为“射出的箭”，“双雕”则应被广泛理解为“射出的箭能够同时射中两只鸟”。在军事或非正式语境下，应将“一箭双雕”的扩展意义理解为“射出的箭能够同时射中两只鸟”，而非字面意义上的“射出的箭能够同时射中两只鸟”。（“Reimbursement” is used as a euphemism for eliminating or killing an enemy with a single bullet.)</td>
    <td>“报酬”使用作隐喻，表示“一箭双雕”中“一箭”应理解为“射出的箭”，“双雕”应理解为“射出的箭能够同时射中两只鸟”。（“Reimbursement” is used as a euphemism for eliminating or killing an enemy with a single bullet.)</td>
    <td>One bullet eliminates one enemy</td>
    <td>A bullet eliminates an enemy.</td>
  </tr>
  <tr>
    <td>2</td>
    <td>“慧眼识才善于发现人才。”（Citation refers to utilizing a certain resource or opportunity to gain benefits or achieve success.)</td>
    <td>慧眼表示利用某种资源或机会来获得好处或成功。（Citation refers to utilizing a certain resource or opportunity to gain benefits or achieve success.)</td>
    <td>He excels at tapping into undiscovered talents.</td>
    <td>“提引” means suggesting someone for a position, role or recognition.</td>
    <td>“提引”可以理解为提供支持或帮助。（“Citation” can be understood as providing support or assistance.)</td>
    <td>He excels at recommending undiscovered talents for positions.</td>
    <td>He is good at recommending undiscovered talent.</td>
  </tr>
  <tr>
    <td>3</td>
    <td>“我们决不让祖国的江山变色。”（“变色” refers more to “changing the status quo.”)</td>
    <td>“变色”更多是指“改变现状。”（“变色” refers more to “changing the status quo.”)</td>
    <td>We will never allow our country’s borders to change.</td>
    <td>“变色” symbolizes any form of alteration that could compromise the nation’s stability and governance.</td>
    <td>“变色”可以理解为任何可能影响国家稳定和治理的形式。（“变色” is understood as any form of alteration that could compromise the nation’s stability and governance.)</td>
    <td>We will never allow our nation’s condition to change for the worse.</td>
    <td>We will never let the land, mountains and rivers change in a wrong direction.</td>
  </tr>
</table>

Table 14: Examples Demonstrating IBUT’s Iterative refinement of Translation (Chinese to English) Based on Bilingual Supervision Signals. Gray text indicates English annotations for the Chinese.

Reviewer paper summaries: The paper introduces Iterative Bilingual Understanding Translation (IBUT), a technique to improve LLM-based machine translation. Unlike previous approaches that rely on a one-time interpretation of the source text, IBUT repeatedly refines both the source and target “contextual understanding.” It uses bilingual feedback (checking alignment between source and target interpretations) to fix misunderstandings before finalizing the translation. Through experiments on WMT22/23, commonsense, and cultural translation tasks, IBUT outperforms baselines in both automatic metrics and human evaluations.
Summary of strengths: 1. The idea is simple and intuitive.  IBUT aligns and refines both source and target interpretations, which effectively fixes errors that appear in single-language explanations.

2. Experiments show that IBUT improves over standard zero-shot, few-shot, and existing multi-step prompting methods on multiple LLMs such as ChatGPT, GPT-4, Alpaca, Vicuna, and Qwen.
Summary of weaknesses: 1. Iterative prompting uses more tokens and time, which may be expensive.

2. If the LLM starts with incorrect background knowledge or logic, refining might not fully fix the issue.

3. IBUT depends on carefully designed prompts, which might require tuning across tasks.
Comments, suggestions and typos: See weakness.
Overall assessment scores: 2.5