Efficiently Exploring Large Language Models for Document-Level Machine Translation with In-context Learning (2024)

Abstract: Large language models (LLMs) exhibit outstanding performance in machine translation via in-context learning. In contrast to sentencelevel translation, document-level translation (DOCMT) by LLMs based on in-context learning faces two major challenges: (1) document translations generated by LLMs are often incoherent; (2) the length of demonstration for in-context learning is usually limited. To address these issues, we propose a ContextAware Prompting method (CAP), which enables LLMs to generate more accurate, cohesive, and coherent translations via in-context learning. CAP takes into account multi-level attention, selects the most relevant sentences to the current one as context, and then generates a summary from these collected sentences. Subsequently, sentences most similar to the summary are retrieved from the datastore as demonstrations, which effectively guide LLMs in generating cohesive and coherent translations. We conduct extensive experiments across various DOCMT tasks, and the results demonstrate the effectiveness of our approach, particularly in zero pronoun translation (ZPT) and literary translation tasks.

heading: 1 Introduction
text: With the increasing scale of parameters and training corpus, large language models (LLMs) have gained remarkable abilities to handle a variety of tasks via in-context learning (Li et al., 2023a; Wang et al., 2023b), which allows language models to perform tasks with a few given demonstrations and humanwritten prompts as context. One particular area where LLMs have shown outstanding potential is machine translation (MT) (Moslem et al., 2023; Zhang et al., 2023a,b; Gao et al., 2024). However, most existing studies (Zhu et al., 2024) have focused on sentence-level translation, which makes
* Equal contribution † Corresponding author
LLM-based document-level machine translation underexplored (Wang et al., 2023a; Li et al., 2024). To bridge this gap, we adapt LLMs to documentlevel machine translation (DOCMT) in this paper.
As a complex task different from sentence-level translation, one major challenge of DOCMT with LLMs is that the length of demonstrations for in-context learning is limited. For sentence-level MT, Agrawal et al. (2023) and Zhang et al. (2023a) show that using 32 or more randomly sampled bilingual parallel sentence pairs as prompts can effectively enhance the translation abilities of LLMs. However, for DOCMT, the length of the text segments to be translated or to be used as demonstrations inherently increase (Zeng et al., 2024). Consequently, the limited input window of large language models poses a significant obstacle (Pawar et al., 2024; Tang et al., 2023). Unlike sentence-level translation, the constraints of DOCMT hinder the seamless integration of such demonstrations. Moreover, the lengthier nature of document-level inputs exacerbates issues related to inference speed. As the length of the demonstrations increases, the computational resources required for processing also escalate, potentially impeding real-time translation capabilities.
Previous efforts to DOCMT with LLMs, e.g., the method presented in (Wu et al., 2024), involve fine-tuning LLMs using two strategies: ParameterEfficient Fine-Tuning and Full Parameter FineTuning. Such a framework may not be able to sufficiently overcome the limitation of the length of demonstration for in-context learning in LLMs. And it also reveals a significant issue: contextindependent translations, which can result in translations that lack coherence and disambiguation (Zhang et al., 2022a; Macé and Servan, 2019).
To address these issues, we propose a novel method called CAP, which stands for ContextAware Prompting. It consists of three essential steps. Firstly, by combining token- and sentence-
10885
level attention scores, we obtain sentences with the highest sentence-level attention scores as context for each sentence. This step facilitates LLMs in generating coherent translations. Secondly, to address the limitation of the length of demonstration for in-context learning in LLMs, we summarize the context and retrieve parallel sentences from the datastore, which are most similar to the generated summary. Finally, we use the retrieved example pairs as demonstrations, prompting LLMs to produce document translations.
In summary, the contributions of this work are as follows:
• We investigate document-level translation via LLMs. Specially, we explore linguistic phenomena in document-level translation with LLMs, such as ZPT.
• We propose a novel method, CAP, to enable LLMs to perform accurate and coherent translation via in-context learning. Our method addresses the issue of translation coherence deficiency by employing a dynamic context window and resolves the limitation concerning the length of demonstration for in-context learning by summarizing the context.
• To demonstrate the effectiveness of our method, we conduct experiments on various DOCMT tasks. Experimental results show that our method significantly outperforms the baselines, especially for the ZPT task.

heading: 2 Related Work
text: Document-Level Machine Translation In recent years, numerous approaches have been proposed for DOCMT (Maruf et al., 2022; Lei et al., 2022; Zhang et al., 2022b; Tan et al., 2022; Zhang et al., 2020b). These studies could be further roughly categorized into two groups. Studies of the first group extend the source-side context from a sentence to the local context of few sentences (Tan et al., 2021; Lyu et al., 2021; Xu et al., 2021). Studies of the second group have focused on document-to-document (Doc2Doc) translation. The initial exploration of Doc2Doc NMT has involved concatenating multiple sentences into a single unit (Ma et al., 2020; Zhang et al., 2020a; Tan et al., 2019). Recent studies have successfully trained vanilla Transformer models for Doc2Doc translation by leveraging either large augmented datasets (Lupo et al., 2022; Sun et al., 2022) or pre-trained models (Yang et al.,
2021). Additionally, some latest work has achieved the truly Doc2Doc translation by enhancing the attention mechanism in the Transformer model (Bao et al., 2021; Li et al., 2023b).
Translation Oriented LLMs LLMs have demonstrated remarkable proficiency across a wide range of NLP tasks (Scao et al., 2022; Touvron et al., 2023; Sun and Xiong, 2022; Min et al., 2024; Wei et al., 2022). Recent research has shown that in-context learning can significantly enhance their performance when following general language instructions. Specifically, there is a growing body of work exploring the translation capabilities of LLMs (Lu et al., 2023; Zhang et al., 2023a; Hendy et al., 2023). However, it is important to note that these efforts have primarily focused on sentencelevel machine translation and have not delved into DOCMT (Wu et al., 2024). Latest noteworthy studies in DOCMT are conducted by Wu et al. (2024) and Wang et al. (2023a). Wang et al. (2023a) investigate the document-level translation capabilities of GPT-3.5-TURBO, making it the most closely related work to ours that explores the DOCMT ability of LLMs. Wu et al. (2024) explore fine-tuning LLMs using two strategies: Parameter-Efficient Fine-Tuning and Full Parameter Fine-Tuning, to enhance the DOCMT ability of LLMs. Such a framework may not be able to sufficiently overcome the limitation of the length of demonstration for incontext learning in LLMs. We observe that the existing translation-oriented LLMs tend to struggle with effectively translating sentences in the middle and ending of a document. Existing methods that explore DOCMT with LLMs are rarely considering the issues of (1) the length limitation of demonstrations for in-context learning and (2) maintaining coherence and resolving ambiguities. In this work, we mainly concentrate on mitigating the length limitation of in-context learning for DOCMT and addressing the issues of incoherence and ambiguity in DOCMT with LLMs.

heading: 3 CAP: Context-Aware Prompting
text: In this section, to address the challenges associated with utilizing LLMs for DOCMT via in-context learning, we propose a Context-Aware Prompting method.
As shown in Figure 1, our approach consists of three steps. Firstly, we take into consideration multi-level attention, subsequently opting for the sentences with the highest sentence-level attention
scores as the context. Secondly, we input the context into an LLM to obtain its summary, then retrieve K example pairs from the datastore, which are most similar to the summary. Thirdly, we use the retrieved example pairs as demonstrations in the prompt, guiding the model to generate accurate and coherent translations through a few-shot approach.

heading: 3.1 Dynamic Context Window
text: In the context of document-level translation for LLMs, issues of incoherence and disambiguation may arise, leading to context-independent translations. In order to address this problem, we propose a Dynamic Context Window as a solution. By extracting the most attended context surrounding the current sentence, it facilitates the model in generating accurate and coherent translation outputs.
Initially, we meticulously devise a method for computing sentence-level attention. Specifically, as demonstrated in Eq. (1), we extract the attention weights from the final layer of the LLM, followed by averaging across all attention heads to obtain the token-token attention scores. Next, as illustrated in
Eq. (2), the maximum value of token-token attention scores between tokens in the current sentence and tokens in all other sentences is computed as the token-sentence attention score. Ultimately, as shown in Eq. (3), the average of attention scores between all tokens in the current sentence and other sentences is calculated to derive the sentence-level attention score.
attni,j = 1
H
H∑
h=1
attnhi,j (1)
TSi,S = max(attni,k), k ∈ tokens(S) (2)
SSI,S = avg(TSi,S), i ∈ tokens(I) (3)
where TSi,S is the attention score of Token i and Sentence S, SSI,S is the attention score between Sentence I and Sentence S. attnhi,j denotes the attention score of the h-th head in the final transformer block, calculated between the i-th token and the j-th token in the current document. H is the number of attention heads in each transformer block.
After obtaining the sentence-level attention scores, we extract the top N sentences with the highest sentence-level attention scores as dynamic context, while maintaining their original sequential positions within the document.

heading: 3.2 Example-Specific In-Context Demonstrations Selection
text: Unlike sentence-level translation, the challenge in utilizing LLMs for DOCMT arises from the constrained context length, as the length of the text segments to be translated or used as demonstrations inherently increases. To effectively address these challenges, it is necessary to further process the context and extract as much relevant information as possible from the preceding sentences. We then input this extracted context into an LLM to generate a concise summary that encapsulates key topics and information.
Subsequently, as shown in Eq. (4), we utilize sentence-transformers (Reimers and Gurevych, 2020) to retrieve carefully designed example pairs from a specified datastore. These examples are selected based on their cosine similarity with the embedding of the summary sentences, ensuring that the retrieved examples exhibit semantic similarity to the context and content of the document.
score(S,X) = cos(S,X) (4)
where S, and X are the summary we obtained at the preceding step and the example sentence in datastore respectively. S and X are the embeddings of S and X respectively.
By employing these selected examples as fewshot demonstrations, we provide the LLM with a more precise understanding of the document. This approach not only enhances the model’s accuracy in utilizing in-context learning for translation but also contributes to enhancing the robustness of document translation tasks.

heading: 3.3 Inference
text: We employ a prompt template, which is identical to that utilized in (He et al., 2023). In this approach, we integrate the retrieved example pairs from the preceding step seamlessly into the prompt template, which will then be input into the target LLM. This strategic integration serves to guide the LLMs in generating translations that exhibit enhanced cohesion and coherence.
Following the translation process, we employ a diverse set of evaluation metrics to thoroughly
assess the quality of the translations generated by the LLM. This multifaceted evaluation allows us to gain comprehensive insights into the effectiveness and performance of the language models in producing accurate and appropriate translations.

heading: 4 Experiments
text: In this section, we conducted extensive experiments on various DOCMT tasks to examine the effectiveness of the proposed method.

heading: 4.1 Setup
text: Datasets: We used OPUS-1001 and newscommentary-v15 as the datastore to retrieve the in-context demonstrations on four language pairs. To assess the effectiveness of our proposed method on LLMs, we employed WMT22 newstest as the test set. Following (Agrawal et al., 2023), we normalized punctuation using Moses2 and removed sentence pairs with a source/target length ratio exceeding 1.5. To assess the accuracy of our method in ZPT, we utilized the GuoFeng (Xu et al., 2022) dataset, which covers five domains: movie subtitle, Q&A forum, government news, web fiction, and personal profile. To validate the capability of our approach in guiding the model to generate more cohesive and more coherent translations, we employed the WMT23 Literary Translation dataset,3 which comprises two test sets: an in-domain test set and an out-of-domain test set. Additionally, we used its training set to retrieve in-context demonstrations.
Models: To compare the translation performance of LLMs using our method, we compared against the following three encoder-decoder based strong NMT models as baselines.
• MarianMT: A framework for translation models, using the same models as BART (Lewis et al., 2020). In our experiment, each language pair is associated with a distinct pre-trained model.
• M2M100 (Fan et al., 2021): It is a multilingual encoder-decoder model, which is pretrained on 100 languages and is capable of translating 9,900 directions.
1https://opus.nlpl.eu/opus-100.php 2http://www2.statmt.org/moses/ 3https://www2.statmt.org/wmt23/literary-translation-
task.html
• NLLB (Costa-jussà et al., 2022): It is a neural multilingual machine translation model, which supports translation between any pair of 200 languages. There are numerous variants of it, and the one utilized in our experiments is NLLB-3.3B.
To evaluate the translation abilities of LLMs, we utilized LLMs that are tuned with SFT, as they can follow instructions to generate translations. Hence, we chose the following two LLMs, which support both Chinese and English and have undergone instruction tuning:
• Baichuan (Yang et al., 2023): It is trained on a high-quality corpus with 2.6 trillion tokens and has achieved the best performance in authoritative Chinese and English benchmarks of the same size.
• Qwen (Bai et al., 2023): It is pretrained on over 3 trillion tokens, including Chinese, English, multilingual texts, code, and mathematics, covering general and professional fields.
Prompt Selection Strategies: To assess the effectiveness of the proposed method, we compared our method against five prompt selection strategies:
• Zero-shot: Zero-shot is a powerful baseline. Previous studies (Radford et al., 2019) show that LLMs have excellent zero-shot abilities in many NLP tasks, including reading comprehension, translation, and summarization.
• Random: This strategy randomly selects K sentence pairs from the datastore as demonstrations.
• BM25: It calculates the similarity between translated sentences and sentences in the corpus with BM25, and then selects top K sentences with the highest similarity as demonstrations.
• Similar: This method uses sentence embedding similarity-based retrieval to select K demonstrations (Moslem et al., 2023). In our work, we used sentence-transformers (Reimers and Gurevych, 2020) to implement this method.
• Precedent: This approach guides large language models in generating translations by utilizing the preceding K sentences of the current sentence within a document, along with those translations generated in previous steps as demonstrations.
For prompt selection experiments, we employed models from four distinct sizes of the Qwen LLMs. Meanwhile, for a trade-off between performance and cost, we set K to 3.
Metrics: We employed a variety of metrics to assess the quality of document-level translation generated by LLMs, including document-level BLEU(d-BLEU), and ChrF2 using SacreBLEU for document-level evaluation. For the ZPT task, we employed accuracy to evaluate the precision of zero pronoun translation. Additionally, we have also evaluated document-level translation quality with Blonde (Jiang et al., 2022), which is an automatic evaluation metric for document-level machine translation. More experimental results can be found in Appendix A.
All experiments were performed on 4 NVIDIA A100 GPUs.

heading: 4.2 Comparing Different Prompt Selection Strategies
text: Experimental results are presented in Table 1. We find that the prompt guidance constructed using our approach yields the highest translation quality across LLMs of different sizes, except for Qwen1.8B-Chat. We attribute this to the fact that small
models often struggle to extract meaningful context.
Additionally, in the same few-shot experimental setup, the translation quality of selecting demonstrations based on sentence embedding similarity surpasses that of the random approach. This is also corroborated in (Agrawal et al., 2023). Additionally, we observe experimental results consistent with (Radford et al., 2019), indicating that the quality of generative outputs guided in a “Zero-shot” method is generally superior to those generated through “Random” selection or “Similarity-based” selection in a few-shot method.
We observe that “Precedent” serves as a strong baseline and exhibits a similar performance trend to our method: when the model size is too small, it struggles to produce high-quality translations. As noted by Agrawal et al. (2023), the quality of demonstrations in prompt is crucial for translation performance. Therefore, we attribute this to the cumulative effect of low-quality translations produced by small models, which leads to progressively worse translations.
Experiment results suggest that as the model size increases, our method continues to effectively guide LLMs in generating high-quality translations. This also demonstrates the universality of our approach.

heading: 4.3 Comparing with Traditional NMT Models
text: We compared our method against traditional NMT models. As shown in Table 2, the results show that when the model size is large enough, the translation abilities of LLMs comprehensively exceed those of dedicated translation models. This is observed except in the en-de language pair, where we attribute the discrepancy to the insufficient understanding of the German language by the Baichuan and Qwen models, primarily trained on Chinese and English corpora. Specifically, in the en-zh translation direction where Baichuan and Qwen excel, the translation results of LLMs significantly surpass those of specialized models.
Moreover, our experiments reveal interesting insights into the impact of model size on translation performance. As expected, larger LLMs, such as Qwen-72B-Chat, consistently outperforms smaller counterparts across various language pairs and evaluation metrics. This trend demonstrates the importance of model scale in harnessing the full potential of LLMs for translation tasks.
However, the most striking finding is the competitive performance of small LLMs, particularly those with parameters in the 7B to 14B range. Despite their smaller size compared to their larger counterparts, these models exhibit translation capabilities on par with or even outperforming dedicated translation models. This suggests that with an appropriate prompt strategy, small LLMs can
offer a cost-effective solution for translation tasks. Our method has improved document translation performance on different LLMs, further verifying the robustness of our method. This underscores the generality and efficiency of our approach in leveraging LLMs for translation tasks across diverse domains and languages.

heading: 4.4 Analyzing the Effectiveness on ZPT
text: The increasing interest among researchers in ZPT can be attributed to its increasing complexity and significance within the field of language translation. ZPT involves the intricate task of proficiently addressing the omission of zero pronouns in the source language text when it undergoes translation into the target language. The primary aim is to ensure the meticulous restoration or expression of these omitted pronouns in the translated output, thereby upholding linguistic accuracy, fluency, and contextual consistency.
To assess the effectiveness of our proposed method in the context of the ZPT task, a series of experiments were conducted on the comprehensive GuoFeng dataset. Experimental results are displayed in Table 3. Remarkably, across models of varying sizes, a discernible enhancement in ZPT accuracy is consistently observed when compared to the baseline performance. It is noteworthy, however, that the relationship between the model size and the accuracy of ZPT is not characterized by continuous improvement. This phenomenon can be ascribed to the realization that a high overall translation quality of the model does not necessarily translate into a proportionate increase in accuracy specifically related to ZPT. Additionally, we found that the “Precedent” method outperforms similarity-based prompt selection strategies, indicating the importance of contextual information in the ZPT task. Our approach, which considers dynamic contextual information, effectively captures relevant context while filtering out irrelevant
information. As a result, our method demonstrates superior performance in the ZPT task.
Hence, it becomes evident that a mere augmentation of the model size does not suffice for the improvement of ZPT accuracy. Consequently, the exploration and adoption of more effective methods become imperative in the pursuit of advancing the state-of-the-art in this domain.

heading: 4.5 Analyzing the Effectiveness on Literary Translation
text: The significant challenges for translating literary works include entity consistency, anaphora resolution, and lexical choice (Matusov, 2019). Literary works exhibit longer contextual spans compared to texts in other domains, such as news articles. Therefore, LLMs must possess the ability to model distant contexts in order to learn translation consistency and appropriate lexical choices. Our approach, by focusing on the context relevant to the current sentence and subsequently summarizing the context, is able to encapsulate more contextual information within a limited-length prompt. To validate whether our method can guide the model to generate more consistent translations, we conducted experiments on the WMT23 Literary Translation dataset.
The experimental results are presented in Table 4. We observe consistent improvements of our approach over the baseline method, both on indomain and out-of-domain test sets. Consistent with the above experimental results, the strategy of simply using sentence embedding similarity to select demonstrations often performs even worse than random selection. This implies that a straightforward prompt selection strategy based on similarity is not suitable for challenging tasks like literary translation. However, our method is able to effectively extract contextual information, thereby guiding LLMs to generate higher-quality translations, specifically achieving better entity consistency.

heading: 5 Analysis
text: In this section, we conducted experiments to further analyze the effectiveness of our approach.

heading: 5.1 Ablation Study
text: To verify the effectiveness of various factors on our method, we further compared our method with the following variants and present the results in Table 5:
(1) w/o DCW. In this variant, we directly used a Fixed Context Window to select the context. Specifically, we utilized the preceding and subsequent two sentences of the current sentence as the context. As reported in Table 5, this variant dramatically decreases performance on two language pair test sets. It reveals the significance of dynamically selecting context through attention scores.
(2) max ⇒ avg. In this variant, we replaced the max function with the average function in Eq. (2). From Table 5, we can observe that this variant performs worse than our method. It demonstrates the effectiveness of our approach in computing the sentence-level attention score.

heading: 5.2 Visualization
text: In order to explain why our approach successfully translates omitted zero pronouns in Chinese, we extracted the attention weights from the final layer of the Qwen-14B-Chat model and visualized it using
appropriate tools.4 As illustrated in Figure 2, we observe that when the model attempts to translate the zero pronoun “He”, it exhibits greater attention to the subject in the context and the associated information about the subject. The visualized results reflect the effectiveness of our approach, showcasing its ability to capture relevant contextual information and guide the model in generating accurate and coherent translations. This indicates that our approach involves more than merely translating words at the surface level; rather, it entails translating based on a profound comprehension of sentence structure and context, thereby offering guidance and support for the model to generate more precise translations, thereby demonstrating the effectiveness of our method.

heading: 5.3 Case Study
text: In order to assess the precision of ZPT across various methods in the ZPT task, a comprehensive case study was undertaken. As illustrated in Table 6, it is observed that the “Zero-shot”, “Random”, and “Similar” methods exhibit inaccuracies in translating omitted zero pronouns in Chinese. Conversely, our approach demonstrates proficiency by accurately translating the omitted zero pronouns as “His”. This success can be attributed to the method’s adeptness at extracting relevant contextual information, enabling the model to deduce the omitted zero pronouns effectively.
In summary, our method stands out in its capacity to extract valuable contextual information, guiding the model towards generating translations that are not only more accurate but also more cohesive and coherent. Further examples and elaborate experimental results are available in Appendix B for reference.
4https://github.com/alan-cooney/CircuitsVis

heading: 6 Conclusion
text: In this paper, we have presented a novel approach that guides the model to generate more accurate, cohesive, and coherent document-level translations via in-context learning. By selecting relevant contexts using our carefully designed method, we can extract topic information from the context summaries. These summaries serve as a guide for selecting the most beneficial examples from the datastore to enhance the in-context learning procedure of LLMs, hence mitigating the length limitation issue of in-context learning for DOCMT. Experimental results on various benchmarks demonstrate the effectiveness of our approach, particularly in ZPT and literary translation tasks, where our method outperforms the baseline significantly.
Limitations
By selecting better contexts and demonstrations, we have effectively increased the ZPT accuracy and BLEU score of DOCMT with LLMs via in-context learning. We have validated the effectiveness of our method using ZPT accuracy and BLEU score. However, beyond the general BLEU and Blonde metric, DOCMT encompasses challenges such as coreference consistency. We also need to make improvements to address these challenges. Moreover, we should also investigate the performance on more language pairs, which we consider as future work.
Yet another limitation in our work is the increased computational demand. Since our approach necessitates the computation of dynamic context and summarization of context, the computational load is slightly higher compared to the baseline. We intend to explore more efficient document-
level machine translation methods in future work.

heading: Acknowledgements
text: The present research was supported by the National Key Research and Development Program of China (Grant No. 2023YFE0116400). We would like to thank the anonymous reviewers for their insightful comments.

heading: In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume
text: 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 4176–4192. Association for Computational Linguistics.
Biao Zhang, Barry Haddow, and Alexandra Birch. 2023a. Prompting large language model for machine translation: A case study. In International Conference on Machine Learning, ICML 2023, 23- 29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 41092–41110. PMLR.
Linlin Zhang, Zhirui Zhang, Boxing Chen, Weihua Luo, and Luo Si. 2022b. Context-adaptive document-level neural machine translation. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2022, Virtual and Singapore, 23-27 May 2022, pages 6232–6236. IEEE.
Pei Zhang, Boxing Chen, Niyu Ge, and Kai Fan. 2020a. Long-short term masking transformer: A simple but effective baseline for document-level neural machine translation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 1081–1087. Association for Computational Linguistics.
Pei Zhang, Xu Zhang, Wei Chen, Jian Yu, Yanfeng Wang, and Deyi Xiong. 2020b. Learning contextualized sentence representations for document-level neural machine translation. In ECAI 2020 - 24th European Conference on Artificial Intelligence, 29 August-8 September 2020, Santiago de Compostela, Spain, August 29 - September 8, 2020 - Including 10th Conference on Prestigious Applications of Artificial Intelligence (PAIS 2020), volume 325 of Frontiers in Artificial Intelligence and Applications, pages 2298–2305. IOS Press.
Shaolei Zhang, Qingkai Fang, Zhuocheng Zhang, Zhengrui Ma, Yan Zhou, Langlin Huang, Mengyu Bu, Shangtong Gui, Yunji Chen, Xilin Chen, and Yang Feng. 2023b. Bayling: Bridging cross-lingual alignment and instruction following through interactive translation for large language models. CoRR, abs/2306.10968.
Shaolin Zhu, Menglong Cui, and Deyi Xiong. 2024. Towards robust in-context learning for machine translation with large language models. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, LREC/COLING 2024, 20-25 May, 2024, Torino, Italy, pages 16619–16629. ELRA and ICCL.
A Additional Experimental Results
In Section 4.2, we have employed only d-BLEU and ChrF2 to evaluate translation quality. We use Blonde scores here to further measure translation quality.
Specifically, we evaluated the translation performance of Qwen-7B-Chat and Qwen-14B-Chat on the WMT22 newtest test set using the Blonde metric under different prompt strategies. Experimental results, shown in Table 8, indicate that our proposed method significantly outperforms the baselines, demonstrating its effectiveness.

heading: B Case Study and Visualization Analysis
text: In this section, we present more examples to further validate our approach with case study and attention visualization.
B.1 Case Study As shown in Table 7, we provide another example where in the original Chinese text, the subject “你” (you) is omitted due to language convention. In English, it should be correctly restored and translated as “you”. We find that the “Zero-shot”, “Random”, and “Similar” methods all incorrectly recover zero pronouns, whereas our method successfully recovers the correct zero pronoun “you”. This demonstrates that our method outperforms better than baselines in ZPT.
B.2 Visualization As illustrated in Figure 3, we observe that when the model is tasked with translating “He”, it predominantly focuses on the subject of the context and its associated information, encompassing entities such as “白崇恩” and the designation “教 授”. This experimental result illustrates how our approach guides LLMs to generate more accurate translations via in-context learning, thus proving the effectiveness of our method.

heading: Future Work
text: Moreover, we should also investigate the performance on more language pairs, which we consider as future work. Yet another limitation in our work is the increased computational demand. Since our approach necessitates the computation of dynamic context and summarization of context, the computational load is slightly higher compared to the baseline. We intend to explore more efficient document-
level machine translation methods in future work