DelTA: An Online Document-Level Translation Agent Based on Multi-Level Memory
Authors: Yutong Wang
Jiali Zeng
Xuebo Liu
Derek F. Wong
Fandong Meng
Jie Zhou
Min Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: Document-Level Translation
Large Language Models
Autonomous Agents
Natural Language Processing
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We develop a document-level translation agent based on multi-level memory to address the challenges associated with the DocMT task.

Abstract: Large language models (LLMs) have achieved reasonable quality improvements in machine translation (MT).
However, most current research on MT-LLMs still faces significant challenges in maintaining translation consistency and accuracy when processing entire documents.
In this paper, we introduce DelTA, a Document-levEL Translation Agent designed to overcome these limitations.
DelTA features a multi-level memory structure that stores information across various granularities and spans, including Proper Noun Records, Bilingual Summary, Long-Term Memory, and Short-Term Memory, which are continuously retrieved and updated by auxiliary LLM-based components.
Experimental results indicate that DelTA significantly outperforms strong baselines in terms of translation consistency and quality across four open/closed-source LLMs and two representative document translation datasets, achieving an increase in consistency scores by up to 4.58 percentage points and in COMET scores by up to 3.16 points on average.
DelTA employs a sentence-by-sentence translation strategy, ensuring no sentence omissions and offering a memory-efficient solution compared to the mainstream method.
Furthermore, DelTA improves pronoun and context-dependent translation accuracy, and the summary component of the agent also shows promise as a tool for query-based summarization tasks.
The code and data of our approach are released at https://github.com/YutongWang1216/DocMTAgent.

Published as a conference paper at ICLR 2025
DELTA: AN ONLINE DOCUMENT-LEVEL TRANSLA- TION AGENT BASED ON BULTI-LEVEL MEMORY
Yutong Wang Jiali Zeng" Xuebo Liu't Derek F. Wong" Fandong Meng² Jie Zhou² Min Zhang' "Inttittee of Computing and Intelligence, Harbin Institute of Technology, Shenzhen, China PPattern Recognition Center, WeChat AI, Tencent Inc, China
"NLP'CT Lab, Department of Computer and Information Science, University of Macau wangyutongéstt. hit, edu, cn, { {liuxuebo, zhangmin2821 Jehit edu. cn [emonzenn, Fandongmeng, wi thtoooohooo com, derekfw@um. edu, com
ABSTRACT
Large language models (LLMs) have achieved reasonable quality improvements in machine translation (MT). However, most current research on MT-LLMs still faces significant challenges in maintaining translation consistency and ac- curacy when processing entire documents. In this paper, we introduce DELTA, a Document-lev Translation Agent designed to overcome these limitations. DELTA features multi-level memory structure that stores information across var- ious gramularities and spans, including Proper Noun Records, Bilingual Summary, Long-Term Memory, and Short- Memory, which are continuously retrieved and updated by auxiliary LLM-based components. Experimental results indicate that DELTA significantly outperforms strong baselines in terms of translation con- sistency and quality across four open/closed-ee spercclosed-souree LLMs and two representative document translation datasets, achieving an increase in consistency scores by up to 4.58 percentage points and in COMET scores by up to 3.16 points on average. DELTA employs a sentence-by bereence-----eeence translation strategy, ensuring no sentence omissions and offering memory-efficien solution compared to the mainstream method. Furthermore, DELTA improves pronoun and context-dependent transla- tion accuracy, and the summary component of the agent also shows promise as a tool for query-based summarization tasks. The code and data of our approach are released /t/https://gitiiie wotp://///o/mottttgggnnon 2216///nngg11/////// 6/DocMT/ggett
INTRODUCTION
Large language models (LLMs) such as GPT-4 (OpenAl, 2023) have recently demonstrated rea- sonable performance on the machine translation (MT) task within the natural language processing
domain (Garcia & Firat 2022; Hendy et al., 2023 Zhang et al. 2023 Siu 2023 Jiao et al. 2023).
Numerous studies have been carried out to further unleash LLMs potential for MT (Ghazvininejad
ctt al., 2023 Peng et al. 2023 Zeng et al. 2024, He et al., 2024, Sun et al., 2024] Wang et al.
2024c Liu al. 2024). However, the majority of these researches mainly focus sentence-leve translation, operating under the strong assumption that source sentences are independent of one an- other. This isolated approach may fail to model the discourse structure and overlook the coherence
in continuous document texts (Scarton & Specia 2015 Bawden et al. 20188).
Document-level machine translation (DocMT) systems have been receiving growing focus in recent years, which involves the whole document or some part of it to capture more context information to guide the translation process (Kim et al. 2019 Maruf et al. 2021). Researchers find that modeling discourse phenomena (Bawden et al. 2018) while translating the whole document helps increase the coherence and consistency in the generated translation (Maruf Haffari] 2018, Wang et al. 2017 Zhang et al. 2018) TTn et al., 2019). Recently, few studies have proposed to introduce LLMs to the DocMT task, utilizing their inherent context information modeling and long text processing ca- pabilities (Wang et al. 2223b;; Wu & Hu 2023 Wu et al. 2024a). However, existing DocMT-LLMs
Work was done when Yutong Wang was interning at Pattern Recognition Center, WeChat AI, Tencent Inc.
Xucho Liu the corresponding author.

---

Published as a conference paper at ICLR 2025
still suffer from critical issues such occasional content omissions and terminology translation in- consistency (Karpinska & lyyer 2023). These issues seriously affect the reliability of the developed system, especially when accurate document translations are required.
LLM-based autonomous agents equipped with specially designed memory components can effi- ciently store and retrieve key information embedded in the environment. These data assist the inference process of LLMs, facilitating the handling of complex tasks and environments through self-directed planning and actions (Wang et al., 2024a; 2023a Park al. 2023 Lee et al. 2024). Inspired by this, we propose DELTA, an online Document-lev EL Translation Agent based on multi- level memory components. Specifically, we store information four memory components: Proper Noun Records, Bilingual Summary, Long- Term Memory, and Short-term Memory, and utilize LLMs to update and retrieve them. Proper Noun Records maintain a repository of previously encountered proper nouns and their initial translations within the document, ensuring consistency by reusing the same translation for each subsequent occurrence of the same proper noun. The Bilingual Summary contains summaries of both the source and target texts, capturing the core meanings and genre char- acteristics the documents to enhance translation coherence. Long-Tern Memory and Short-Term Memory store contextual sentences over broader and narrower scopes, respectively. Long-Tern Memory is accessed by LLMs to retrieve sentences most relevant to the current source sentence, while Short-Term Memory provides instant context support the translation process. During trans- lation, sentence pairs are drawn from the Long-Term and Short-Term memory as the exemplars for few-shot learning demonstration, and the proper noun translation records and bilingual summaries are also integrated into the prompt for DocMT-LLMs as auxiliary information.
Experimental results indicate that DELTA achieves improvements both translation consistency and quality. For translation consistency, DELTA achieves an average improvement of 4.36 percent- age points across four translation directions from English and 4.58 percentage points across four directions into English. In terms of translation quality, DELTA yields an average improvement of 3. 3.14 COMET points for four translation directions from English and 3.16 COMET points for four directions into English. Moreover, DELTA translate documents in sentence-by seitence-by-sentncce manner (following an online approach) to avoid content omissions, ensuring sentence-level alignment of target documents with source documents. This manner also prevents memory bloat caused by data accumulation, making it more suitable for practical application scenarios.
Our main contributions are summarized as follows:
We develop DELTA,, an online DocMT agent employing a multi-level memory structure, which stores information across different granularities and spans.
We demonstrate that DELTA substantially improves the consistency and quality of doc- ument translations. Additionally, the summary component of DELTA can function as an independent tool for query-based summarization tasks.
We certificate that the centence-wiss translation approach employed by DELTA incurs lower memory cost compared to existing document translation methods.
We observe that DELTA is particularly effective in maintaining translation consistency over expended spans. Moreover, it enhances the pronoun translation accuracy in the document.
2 RELATED WORK
Document-Level Machine Translation In recent years, studies on DocMT have achieved rich results (Kim et al. 2019 Maruf et al. 2021: Fernandes et al., (021). These studies can be separated into two categories. Studies of the first group employ a document-to-sentenn
(Doe2Sent) approach, where the source-side context sentences are encoded to generate the current target sentence (Wang ctt al., 2017, Tan et al. 2021 Lyu et al. 202I). However, these approaches suffer from limitations caused by separated encoding modules of the current sentences and their context (Sun et al. 2022
Bao et al. 2021). as well as the failure to utilize target-side context (Li et al.] 2023b). Studies of the second group employ document-tee to-euummmnntttt (Doc2Doc) approach, where the translation unit is extended from a single sentence to multiple sentences (Zhang et al. 2020 Liu et al. 2020 Lupo
et al. 2022
Bao al. 2021 [i et al. 2023b).
2

---

Published as a conference paper at ICLR 2025
Window
LTCR-1
LTCR-1;
#Missing Sents
sCOMET
dCOMET
75.09
88.24
84.04
6.62
5
80.49
88.15
84.30
6.70
10
79.65
90.81
84.27
30
83.08
95.83
50
86.94
95.90
83.70
6.66
Table Translation results with different translation window sizes. "#Missing Sents" represents the number of missing target sentences in the translated document.
Autonomous Agents LLM--ased autonomous agents have recently achieved remarkable perfor- mance in various NLP tasks. Park et al. (2023); Wang et al. (2023a); [eee et al.] (2024) deal with long-context understanding and processing tasks by introducing carefully designed memory and re- trieval workflows. Xu et al. (2024); Wang et al. (2024c); [Feng et al. (2024) prompt L.LSs to evaluate their own outputs and conduct refinement accordingly to improve the quality the outputs. Li et al. (023a): Liang et al. (2024); Li ct al. (2024); Wu et al. (2024b) enhance the performance of LLMs on specific tasks by multi-agent interaction.
Our method in this paper represents Doc2Sent approach implemented through LLM-based automatic agent. Instead of simply encoding source-side context to generate target sentences, our method directs LLMs to retrieve key information across varying granularities and spans, and store this data in memory components. During document translation, relevant information is incorporated into the prompts for DocMT-ILLMs to assist the translation process.
3 MOTIVATION
3.1 MAIN CHALLENGES FOR DoCMT-LLMS
Due to the maximum context limitation inherent in LLMs, translating lengthy document in single pass becomes unfeasible. conventional strategy involves segmenting the document into smaller translation windows and translating them sequentially. In our study, we initially leverage the GPT-3.5 5- -turbo-0122 model translate the IWSL12017 documents sourced from TED talks. We employ a window of size to facilitate document translation, where source sentences are simultancously processed to generate hypothesis sentences. Once all source sentences are translated, they are concatenated to form the complete target document. The primary challenges associated with DocMT-LLMs arise from the following two aspects.
= Zh test set, comprising 12
Translation Inconsistency Given a source document D, (s1, $2,.... 8N) and its correspond- ing target document D, (tz, ......, ......,ty .tt.......................) tv), if there exists a proper noun p E denotes the set of all proper nouns in D., including names of people, locations, and organizations), and p appears multiple times in D., we expect that all occurrences of translation D, should consistent.
Lyu al. (2021) propose the Lexical Translation Consistency Ratio (LTCR), a metric that quantifies the proportion of consistent translation pairs among all proper noun translation pairs in the target document. However, we argue that the translations of the proper nouns are supposed to not only maintain consistency throughout the document but also align their first appearance. This consid- eration is particularly important for enhancing the reading experience of audiences. Therefore, introduce the LTCR-1 metric for the DocMT- MM,s, which calculates the proportion of proper noun translations that consistent with the initial translation within the document:
LTCR-1(D.. D.) EpEp 1h,,1(T(P)) ((T(p) (p))
(1)
Epep (kp 1)
Tp(p) represents the i-th translation of p in D., and hip denotes the number of occurrences of p in the document. The indicator function 1(T((p) = (p)) returns if the translations T((p) and (p()) identical, and 0 otherwise. The numerator is the number of times the proper nouns appear again and their translation remains the same as their first appearance, and the denominator represents the sum of all occurrences except the first of all proper nouns. To compute this metric,
3

---

Published as a conference paper at ICLR 2025
initially annotate all proper nouns in the source document using spaCy | Subsequently, utilize the token align tool aaiign (Dou & Neubig 2021 to determine the translations of these proper nouns the target document. To mitigate the impact of errors from the alignment tool, we introduce a fuzzy match version of this metric, where two proper noun translations are considered consistent when one a substring of the other:
LTCR-I,(D,, D.) EpEp Eie ((T(P) E (T(() & T(())
ΣpEp Skp 1)
As shown in Table I translating every sentence separately (window size = 1) causes poor translation consistency. An example is illustrated in Appendix A] Increasing the window size consistently leads to higher scores across all three consistency metrics. This suggests that when more sentences are processed within single translation pass, the LLM is better able to model discourse phenomena and maintain consistent translation of proper nouns throughout the document. However, due to the inherent limitations in the context length of LLMs, resolving translation inconsistencies cannot be achieved solely by indefinitely expanding the window size.
Translation Inaccuracy When employing a large window size for document translation, LLMs tend to process the input source sentences cohesive documents rather than individual sentences. As result, the model prioritizes maintaining the general meaning of the text and loses track of the detailed information in each sentence. This can lead undertranslation issues and decline in translation quality (Karpinska & lyyer 2023 Wu et al. 2024a). We utilize two neural metrics to as- sess the quality of document translation. The first is the sentence-level COMET (sCOMET) scord for which utilize the model taabbellwwmt22-co000
obtain the scores. The second metric the document-level COMET (dCOMET) score" proposed by Vernikos et al. (022)). for which we use -ootet qqe-mtt----mm"" to derive reference-free ffre scores. In calculating this document-level metric, the model encodes previous sentences as context rather than encoding only the hypothesis, making this approach more accurate for evaluating document translations.
As illustrated in Table [ increase in window size correlates with higher tendency for the LLM to omit sentences from the source document, resulting in missing translations the target document. An example of this undertranslation issue is presented in Appendix A Additionally, quality metrics such as SCOMET and dCOMET do not demonstrate a consistent improvement with larger translation windows. Therefore,
conclude that translating documents in batches of several sentences at a time may introduce translation inaccuracy issues. These concerns are particularly sigmificant in contexts where precise translations are essential, such in technical manuals official documents.
3.2 WHY USING DOC2SENT APPROACH?
Previous experiments indicate that translating document by processing multiple sentences at once may occasionally result in sentence omissions. Although human translators often translate entire paragraphs simultaneously, which can also lead to occasional omissions, their underlying transla- tion mechanism is fundamentally distinct from that of DocMT-LLMs. DocMT-LLMs are prone to omitting source sentences due to hallucination issues limited capabilities in handling long texts effectively. Therefore, we argue that, at this moment, a Doc2Sent approach offers a more promis- ing alternative for DocMT-LLMs to produce precise and high-quality document translations. In our study, we provide LLMs with contextual information from the document while asking them translate each source sentence separately, Once all sentences translated, concatenate the target sentences to form the final target document.
https://spacy.io/ 1o/
httess//gsthhb.ccm/uuaaae///////
http::/uumballlsspprrintettttti www.aaawssaaaoommmmm http://sssss...aoaaoooooooooocoo h-mm://
gz

---

Published as a conference paper at ICLR 2025
Target Document
Document
Proper Extractor
Proper Noun Records
Document Translator
Source mary Writgr
Source Summary
Target Summary
Target
mary
Retrisyer
Long-Term Memory
Short-Term Memory
Figure 1: Framework DELTA. The modules outlined with dashed lines represent the multi-level memory components, while those outlined with solid lines denote the LLMM-based components. Memories closer to the top are more global, abstract, and densely packed with information. During translation, memory information retrieved and incorporated into the translator LLM's prompt. After the translation each sentence, the LLM-based components extract key information from both the source and target documents and update the multi-leve memory components.
DELTA: DOCMT AGENT BASED ON MULTI-LEVEL MEMORY
Considering the multi giriiirraatuttyyy and multi-scale of key information in the document during transla- tion, we introduce DELTA, an online DocMT agent. DELTA employs a multi-level memory stream that captures and preserves critical information encountered throughout the translation process. This memory stream accommodates wide range of perspectives, spanning from recent to historical, con- crete to abstract, and coarse-gras grained to fine- fine-grained details. DELTA translate the source document in sentence-by sentence--b-senteeee be-sentence manner while updating its memory in real-time. This approach addresses the context limitations of large language models and ensures the generation a sentence-level aligned target document, thereby preserving both the quality and rigorousness of the translation. The main framework of DELTA is illustrated Figure [ the algorithm DELTA detailed in Algorithm and the prompts used for each module are given in Appendix C
Proper Noun Records The first level of the agent's memory component we introduce is dictio- nary called the Proper Noun Records to store proper nouns p in the document along with their translations upon first encounter Ti (p) within the document: (( ((p Ti (p)) p E sj, 1 S
i), where ( represents the state of R before translating the i-th sentence, and the same applies to other components. When translating the subsequent sentence s,, the agent consults R(0) obtain all recorded proper nouns that are also contained in s;; (() {p. (i)) p E s,. (p. Ti()) E R()).
The Proper Noun Records are continuously updated by LLM-based component known as the Proper Noun Extractor LExamt After each sentence is translated, extracts newly en- countered proper nouns from the source sentence and their translations from the target sentence LExtre (s;, f4)
(p. (p))
s; (p)) E t, v(p., (x()) R(), ,p + p/} and add them to R.
Bilingual Summary Unlike previous studies (Wang et al.. 2023a Lee et al. 2024), our research implements a bilingual summary approach the second level of the agent's 's memory component to address the challenges extensive context on both the source and target sides. We maintain a pair of summaries throughout the translation process enhance accuracy and fluency. The Source-Side Summary Ag encapsulates the main content, domain, style, and tone of the previously translated sections of the document. This summary serves to preserve coherent understanding of the text's overall context, thereby aiding the LLMs in producing more accurate translations. Conversely, the Target-Side Summary A, focuses solely the main content of the previously translated target text.
The pair of summaries are generated by two LLM--baed components of the agent: the Source Summary Writer Lwites and the Target Summary Writer LWise. These summaries are updated every m sentences through a two-step process. Initially, the writers generate segment summaries for the last m sentences from both the source and target texts: (!++1) Lwitee fi-m-l,.....
5

---

Published as a conference paper at ICLR 2025
Algorithu 1: The Overall Framework of DELTA
imput
Source document D, (.1...., By J. Large Language Model Proper Noun Records R = Source-Side Summary A, = 0. Target-Side Summary A, 6,
Short-Term Memory M 0. Long-Term Memory N
output: Target document D. (N)
D.
for
1 N do
/x Retrieve memory
x/
/+ Search Proper Noun Records */ N + (81, /+ Match relative sentences from Long-Term Memory
R +
{(p. T((p)) 1p 6,, Ti(p)) E ER)
/N Translate with hybrid nenory information
+ [Tnnaee( A.. (1,)
Add hypothesis target documentt
/x Update memory
R + R.U. Lismaals;
/x Extract new proper nouns and add to records x/
N + - : [(91.
Last sentences as Long-Term Memory
.M 1i-k+1jjffe (G,,))
/+ Last k /4 Update Bilingual Summary every sentences x/
Short-Term Memory
if mod m
then
/x Generate source and target segnent summaries A) A, + --mar(t--mtts...
A. +
/* Herge segnent summaries into document summaries
A. + Lxtepss
A +- (At
end
end
A(++1) Cwiter -t. ty). Subsequently, these segment summaries are merged with the previous overall summaries for both sides to summary mergers to obtain new overall summaries: ( M (A, (A()) A(++1)), A
Marge (A A A((+))). This process repeated itera- tively until all sentences in the source document have been read.
Long-Term & Short-Term Memory The last two levels of the agent's memory component are the Long-Term Memory and the Short-Term Memory, respectively. These two components are designed to address the requisite coherence across document-level translations. The Short-Term Memory M retains the last & source sentences along with their corresponding translations, where represents a relatively small number: M = {((i-ky l(-x)..... (8--144-1) This component is specifically designed to capture immediate contextual information in adjacent sentences, which is then seamlessly integrated into the translation prompt, serving as the context for the current sentence.
Similarly, the Long-Tere Term Memory / component preserves broader range of context by maintain- ing a window of the last sentences from the source document, with being significantly greater than k storing extended coherent information throughout the document. Before translating given source sentence, an LLM-based component called the Memory Retriever LRemivv chooses n source sentences that are most relevant to the current translation query: A/(1) = LRetrieve N()).. These m sentences with their translations are subsequently employed as demonstration exemplars.
Document Translator We utilize LLM-based component called Document Translator Lraasstate to perform the final translation process. Information from the multi-level memory is integrated into the prompt to support the translator in producing high-quality and consistent translations:
4; = Transtatte (s;, R(0) ,A()), A("), AA (M()). The sentence-by-sentene approach ensures that the resulting target document is consistently aligned with the source document at the sentence level, effectively minimizing the risk of missing target sentences. Additionally, this method allows for straightforward evaluation of translation quality using sentence-level metrics such as sCOMET.
5 EXPERIMENTS
5.1 SETTINGS
Datasets Metrics We conduct our experiments on the two test sets. The first the tst2017 test sets from the IWSLT2017 translation tast (Akiba et al., 2004), which consists of parallel doc- uments sourced from TED talks, covering 12 language pairs. Our experiments are conducted on eight language pairs: En < Zh, De, Fr, and There are 10 sentence-leve aligned parallel
httpps://with fb..it....k....... eu/2017-01-d/

---

Published as a conference paper at ICLR 2025
En Xx
Xx En
System
SCOMET
dCOMET
LTCR-1
LTCR-1,
SCOMET
dCOMET
LTCR-1
LTCR-1;
NLLB
82.11
6.36
74.56
81.87
84.10
6.98
79.03
90.76
GOOGLE
80.41
5.83
81.38
84.72
80.17
5.96
81.43
90.81
GPT UPT-375-11
Turbo
Sentence
84,80
6.58
77.06
82.81
84.47
7.05
81.98
91.86
Context
85.40
6.70
77.34
83.12
84.97
7.15
85.03
95.27
Doc2Doc
6.62
79.12
86.39
6.96
85.17
92.98
DELTA 85.58
6.73
82,96
88,83
84,95
7.15
86.53
96,26
GPT-4o-mini
Sentence
81.51
6.35
78.59
85.07
84.01
6.99
81,42
91.34
Context
84.78
6.65
80.01
86,99
84,95
7.15
84,40
94.34
Doc2Doc
6.75
80.54
85.39
7.01
83.50
93.39
DELTA 85.85
6.80
81,80
86.33
85.26
7.24
85.25
95,89
Twen2-7B-Instrune
Sentence
80.03
5.96
73.91
79.54
77.10
6.48
76.39
87.94
Context
80.84
6.08
79.59
85.35
83,09
6.84
81.48
92.56
Doc2Doc
5.83
77.32
84.59
6.59
85.03
93.68
DELTA 81.02
6.07
80.09
87.78
83.36
6.84
82,05
93.30
Qwen2- 72 Instruct
Sentence 78.53
5.97
79.54
85.09
80.53
82.25
92.05
Context
80.79
6.22
79.14
85.40
83.27
82.86
Doc2Doc
6.45
73.58
78.64
83.00
DELTA 84.99
6.66
88.34
85.19
7.21
86.53
96.48
Averuge
Sentence 81.22
6.21
77.27
83.13
81.53
80.51
90.80
Context 82.95
6.41
79.02
85.21
84.07
7.03
83.44
93.59
Doc2Doc
6.41
77.64
83.75
6.86
84.18
92.70
DELTA
84.36
6.57
81.63
87.82
84.69
7.11
85.09
95.48
Table 2: Test results on the IWSLT2017 dataset. Since the translations produced by the Doc2Doc method are aligned at the sentence level with the source text, we do not report the sCOMET scores for this method. The highest score in each block is highlighted in bold font The results in the "Average" block represent the mean scores across the four backbone models.
documents with approximately 1.5K sentences for each language pair. The second Guofeng Web- nove Wang et al. 2023cc] 2024b), high-quality and discourse-level corpus of web fiction. We conduct our experiments on the Guofeng VI TEST_2 set the = En direction. The detailed dataset statistics are demonstrated in Appendix B We employ LTCR-1 and LTCR-1, I for proper noun translation consistency evaluation and adopt SCOMET and dCOMET as translation quality metrics, which are all introduced in $3.1
Models and Hyperparameters In this work, we utilize two versions of GPT models, and GPT-4o-mini, as our base models. We get access to these models through the official API provided by OpenA We also introduce the open-source (nnstr-cn and set to 2048 and other hyper-parameters remain default. The updating window of Bilingual summary and length of Long-Term Memory / are set to 20. The number of retrieved relative sentences from Long-Term Memory n set to 2. The length Short- Term Memory à is set to
in our experiments. The nax_new_tokens is
Baseline Methods We include the following three approaches as our baselines: a) Sentence: We employ the same LLMs but conduct sentence-level translation process to obtain the baseline sults. b) Context: We follow Wu et al. (2024a) to provide the LLMss with three previously obtained source-target sentence pairs to integrate more contextual information and adapt LLMs for document- level translation tasks. c) Doc2Doc: We reproduce the approach proposed by Wang et al. (2023b), translating 10 sentences in single conversation turn and processing the entire document within
https://huggingfface httpp//ggnnnaaaeee//////////e/ee

---

Published as a conference paper at ICLR 2025
System
sCOMET
dCOMET
LTCR-1
LTCR-I, sCOMET
sCOMET
LTCR-1
LTCR-I,
GPT-3. 5-Tubbo
GPT-40-min
Sentence
77.62
3.07
61.58
78.82
77.87
3.10
58.82
70.59
Context
78.57
3.19
70.10
81.37
78.56
3.19
64.32
74.37
Doc2Doc
2.82
77.46
89.02
2.96
82.04
91.62
DELTA
78.45
3.17
85,57
96.52
78.77
3.34
88.94
96,48
Qwen2 10-12-78-iae
Instruct
Qwen2-722 7
21122772-Inntrrct
Instruct
Sentence
73.65
2.62
37.00
50,00
75.15
2.98
58.00
71.50
Context
76.54
3.01
52.82
61.54
77.87
3.20
58.21
70.15
Doc2Doe
73.25
84.08
2.77
80.79
90.07
DELTA
76.95
85.50
94.00
78.32
3.31
86.93
95.98
Table 3: Test results on the Guofeng dataset.
single chat box, thereby leveraging LLMs' long-term modeling ability. In computing the metric scores for the Doc2Doc results, first perform sentence alignment using Bleualign [ to obtain aligned source-target documents, then we calculate the involved metrics. Furthermore, we also in- troduce the results of NLLB-3. (Costa-jussà al. 2022) and GoogleTrans for comparison.
5.2 RESULTS
Test Results IWSLT2017 The main experiment results the IWSLT2017 test demon- strated in Table 2 For more detailed scores, please refer to Appendix [ It is evident that DELTA outperforms baseline approaches on LTCR-1 and LTCR-1, metric scores across nearly all translation directions and models. This indicates that our approach yields significant enhancements in proper noun translation consistency for document-level translation. Furthermore, DELTA significantly im- proves the overall quality of document translation, as evidenced by consistently higher SCOMET and dCOMET scores. The superior dCOMET scores indicate that DELTA effectively captures contex- tual information to support the translation process. Translation consistency is improved significantly in the En = Zh direction, while gains in directions like En + De are modest. For instance, with UPT-3.5-Turbo LTCR-1 improves by 6.17 percentage points (86.44 vs. 80.27) for En Zh, but only 1.40 points (93.46 vs. 92.06) for En + De (see Table 13 of Appendix DD. This disparity stems from linguistic differences: English proper nouns require conversion into Chinese characters, posing challenges for maintaining consistency, whereas in German, they can be directly copied due to the shared alphabet. Despite this, reasonable LTCR-1 improvement in En + De still demonstrates our method's effectiveness. The p-values of t-tests for DELTA vs Sentence/Context quality are less than 0.05 for En <> Xx, whereas that in translation consistency less than 0.05 in Zh. We also test DELTA on the low-resource language pair, as demonstrated in Appendix
translation
En +
Test Results on Guofeng The test results on Guofeng are illustrated Table 3 Our ap- proach achieves superior results across almost all metrics and backbone models, demonstrating its robustness to data of the novel domain. Notably, stronger models, such as GPT-4o-mini and Owen2-72B-Instruce achieve greater improvements in translation consistency and quality metries. This suggests that the stronger the backbone models are, the more substantial the gains achieved by DELTA. The Guofeng test poses particular challenges for maintaining translation consistency due to the prevalence of proper nouns (mainly names) in the source text. Nevertheless, DELTA demonstrates significant improvements in the relevant metrics, with an increase in LTCR-1 of up to 48.50 percentage points (85.50 vs. 37.00). This indicates that DELTA represents a strong tool for addressing translation inconsistency issues and holds great potential for novel translation, as it ef- fectively reduces inconsistent noun translations, thereby minimizing potential confusion for readers.
6 ANALYSIS
Ablation Study Table 4 presents ablation study in the En Zh direction using
as the backbone model. For more detailed ablation studies of each memory
https:://p--ooglettree itt.rraan..raathedoss.....
8

---

Published as a conference paper at ICLR 2025
Id
Setting
sCOMET
dCOMET
LTCR-1
LTCR-I;
Sentence-
83.78
6,55
80.27
88.78
2
1 + Short-Term Memory
84.50
6.68
77.89
87.41
3
2 + Long-Term Memory
84.54
6.67
79.23
89.44
4
3 Source Summary
84.61
6.68
76.09
91.25
5
3 Target Summmar
84.70
6.72
82.14
92.86
6
3 Bilingual Summary
84.72
6.74
82.49
93.60
7
6 Record (DELTA)
84.70
6.72
86.44
95.25
Table 4: Ablation Study.
component and the effects of syper-parameeers, please refer Appendix When provided with context sentences (Model 2), the model exhibits improved translation quality scores, but signif- icant enhancement in translation consistency is observed. The introduction long-term memory contributes to more consistent translations (Model 3). Incorporating bilingual summaries (Model 6) led to an increase in COMET scores as well as consistency metrics, indicating that this compo- not only enhances translation quality but also reinforces translation consistency. When proper noun records are introduced (Model 7), a slight decrease in sCOMET and dCOMET scores is ob- served, likely due the perturbation introduced by incorporating additional information. However, translation consistency improves significantly, with LTCR-1 increasing by 3.95 points and LTCR-I, increasing by 1.65 points compared Model 6. Moreover,
evident that the bilingual summary has superior impact on both translation quality and consistency compared to using summary on either the source side or the target side alone. Among Models 4, and 6, Bilingual Summary achieves the highest scores across all four metrics.
Consistency Distance One significant challenge in document-level translation is maintaining long-term consistency. To evaluate whether approach ad- dresses this challenge, we divide the sentence-wise distance between each proper noun's translation and its first occurrence into several intervals. We then report the proportion of consistent translations in each inter- val in En + Xx (upper) and Xx = En (lower) in Figure 2 We observe that our approach almost outperforms the Sentence and Context methods in achieving proper noun translation consistency across all distance inter- vals. Notably, our approach excels when the distances exceed 50 sentences, yielding a larger proportion of consistent translations than the baseline methods. This demonstrates the effectiveness of our approach in en- hancing long-context translation consistency.
Consistent Translations of Different Distances
Comioens Trastatio Condatem Traniation
0.8
0.9
0.7
11-20
21-50
Sentene-Wise Distanse
Figure 2: Proportions of consistent transla- tions in different sentence-wise distances.
Pronoun & Context-Dependent Translation We follow Miculicich et al. (2018); TTnn el al.] (201)); Lyu et al. (2021) to evaluate the accuracy of pro- noun translation (APT) of our system in En = Zh using the reference-based metric proposed by Mi- culicich Werlen & Popescu-Belis (2017). We also evaluate our system on the first 1000 instances in of
Metrie
Sentence
Context
Doc2Doc
DELTA
APT
59.96
60.84
$6.11
61.07
Table 5:
Evaluation results of pronoun
translation accuracy (APT).
Metric
Sentence
DELTA
the En De subset "mini gender.opensuubiinn. context-dependent translation benchmark called CTX- PRO Wicks & Post (2023). The results achieved by GPT-3_5-Turbo-011 25 are demonstrated in Tand Table DELTA improves the performance of pronoun translation compared to the Sentence and Context baselines, and enhances translations where con- text information is explicitly needed, These results indicate that the multi-level memory in DELTA is beneficial to resolving coreference and discourse issues in the document.
Generative Accuracy (%)
29.7
$1.0
Table 6: Evaluation results of context- dependent translation.

---

Published as a conference paper at ICLR 2025
DELTA a Summarize Writer To assess the effec- tiveness of our summary component, we conduct an exper- iment QMSum (Zhong et al., 202I), a benchmark for query-based summarization, where systems are required to generate summaries of the meeting transcripts in re- sponse to a specified query. In our experiment, the query incorporated into the prompts for both summary gener- ation and the merging process. We compare our results with those Le et al. (2024), who paginate the document, generate summaries for each page, and then perform a lookup process on these sum- maries according to the query. The results are shown in Table 7 summarization approach enables us to effectively locate relevant portions of the document and syn- thesize the information through the summary merging process. This demonstrates that the summary component of DELTA is also well-suited for general summarization tasks.
System
ROUGE-L
Length
READAENT
21.50
67.86
DELTA
23.60
82.28
Table 7: QMSum test results ReadA- gent and DELTA. "Length" denotes the word-wise length of the response.
Our system' 's sigment-by-segmeeet
Memory Cost of Two Appronches
Memory Costs Our agent system consumes less mem-
ory than LLM-based Doc2Doc methods, such as those described by Wang et al. (2023b). where the whole document is translated within single chat box, suf- fering from significant memory costs. As shown in
MocupledMMann (GB)
mmww
150-
140-
Figure 3. we compared the memory usage by utilizing Twen2-72B-Instruee
130
DoeoTTuee
DELTA
to translate a document in En + Zh on a device with 2 NVIDIA A800 80GB GPUs. Our method demonstrates relatively slow memory growth as the number of processed sentences increases, primarily due to the increasing length of the summaries. In contrast, the memory usage of the Doc2Doc method increases significantly as the number of processed sen- tences grows, ultimately leading to memory exhaustion when the count reaches 490. This indicates that our approach is more memory-efficient and cost-effective for deployment on local devices.
0
200
400
600
800
1000
1200
Figure 3: Memory cost of the Doc2Doc approach and our approach.
7 CONCLUSION
In this paper, we analyze critical challenges for DocMT-LL.Ms: translation inconsistency and inaccuracy. We design an online document-level translation agent equipped with a multi-level mem- ory component to tackle them. This memory structure retrieves and stores key information assist the translation process, significantly enhancing translation consistency and quality. Notably, the ef- fectiveness in maintaining proper noun translation consistency is particularly pronounced in novel translation, and our approach is still able to maintain consistency even when there is a large dis- tance between the occurrences method avoids sentence omissions and reduces GPU memory consumption, in contrast to main- stream Doc2Doc approaches. Further analysis indicates that our framework can model document discourse structures to improve pronoun translation and context-dependent translation accuracy and the summarizer component our agent also capable of the query-based summarization task.
proper noun pair. The sentence-by-sentence online translation
LIMITATIONS
In this work, we present framework for the DocMT agent, without prioritizing its inference ef- ficiency. Given the complexity of DELTA's 's inference process, LLMs are frequently invoked dur- ing document translation, leading to prolonged runtime. To address this issue, we identify several potential directions for improvement. First, by explicitly marking sentence boundaries with spe- cial boundary tags, we can enforce sentence-level alignment within generated paragraphs, allowing LLMSs to process multiple sentences concurrently and thereby reduce invocation frequency. Second, employing more precise alignment tools and scripts to extract proper nouns and their translations, rather than relying LLMs, can further enhance the efficiency of DELTA. Other components, such as Long- Term Retriever, could be implemented using a dense retriever rather than employing LLMs to identify related sentences. Finally, in the summary component, reducing the summary generation to a single step by directly merging sentences within the window into an overall summary can also decrease runtime. We consider these optimizations to be future directions for work.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported in part by the National Natural Science Foundation of China (Grant No. 62206076), Guangdong Basic and Applied Basic Research Foundation
(Grant No. 2024A151501 1491), Shenzhen Science and Technology Program (Grant Nos.
KQTD2004072910115500 KIZD202310230990000000 Wong was supported in part by the Science and Technology Development Fund of Macau SAR
DDSYS222306622912000009
Derek F.
(Grant Nos. 0007/2024/AKP, FDCCT0000/2022/MMM FDCC//60/2022/FFF), and the UM and UMDF (Grant Nos. MYRG-GRRC0033-0006-TTTT-T
( EF2024-00185-FST, EF2023-00151-FST. EF2023-00090-FST We would like thank the anony- mous reviewers and meta-reviewer for their insightful suggestions.
REFERENCES
Yasuhiro Akiba, Marcello Federico, Noriko Kando, Hiromi Nakaiwa, Michael Paul, and Jun ichhi Tsujüi. Overview of the IWSLT evaluation campaign. In Proceedings of the First International Workshop on Spoken Language Translation: Evaluation Campaign, Kyoto, Japan, 2004. URL https://aclanthol.. org/2004, wew.lllt-valltiin.11...
Guangsheng Bao, Yue Zhang, Zhiyang Teng, Boxing Chen, and Weihua Luo. G-transformer for document-level machine translation. Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeling of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 3442-3455, Online, 2021. Association for Computational Linguistics. v1/2021 1//2021.aclo www.accl-oon.0000 267. https: / http:///alaathooogooooooooo org/ /201//2211222 acl long.
LingutttiionnllLLnuuttttessss doi: 10.18653/
Rachel Bawden, Rico Sennrich, Alexandra Birch, and Barry Haddow. Evaluating discourse phenom- ena in neural machine translation. In Marilyn Walker, Heng Ji, and Amanda Stent (eds.), Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Laguuge Technologies, Volume I (Long Papers), pp. 1304-1313, New Or- leans, Louisiana, 2018 Association for Computational Linguistics. doi: 10. 10.18853/ 1118. URL https://aaclantholooyy 1ttpa//annoooooooooor//111-1111-
Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. No language left behind: Scaling human-centere machine translation. ArXiv preprint, 0/5/2207.046722 2022. URL https://
arxiv.org/ www..rrr/ab///.....mm 22670
Zi-Yi Dou and Graham Neubig. Word alignment by fine-tuning embeddings parallel corpora. In Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty (eds.), Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguisties: Main Volume, pp- 2112-2128, Online, 2021. Association for Computational Linguistics. doi: 10.18653/v12021. cacl-main.181. URL htttps:://aclaatooooog/// org/ www.-lallnnn....
Zhaopeng Feng, Yan Zhang, Hao Li, Wenqiang Liu, Jun Lang, Yang Feng. Jian Wu, and Zuozhu Liu. Improving llmebased machine translation with systematic self-correction. ArXiv preprint, abs/2402.16379, 2024.
httpxxvv.orr//eo hwtg//bs/200 /twp///2..... 16379
Patrick Fernandes, Kayo Yin, Graham Neubig, and André F. T. Martins. Mcasuring and increasing context usage in context-aware machine translation. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computa- tional Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 6467-6478, Online, 2021. Association for Computational Linguis- tics. doi: 10.18653/v1/2021 1ttp://////211.lllllolooo 505. URL htw.htpss://m http::s:s//alaanhooogg..oo 202/ acng. 505
Xavier Garcia and Orhan Firat. Using natural language prompts for machine translation. ArXiv preprint, abs/2202. 1822, 2022. URL
11822
Marjan Ghazvininejad, Hila Gonen, and Luke Zettlemoyer. Dictionary-basee phrase-level prompt- ing of large language models for machine translation. ArXiv preprint, abs/2302.07856, 2023. URL http:tttss///ii...oooo wwtp:xwwwwwgggb/////eee 2288000022220066666

---

Published as a conference paper at ICLR 2025
Zhiwei He, Tian Liang, Wenxiang Jiao, Zhuosheng Zhang, Yujiu Yang, Rui Wang, Zhaopeng Tu, Shuming Shi, and Xing Wang. Exploring human-like translation strategy with large language models. Transactions of the Association for Computational Linguistics, 12:229-244 2024. doi: 10.1162/tacl a 00642. URI htt.hhttss///allantoooooo
org/ tacl-1.13
Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Mat- sushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. How good are gpt models at machine translation? a comprehensive evaluation. ArXiv preprint, abs/2302.09210. 2023. URL
URL http://aarxiv..ogabbb//////
09210
Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, Shuming Shi, and Zhaopeng Tu. Is chatgpt a good translator? yes with gpt-4 as the engine, ArXiv preprint, abs/2301.08745, 2023.
www.orgg////////eee//.../eeeee/e 2301 08745
Marzena Karpinska and Mohit lyyer. Large language models effectively leverage document-level context for literary translation, but critical errors persist. In Philipp Koehn, Barry Haddow, Tom Koemi, and Christof Monz (eds.). Proceedings of the Eighth Conference on Machine Translation, pp. 419 451, Singapore, 2023, Association for Computational Linguistics. doi: 10.18653/v1/ 2023. wmn- 1.41. URL |
org/ 2023. wmt-1.41
Yunsu Kim, Due Thanh Tran, and Hermann Ney. When and why document-level context useful in neural machine translation? In Andrei Popescu-Belis Sharid Loáiciga, Christian Hardmeier, and Deyi Xiong (eds.). Proceedings of the Fourth Workshop on Discourse in Machine Translation (DiscoMT 2019), pp. 24-34, Hong Kong, China, 2019. Association for Computational Linguis- tics. doi: 10.1 16653/11/11--./.0
Kuang-Huei Lee, Xinyun Chen, Hiroki Furuta, John Canny, and Ian Fischer. A human-inspired reading agent with gist memory of very long contexts. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Pro- ceedings of Machine Learning Research, pp. 26396-26413 PMLR, 21-27 Jul 2024. URL https:: https://prooceddngg.s mlrm wwwpppeessss/seeeee httpp//s.......rees//////////eee html
Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. CAMEL: communicative agents for "mind" exploration of large language model society. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neu- ral Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, Decem- ber 10 16. 2023, 2023a. URL http: / http::/paappssss..c nips. wwtpp//pppp.leppppppeepeeeeeeeee /ipilee//pperr/22//aaalll h023/hash/ a362 htt.e//eeeeeeeooooooaaaeeeeeeeee o22e2
html
Junkai Li, Siyu Wang. Meng Zhang, Weitao Li, Yunghwei Lai, Xinhui Kang, Weizhi Ma, and Yang Liu. Agent hospital: A simulacrum of hospital with evolvable medical agents. ArXiv preprint,
abs/2405.02957,, 2024. URL htt.:/tpss//arii..o///////////// wwg./b///////..e..oooooo /t0p//0000/20000555577
Yachao Li, Junhui Li, Jing Jiang. Shimin Tao, Hao Yang, and Min Zhang. P-transformer: Towards better couumennt-to-ocumeee neural machine translation. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023b. URL htttss://ieexpppoot/ wewe..eeggg................ wtt.///b///ar///////o/oo/////..t
(ocument/10255243/
Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang. Yan Wang, Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu. Encouraging divergent thinking in large language models through multi- agent debate. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 17889-17904,
Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 18653/v1/200
URL httph::/ttttcaneeet
org/ 2024. emnlp main. 9922
Liangxin Liu, Xuebo Liu, Derek F. Wong, Dongfang Li, Ziyi Wang, Baotian Hu, and Min Zhang. SelectI:: Selective instruction tuning for LLMs via uncertainty aware self-reflection. In The Thirty-eigh eighth Annual Conference on Neural Information Processing Systems, 2024. URL/https:
//openneview.net///e wtt.eewwww....oooooooooooo/otao.
12

---

Published as a conference paper at ICLR 2025
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. Multilingual denoising pre-training for neural machine trans- lation. Transactions of the Association for Computational Linguistics, 8:726-742 2020. doi: 10.1162/tacl a 00343. http::hts:///aal.
org/2020. tac1-1.47
Lorenzo Lupo, Marco Dinarelli, and Laurent Besacier. Focused concatenation for context-aware neural machine translation. In Philipp Koehn, Loic Barrault, Ondřej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Alexander Fraser, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Tom Koemi, André Martins, Makoto Morishita, Christof Monz, Masaaki Nagata, Toshiaki Nakazawa, Matteo Negri, Aurélie Névéol, Mariana Neves, Martin Popel. Marco Turchi, and Marcos Zampieri (eds.). Proceedings of the Seventh Conference on Machine Trans- lation (WMT), pp. 830-842, Abu Dhabi, United Arab Emirates (Hybrid), 2022. Association for Computational Linguistics. URL BtLh:sts::/aaa /etttools//edaattlloooy
tntlology www.gg..rggo... org/2022. wmt-1.77
Xinglin Lyu, Junhui Li, Zhengxian Gong, and Min Zhang. Encouraging lexical translation consis- tency for document-level neural machine translation. In MaarieFranc Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (cds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3265-3277, Online and Punta Cana, Dominican Republic, 2021. Association for Computational Linguistics. doi: 00.18653/v1/2000 1r2222..emmlpnmn. main. 262. URL htttttssss/alaatllllooggg
2021 emnlp- main.
Sameen Maruf and Gholamreza Haffari. Document context neural machine translation with memory networks. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1275-1284, Mcl- bourne, Australia, 2018. Association for Computational Linguistics. doi: 10.18653/v 1/P/65/////118 1118. URL https::/acclanthhlll
org//18 1118
Sameen Maruf, Fahimeh Saleh, and Gholamreza Haffari. A survey on document-level neural ma- chine translation; Methods and evaluation. ACM Computing Surveys (CSUR). 54(2):1-36, 2021.
StRLhhtpss:/dl... acmm://ddlcccm.m....mmmmmmm orwoooggg//oooo /tt///////bbb/////////////// 1145/3441691
Lesly Miculicich, Dhananjay Ram, Nikolaos Pappas, and James Henderson. Document-level neural machine translation with hierarchical attention networks. In Ellen Riloff, David Chiang, Julia Hockenmaier, and inhi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Meth- ods in Natural Language Processing, pp. 2947-2954, Brussels, Belgium, 2018. Association for Computational Linguistics. doi: 10.18653/v 1325. URL https: //acl http//aalattoooyyy..ooooooo D18-1325
Lesly Miculicich Werlen and Andrei Popescu-Belis Validation of an automatic metric for the accuracy of pronoun translation (APT). In Bonnie Webber, Andrei Popescu- Belis, and Jörg Tiedemann (eds.), Proceedings the Third Workshop on Discourse in Machine Translation, pp. 17-25, Copenhagen, Denmark, 2017. Association for Computational Linguistics. doi: 10.18653/11/117-483 UKL//ttps:///al.nnnooooooo.og///
OpenAl. Gpt-4 technical report, 2023. URL
08774
Joon Sung Park, Joseph O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th annual acm symposium on user interface software and technology, pp. 1-22, 2023. URL /ttphhttts//... acm. www.:oorg//oo///oooo
Keqin Peng, Liang Ding, Qihuang Zhong, Li Shen, Xuebo Liu, Min Zhang, Yuanxin Ouyang, and Dacheng Tao. Towards making the most of ChatGPT for machine translation. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Lin- guistics: EMNLP 2023, pp. 5622-5633, Singapore, 2023. Association for Computational Lin- guisties. doi: 10. 10.18//53v1ll 1/2/33//12222222 http:////22....nnggeeeeemmm findings-emmll..37. URL /tttttss///calaatoooogyy findings- 373
Carolina Scarton and Lucia Specia. A quantitative analysis of discourse phenomena ma- chine translation. Discours. Revue de linguistique, psycholinguistique informatique. A jour- nal of linguistics, psycholinguistics and computational linguistics, (16), 2015. URL https: hrgpomail..ppeeentttoooooo////ls
13

---

Published as a conference paper at ICLR 2025
Sai Cheong Siu. Chatgpt and gpt-4 for professional translators: Exploring the potential of large language models in tramslation. Available at SSRN 4448091, 2023. URL https://papers.
aorn.com/ol133 http:///33ppppsr....m// cfm?abstract 1d=444801
Zengkui Sun, Yijin Liu, Fandong Meng. Jinan Xu, Yufeng Chen, and Jie Zhou. LCS: A language converter strategy for zero- shot neural machine translation. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 9201-9214, Bangkok, Thailand, August 2024. Association for Computational I Linguistics. doi: 10.186533/1/2024. wiwddiigsnannl..5555 URL http:hpss//aaaaaatolaaaao org.ohooooooooooooooooooooo ottoorg////// 2024. findings-ach 547/
Zewei Sun, Mingxuan Wang, Hao Zhou, Chengqi Zhao, Shujian Huang, Jiajun Chen, and Lei Li. Rethinking document-level neural machine translation. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Findings of the Association for Computatioonal Linguistics: ACL 2022, pp. 3537-3548, Dublin, Ireland, 2022. Association for Computational Linguistics. doi: 10. 2ttp://w1//222...nnnng--a--ee
URL httpp:://aclantthlllll .rgg 2wwg./202.2222 findings-acl.
279
Xin Tan, Longyin Zhang, Deyi Xiong, and Guodong Zhou. Hierarchical modeling of global con- text for document-level neural machine translation. In Kentaro Inui, Jing Jiang, Vincent Ng. and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Process- ing (EMNLP-IJCNLP), pp. 1576-15655 1585, Hong Kong, China, 2019. Association for Computational Linguisties. doi: 10. 10.11165/11111-11111 1168. htttttsss///ccaattttto
1168
Xin Tan, Longyin Zhang, and Guodong Zhou. Coupling context modeling with zero pronoun re- covering for document-level natural language generation. In KranicnFa Moens. Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 2530- 2540, Online and Punta Cana, Dominican Republic, 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/2021. emnlp-main. 197. URL htpp:ttt////ccllnttoogg wrwp:/thhhhooooooo////eoo 2021. emnlp-main. 197
Giorgos Vernikos, Brian Thompson, Prashant Mathur, and Marcello Federico. Embarrassingly easy document-level MT metrics: How to convert any pretrained metric into a lecument-level metric. In Philipp Koehn, Loic Barrault, Ondřej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa- jusså, Christian Federmann, Mark Fishel, Alexander Fraser, Markus Freitag, Yvette Graham, Ro- man Grundkiewicz, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Tom Koemi, André Martins, Makoto Morishita, Christof Monz, Masaaki Nagata, Toshiaki Nakazawa, Matteo Negri. Aurélic Névéol, Mariana Neves, Martin Popel, Marco Turchi, and Marcos Zampieri (eds.), Proceedings of the Seventh Conference on Machine Translation ( (WMT), pp. 118-128, Abu Dhabi, United Arab Emirates (Hybrid), 2022. Association for Computational Linguistics. URL https://accaanthol.. org 2022. 1m.
Bing Wang, Xinnian Liang, Jian Yang. Hui Huang. Shuangzhi Wu, Peihao Wu, Lu Lu, Zejun Ma, and Zhoujun Li. Enhancing large language model with self-controlled memory framework. ArXiv preprint, abs/2304.13345 2023a. URL htttps://aaxiv.orrtt absaabb//.../. 13343
Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang. Jingsen Zhang, Zhiyuan Chen, Ji- akai Tang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents. Frontiers of Computer Science, 1866):186345, 2024a. URL http::ttts/////t http:://inkk.siiiiii com/article/10. (tteiiccee11..000//1111 00001//11100--9244422000
Longyue Wang, Zhaopeng Tu, Andy Way, and Qun Liu. Exploiting cross-sentence context for neural machine translation. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel (eds.), Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2826-288 Copenhagen, Denmark, 2017. Association for Computational Linguistics. doi: 18.15//3/v1/ D17-1301. URL http::tts://aae /tt.htttp///allaatooooyy.
Longyue Wang, Chenyang Lyu, Tianbo Ji, Zhirui Zhang, Dian Yu, Shuming Shi, and Zhaopeng Tu. Document-level machine translation with large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural
14

---

Published as a conference paper at ICLR 2025
Language Processing, pp.
Singapore, 2023b. Association for Computational Lin- guistics. doi: 10. 10.18653/112023..mmml manip-mmii..136.. URL https:///allanthology/ org/2023.
emnlp-main. 1036
Longyue Wang, Zhaopeng Tu, Yan Gu, Siyou Liu, Dian Yu, Qingsong Ma, Chenyang Lyu, Lit- ing Zhou, Chao-Hong Liu, Yufeng Ma, Weiyu Chen, Yvette Graham, Bonnie Webber, Philipp Koehn, Andy Way, Yulin Yuan, and Shuming Shi. Findings of the WMT 2023 shared task on discourse-level literary translation: A fresh orb in the cosmos of LLMs. In Philipp Koehn, Barry Haddow, Tom Koemi, and Christof Monz (eds.), Proceedings of the Eighth Conference on Ma- chine Translation, pp. 55-67, Singapore, 2023c. Association for Computational Linguistics. doi: wmt-1.3
10.18653/v 1/2013.222mmmmm/m wmt- 1.3.
Longyue Wang. Siyou Liu, Minghao Wu, Wenxiang Jiao, Xing Wang, Jiahao Xu, Zhaopeng Tu, Liting Zhou, Yan Gu, Weiyu Chen, Philipp Kochn, Andy Way, and Yulin Yuan. Findings of the wmt 2024 shared task on discourse-level literary translation. In Proceedings of the Ninth Conference on Machine Translation, 2024b.
Yutong Wang. Jiali Zeng, Xuebo Liu, Fandong Meng, Jie Zhou, and Min Zhang. TasTe: Teach- ing large language models to translate through self-reflection. In Lun-Wet Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pp. 6144 6158, Bangkok, Thailand, August 2024c. Association for Computational Linguistics. doi:
URL
acl long........ 333
Rachel Wicks and Matt Post. Identifying context-dependent translations for evaluation set pro- duction. In Philipp Koehn, Barry Haddow, Tom Kocmi, and Christof Monz (eds.). Proceed- Machine Translation, pp. 467,467, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 122663311/222. URL ptfps://accanttolloe org/ wtw.g/////22./eee wwt.0/2.....1.....mm1....m. wmt-1.44 42
ings of the Eighth Conference
Minghao Wu, Thuy-Trang Vu, Lizhen Qu, George Foster, and Gholamreza Haffari. Adapting large language models for document-level machine translation. ArXiv preprint, 0e//2201.064666 2024a.
URL http:/tsss//aatv.orraaba////////
06468
Minghao Wu, Yulin Yuan, Gholamreza Haffari, and Longyue Wang. (perhaps) beyond human translation: Harnessing multi-agent collaboration for translating ultra-long literary texts. ArXiv preprint, abs/2405. 11804, 2024b. URL https:/:s:///xxii wtg./b//////aa/a//ooo.s
abs 2405. 11804
Yangjian Wu and Gang Hu. Exploring prompt engineering with GPT language models for document-level machine translation: Insights and findings. In Philipp Koehn, Barry Haddow, Tom Kocmi, and Christof Monz (eds.), Proceedings of the Eighth Conference on Machine Translation, pp. 166-169, Singapore, 2023. Association for Computational Linguistics. doi: 10.18653/v1/22 10132/11/223..o wmt- 1.15. URL https://aclaattholggy wwwptnthoooggoooog//..2202 wmt 1. 1.15
Wenda Xu, Daniel Deutsch, Mara Finkelstein, Juraj Juraska, Biao Zhang, Zhongtao Liu, William Yang Wang, Lei Li, and Markus Freitag. LLMRefine: Pinpointing and refining large language models via fine-grained actionable feedback. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Findings of the Association for Computational Linguistics: NAACL 2024, pp. 14499-1445 Mexico City, Mexico, 2024. Association for Computational Linguistics. URL findings-naace http://r//22....iin-n-naaaaa 92
org/2024
Jiali Zeng, Fandong Meng, Yongjing Yin, and Jie Zhou. Teaching large language models to translate with comparison. Proceedings of the AAAI Conference on Artificial Intelligence, 38(17)19988- 19496, Mar. 2024. doi: 10. wttp://..iii11..9999.99 URL https: //ojs. aaai org/index php/ AAT/articleeview/2255
Biao Zhang, Barry Haddow, and Alexandra Birch. Prompting large language model for machine translation: A case study. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engel- hardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 41092-41110 PMLR, 2023. URL https: http:////oceceeen..///e www.mnsss........... .ttpeeeiiiig.....reess.// press/ v202/zhang23m.hh html
15

---

Published as a conference paper at ICLR 2025
Jiacheng Zhang, Huanbo Luan, Maosong Sun, Feifei Zhai, Jingfang Xu, Min Zhang, and Yang Liu. Improving the transformer translation model with document-level context. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun aun'ichi Tsujii (eds.), Proceedings of the 2018 Con- ference on Empirical Methods in Natural Language Processing, pp. 533-542, Brussels, Bel- gium, 2018. Association for Computational Linguisties. doi: 10. 10418699///11810099. URL
Pei Zhang, Boxing Chen, Niyu Ge, and Kai Fan. Long-short term masking transformer: A sim- ple but effective baseline for document-level neural machine translation. Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference Empir- ical Methods in Natural Language Processing (EMNLP), pp. 1081-1087, Online, 2020, Asso- ciation for Computational Linguistics. doi: 2211663//11/022.mmmlpp-- //aclanthology.u or org 2020. emilp-main.81
URL https:
Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadal- lah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, and Dragomir Radev. QMSum: A new benchmark for query-based multi-domain meeting summarization. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5905-5921, Online, 2021. Association for Computational Linguistics. doi: 10.18653/v1/ 2021 me2c.naacl-man main----a...7. URI http:/tttppe//caaatttooooooo///p naacl-main. 472
16

---

Published as a conference paper at ICLR 2025
Proper Noun Translation Inconsistency
SRC
It's a story about this woman, Natalia Rybezynski.
HyP
T de
SRC
Natalia Rybezynski: Yeah, hadd someone call me "Dr. Dead Things." n -
HYP
Undertranslation
But here's 's the Hrurh the opiphany that had that changed my thinking 1970 today, the percentage of the world's population living in starvation levels...
SRC
[ # // *Missing translation // 1970 4 / #
HYP
KUEE ( BE
M 80%
Low Translation Quality
Ánd we make decísions about where to live, who to marry and even who our friends are going
SRC
to be, based on what we already believe.
I F4 dE S
REF
HYP
HYP50
MERN
Table 8: Demonstrations of proper noun translation inconsistency. undertranslation, and low transla- tion quality issues encountered during document-level translation. the first part, texts highlighted in red represent the same source proper noun with two different translations in the target document. In the second part, "//" indicates sentence boundaries in the document, illustrating that the transla- tion of the second source sentence is absent from the target document. In the third part, "HYP," represents the hypothesis generated by the LLM using a translation window of size n.
A EXAMPLES OF TRANSLATION INCONSISTENCY AND INACCURACY
Examples translation inconsistency, undertranslation, and low translation quality issues are pre- sented in Table In the first instance, the name "Natalia Rybezynski", which appears twice in A NE- to म H and "M HH This variation leads to a notable inconsistency the translation. In the second instance, the second sentence in the source document is omitted. Its corresponding translation is absent in the target document. This exemplifies an undertranslation issue in the document trans- lation task. In the third instance, the translation produced by the LLM with translation window of 50 sentences deviates significantly from the customary word order Chinese compared to the sentence-level approach (using window size of 1).
the source document, is translated into two different forms: "M A
B DETAILED STATISTICS OF THE DATASETS OUR EXPERIMENTS
We conduct our experiments on two test sets: IWSL72017 and Guofeng. The IWSLT2017 test set consists of parallel documents sourced from TED talks, covering 12 language pairs. Our experi- ments are conducted eight of these pairs, En + Zh, De, Fr, and Ja. Each language pair contains 10 to 12 sentence-lever aligned parallel documents, totaling approximately 1.5K sentences, with an average of around 120 sentences per document. Detailed statistics for each language pair are shown in the first block of Table
The Guofeng Webnovel corpus is high-quality and discourse-level corpus of web fiction, and conduct our experiments on the Guofeng VI TEST 2 set, which is designed in the Zh = En language pair. The second block of Table illustrated the statistics of the test sets.
C PROMPT TEMPLATES FOR LLM-BASED COMPONENTS IN DELTA
This part details the prompts used for each module of DELTA. The prompt template for the Proper Noun Extractor is depicted in Figure 4 We use a prompt in the few-shot style to ensure accurate and formatted outputs. The prompt templates of the source and target summary writers are shown in
17

---

Published as a conference paper at ICLR 2025
Dataset
Language
[S]
[D]
[S|[D]
Zh En
1459
12
122
De En
1138
10
114
IWSLT2017
Fr + En
1455
12
121
Ja + En
1452
12
121
Guofeng VI TEST.2
Zh
En
857
12
71
Table Statistics of the test sets used in our experiments. "[S!" represents the number of sentences in each test set, "DD"" represents the number of documents, and |S|/|D] represents the average num- ber of sentences per document.
Setting
sCOMET
dCOMET
LICR-1
LICR-1; 1r
1
Sentence-level
83.78
6.55
80.27
88.78
2
T Shöri-Term Měmory
88,30
6.68
77.89
8744
3
Long-Terne Memory
84,48
6.69
78.77
88.01
4
Record
84.11
6.60
81.33
89.33
5
Summary
84.51
6.73
79.73
90.70
6
2 LLongTTerm Memory
84.54
6.67
79.23
89.44
7
2 Record
84.45
6.70
82.37
92.54
8
3 Source Summary
84.66
6.68
76.09
91.23
3 Target Summary
84.70
6.72
82.14
92.86
10
3 Billingual Summaar
84.72
6.74
82.49
93.60
1r
To Record (DELTA)
84.70
6.72
86.44
95.25
Table 10: More detailed results of the ablation study.
Figure 5 and Figure 6 respectively. Note that the prompts for the summary components are written in their respective source or target language avoid off-target issues. The prompt template for the Memory Retriever is shown in Figure The prompt template for the Document Translator is illustrated Figure g All the retrieved information from the multi-level memory components formatted into this prompt to assist the translation process.
D DETAILED RESULTS OF THE MAIN EXPERIMENT
The scores for the En = Zh, De, Fr, Ja translation directions are presented in Table 13 while the scores for the Zh, De, Fr, Ja En are shown Table 14
DELTA achieves improvements in both translation consistency, as indicated by the LTCR-1 and LTCR-I, metrics, and translation quality, as indicated by the SCOMET and dCOMET metrics, across most translation directions compared to several baselines. The Qwen models show signif- icant enhancements in SCOMET and dCOMET scores, demonstrating that our approach provides strong reinforeement for the document translation quality of these open-source models.
The quality improvements are most pronounced in the Ja + En directions across all backbone mod- els, likely due to their modest baseline capabilities for these language pairs, which approach effectively enhances. Translation consistency in the Zh + En directions benefits most from our ap- proach, the distinct character sets of Chinese and English pose challenges in maintaining proper noun consistency, highlighting the effectiveness of our method.
When applying GPT-3. 3-TTrob the backbone model, DELTA outperforms translation-speciillied baselines, such as L.LB-3.3B 3B and GoogleTrans rTans, across most languages, demonstrating the promis- ing capabilities of LLM-based autonomous agents in document translation.
E DETAILED RESULTS OF THE ABLATION STUDY
18

---

Published as a conference paper at ICLR 2025
Short. Window Size
sCOMET
dCOMET
LTCR-1
LTCR-1:
84.51
6.71
87.42
95.36
84.70
6.72
86.44
95.25
5
84.65
6.74
87.42
95.03
Long. Window Size
SCOMET
dCOMET
LTCR-1
LTCR-1; 1r
84.61
6.74
86.67
95.67
84.70
6.72
86.44
95.25
30
84.71
6.73
84.25
94.86
Growing
84.68
6.71
85.57
93.96
Table 11: Effect of the Short- Meort-Termm and Long-Term Term window size.
System sCOMET
dCOMET
LTCR-1
LTCR-I,
Sentence
87.33
7.04
78.65
96.07
Context
87.87
7.19
78.09
96.07
DELTA 87.96
7.20
82.68
96.65
Table 12: Evaluation results on the Lt + En low-resource test set.
Effect of Each Memory Component The ablation results for each individual memory component are presented in Table 10 The backbone model used in our experiments is GPT-3.5-Tune Turbb,, evalu- ated the IWSLT2017 En = Zh test set. We can observe that Short-Term Memory enhances translation quality but negatively impacts consistency due to disruptions caused by the short-span context introduces (Model 2). 2) Similarly, Long-Term Memory improves translation quality but also leads to a decline in consistency (Model 3). However, these two modules complement each other, as their integration leverages information from different spans, resulting in further quality en- hancements while mitigating consistency issues (Model 6). 3) Proper Noun Records contribute to both translation quality and consistency, as improved accuracy in proper noun translation directly enhances overall quality (Model 4). 4) Bilingual Summary also positively impacts translation qual- ity by introducing the document's 'ss main idea, which guides the generation of target sentences more effectively (Model 5).
Each module independently contributes to performance improvements. However, none surpass the combination of all modules working together. We also conducted additional comparative test be- tween DelTA and the context method augmented with Proper Noun Records. The results, presented Model 7 in Table I00 reveal a large performance gap favoring DelTA. We attribute this to the DelTA architecture's 's ability to incorporate information across varying granularities and scales, en- abling a synergistic enhancement of both translation quality and consistency within framework.
Effect of Hyper-Parameters, We perform experiments evaluate the impact of different win- dow sizes for Short-Term and Long-Term Memory (i.e. k and / introduced in $4). The results are presented in Table II Increasing the window size of Short-Term Memory results in slight im- provement in consistency but incurs a substantial additional computational cost, as two sentence pairs extend each sentence's translation prompt. Given the trade-off between computational cost, translation quality, and consistency, we opted to employ a Short-Term window size of 3. Further increasing the Long-Term Memory window does not yield significant benefits. To address the idea that the model might require longer context windows in the later stages of document translation, we also experimented with dynamic window setting, where the window size increases by one sentence pair for every eight sentences translated. However, this approach does not lead to performance im- provements. We attribute this to the Bilingual Summary component, which effectively captures key information from more distant contexts. The iterative process of summary generation and merging inherently functions as a form of context window growth, rendering additional adjustments to the Long-Term Memory window unnecessary.
19

---

Published as a conference paper at ICLR 2025
F PERFORMANCE ON LOW-RESOUCE LANGUAGES
We evaluate how well DELTA scales to low-resource languages. Considering that the spaCy NLP tool used for evaluation does not support many low-resource languages, we select Lt En as the language pair to test. We randomly sample a document (with 556 sentences) from Europarl v9 training se 13 as our evaluation set, and employ GPT-3.5 T-Tbb as the backbone model. The results shown Table 12 indicating that DELTA also performs well on the low-resource language pair.
Prompt for Proper Noun Extractor
You English-Chinese bilingual expert. Given English source sentence with its Chinese translation, you need to annotate all the proper nouns in the English source sentence and their corresponding translations the Chinese translation sentence. Here are some examples for you:
Example 1:
<English source>> NASA's Kepler mission has discovered thousands of potential planets around other stars, indicating that Earth is but of billions of planets in our galaxy.
<Chinese translation>
E fir f (I 41
#1
T (EAT
<Proper nouns>> "NASA" - " E*"
"Kepler" ">F $J", "Farth" -Hb
Example 2:
<English source> had just driven home, was around midnight in the dead of Montreal winter, I had been visiting my friend, Jeff, across town, and the thermometer the front porch read minus 40 degrees and don't bother asking if that's Celsius or Fahrenheit, minus 40 is where the two scales meet it was very cold. <Chinese translation>> A
J E EJ
, ン ル
i
- + +
IT BI
400
+ 400
Proper nouns> "Montreal" "N/A", "Jeff" * "Celsius" "
" "Fahrenheit"
Example 3:
<English source> To make the case to the National Health Service that more resources were needed for autistic children and their families, Lorna and her colleague Judith Gould decided do something that should have been done 30 years earlier.
<Chinese translation>> "TT t E (R
#
Lorna A
Proper nouns> "National Health Service"
x St. "Loma" "N/A",
"Judith Gould" - 15"
If there isn't any proper noun in the sentence, just answer with "N/A". Now annotate all the proper nouns in the following sentence pair:
<Egglish source>> {SOURCE SENTENCE)
Chinese translation (TARGET SENTENCE)
<Proper nouns>>
Figure 4: Prompt template for the Proper Noun Extractor. We provide several few-shot exemplars preceding the current input. This template is designed for the En Zh translation direction. For other translation directions, adjust the corresponding content to match the specific languages.
20

---

Published as a conference paper at ICLR 2025
Prompt for Source Summary Writer (Segment Summary Generation)
Below is a paragraph. Please provide a summary of this paragraph, including the main contents of these sentences, the overall domain, style and tone of it, while preserving key information as much as possible. Paragraph: (SOURCE SEGMENT) Summary:
Prompt Template for Source Summary Writer (Summary Merging)
Below are the summaries of two adjacent paragraphs. Please merge them into single summary, retaining as much key information as possible and ensuring that information about the domain, style, and tone are preserved.
Summary I: {SUMMARY 1} Summary 2: (SUMMARY 2 Merged summary:
Figure 5: Prompt template for Source Summary Writer.
Prompt Template for Target Summary Writer (Segment Summary Generation)
Below is a paragraph. Please provide a summary of this paragraph, while preserving key information as much as possible.
Paragraph: {SOURCE BOOURCE.SEGMENTT SEGMENT] Summary:
Prompt Template for Target Summary Writer (Summary Merging)
Below are the summaries of two adjacent paragraphs. Please merge them into single summary, retaining much key information as possible. Summary 1: {SUMMARY_1}
Summary 2: {SUMMARY 22
Merged summary:
Figure 6: Prompt template for Target Summary Writer. We write the prompt in the target language to better align with the agent profile of the monolingual summary writer and reduce the off-target issues. For demonstration purposes, the prompts provided here are written in English.
21

---

Published as a conference paper at ICLR 2025
Prompt Template for Long- Term Memory Retriever
You are linguistic expert. Given a list sentences and a query, your task is to find the (TOP_NUM) sentences in the list that are most relevant the request.
Sentence list:
{SRC1}
{SRC2)
Query: {QUERY)
Note that you should only respond with list containing the numbers of these (TOP_NUM) sentences. For example, if you choose sentences 15, 16, and 19 your answer, your re- sponse should be "[15, 16, 19]".
Figure 7: Prompt template for Long-Term Memory Retriever.
Prompt Template for Document Translator
bilingual expert, translating a very long {SRC LANG) document. Given the summary of the preceding text in both (SRC LANG) and (TGT LANG), the historical translation of some proper nouns, source and translation texts preceding the current sentence, as well as some relevant translation instances from the preceding text, translate the current {SRC_LAN Please ensure that the translations of proper nouns the source sentence are consistent with their historical translation, and the translation style remains consistent well.
You are an (OCLLANG)-(TGTLLAAN
source sentence into (TGT LANG).
Summaries:
<SSCCLANG) summary> SRC_SUMMARY
<AGTLLANG) summary> SUMMAYYY
Historical translations of proper nouns:
(HISTORY)
Preceding texts:
<SRRCLANC) LANG) text>>
< <TGT LANG) text>>
(TGT CONTEXT)
Relevant instances: |RELEVNNTINSTANNEE
Now translate the following (SRC.LANC source sentence to (TGT_LANC <SSRC.LANG) source>> (SOURCE)
<[GT_LANG) translation>>
Figure 8: Prompt template for Document Translator. Involved proper nouns and their correspond- ing translations formatted into the "HISTORY" field. Source and target sentences from Short- Term Memory are concatenated and formatted into the "SRC.CONTEXT" and "TGT.CONTEXT" fields. respectively. Retrieved instances from Long-Term Memory are formatted into the "RELE- VANT INSTANCES" field as source-target pairs.
22

---

Published as a conference paper at ICLR 2025
System
sCOMET
dCOMET
LTCR-1
LTCR-I,
sCOMET
sCOMET
LTCR-1
LICR-I:
En 7h
En
De
NLLB
76.81
6.20
71.68
84.9
84.15
6,64
91.76
99.61
GOOGLE
78.46
5.76
89.45
92.36
80.23
5.78
93.55
99.19
GPT-3.5-Turba
Sentence
83.78
6.55
80.27
88.78
84.97
6.71
92.06
98.81
Context
84.50
6.68
77.89
87.41
85.12
6.74
93.70
99.21
Doc2Doc
6.29
82.04
94.29
6.81
88.89
97.94
Ours
84.70
6.72
86.44
95.25
85.37
6.78
93.46
99.23
GPT-40-miní
Sentence
82.13
6,43
78.04
91.89
81,41
6.39
90.70
98.84
Context
84.36
6.68
78.95
93.42
84.83
6.70
92.66
99.61
Doc2Doc
6.60
82.33
88.35
6.90
91.05
99.22
Ours
84,94
6.81
85.52
91.72
85.47
6.79
92.19
100.0
Qwen-7B-Instruce
Sentence
83,05
6.51
77.78
83.84
76.24
5.54
82.11
90,65
Context
83.51
6.67
77.29
87.12
76.67
5.56
85.66
92.45
Doc2Doc
6.16
81.85
91.11
5,25
88.60
97.93
Ours
83.98
6.70
80.13
90.55
76.84
5,56
87.40
96.75
Qwen-726 Ins-Inntrut
Sentence
77.76
5.88
73.13
83.96
78.98
6.13
92.18
98.35
Context
81.69
6.24
78.01
86.17
80.96
6.43
89.64
96.81
Doc2Doc
6.22
74.59
78.38
6.66
88.28
98.44
Ours
84.76
6.70
81.56
89.72
84.29
6.70
90.48
98.41
En + Fr
En + Ja
NLLB
85.35
6.23
87.84
89.86
82.12
6.37
46.94
53.06
GOOGLE
82.21
5.53
88.61
91.46
80.74
6.23
$3.92
55.88
9TT-3.5-Turba
Sentence
85.84
6.18
83.55
88.49
84.61
6.89
52.34
55.14
Context
86.49
6.27
83.06
89.25
85.50
7.09
54.72
56.60
Doc2Doc
6.28
92.28
94.63
7.10
53.26
58.70
Ours
86.48
88.96
94.16
85.76
7.13
62.96
66.67
GPT-4o-mini
Sentence
80.79
5.82
88,89
90.91
81.72
6.74
56.73
58,65
Context
85.10
6.14
90.52
92.81
84.84
7.09
$7.89
62.11
Doc2Doc
91.00
94.00
7.25
57.78
60.00
Ours
86.38
6.28
90.94
93.85
86.61
7.32
58.54
59.76
Twen-78-Instreee
Inst
Sentence
80.61
5.47
82.53
84.93
80.21
6.30
53.21
58.72
Context
81.31
89.00
90.72
81.85
6.53
66.41
71.09
Doc2Doc
5.29
87.88
93.56
6.63
50.96
55.77
Ours
81.02
5,46
88.66
91.41
82.23
6.57
64.18
72.39
Owen-728-4
728-Instruct
Sentence
81.02
5.75
90.00
92.33
76.34
6.11
62.86
65.71
Context
84.03
6.06
87.22
89.46
76.48
6.14
61.68
69.16
Doc2Doc
6.25
89.25
91.86
6.66
42.20
45.87
Ours
85.76
6.28
92.08
94.39
85.13
6.94
62.50
70.83
Table 13: Detailed results of our experiments in En + Xx directions.
23

---

Published as a conference paper at ICLR 2025
System
sCOMET
dCOMET
LTCR-1
LTCR-I,
sCOMET
dCOMET
LTCR-1
LICR-I:
7h En
De
En
NLLB
82.14
7.01
75.31
88.27
85.63
7.23
95.98
98.85
GOOGLE
78.06
5.68
72.89
86.14
82.31
6.47
96.53
98.27
GPT-3.5-Turba
Sentence
83.34
7.17
73.99
86.71
85,92
7.26
98.88
100.0
Context
83.88
7.29
76.92
90.53
86.10
7.30
98.30
100.0
Doc2Doc
7.08
76.77
88.39
7.16
98.24
98.82
Ours
83.88
7.30
80.00
93.53
86.14
7.30
98.33
100.0
GPT-40-miní
Sentence
83.55
7.24
71.93
84.80
85.11
7.17
98.20
100.0
Context
83.96
7.35
78.24
91.18
86.12
7.27
98.88
100.0
Doc2Doc
7.15
79.62
92.36
7.19
95.24
97.62
Ours
84.10
7.47
79.41
94.71
86.61
7.31
98.32
100.0
Qwen-7B-Instruch
Sentence
79.44
6.76
71.17
87.73
77.20
6.75
93.25
95,09
Context
82.62
7.04
69.70
83.64
84.52
7.08
98.80
99.40
Doc2Doc
6.53
82.79
92.62
6,88
98.67
99.33
Ours
82.83
7.09
76.47
92.35
84.61
7.05
98.25
100.0
728 Instruct
1120-Instuce
Sentence
80.35
6.95
73.49
84.34
81.17
6,94
97.19
100.0
Context
80.11
6.93
74.25
86.23
84.81
7.24
98.31
99.44
Doc2Doc
6.94
68.21
80.13
7.08
97.59
98.19
Ours
84.51
7.40
83.93
94.05
86.17
7.34
98.29
100.0
Fr + En
Ja
En
NLLB
87.59
6.79
93.56
97.42
81.02
6.90
51.27
78.48
GOOGLE
84.64
6.19
95.63
96.83
75.67
5.50
60.67
82.00
9-T-3.5-Turbo
Sentence
87.60
6.78
94.94
97.89
81.00
6.98
60.12
82,82
Context
88.03
6.84
94.96
97.90
81.85
7.17
69.94
92.64
Doc2Doc
6.78
94.78
97.39
6.80
70.90
87.31
Ours
88.02
96.17
98.30
81.76
7.13
71.60
93.21
GPT-4o-mini
Sentence
87.32
6.77
94.42
97.85
80.04
6.76
61.11
82.72
Context
87.72
6.81
94.85
97.42
82.00
7.17
65.62
88.75
Doc2Doc
94.42
98.28
6.86
64.71
85,29
Ours
88.13
6.90
96.58
98.72
82.20
7.29
66.67
90,12
Qwen- Twen-78-Instrude
Instruct
Sentence
81.21
6.27
87,90
95.56
70.56
6.15
$3.25
73.38
Context
86.55
94.19
97.51
78.68
6.59
63.23
89.68
Doc2Doc
6.57
87.00
97.76
6.39
71.67
85.00
Ours
86.40
6.56
91.20
92.80
79.60
6,66
62.26
88,05
Owen-726- -enm-72---ntuucc
Sentence
82.67
6.38
95.44
98.34
77.94
6.63
62.89
85.53
Context
87.02
6.76
93.25
97.89
81.13
7.02
65.64
85.28
Doc2Doc
6.73
94.67
97.78
6.74
71.53
86.86
Ours
88.01
6.87
95.78
98.73
82.06
7.24
68.12
93.12
Table 14: Detailed results of our experiments in Xx = En directions.
24


Reviews:
[{"review_id": "1YAX0RwVcr", "reviewer": "Reviewer_bC5N", "summary": "This paper presents a methodology that improves translation consistency by using an agent-based approach. DelTA utilizes a modular, multi-level memory system, comprising Proper Noun Records, Bilingual Summary, Long-Term Memory, and Short-Term Memory, which store and manage information at different scales, with auxiliary LLM-driven components ensuring efficient information retrieval and updates.", "questions": "* How well does this scale to extremely low-resource languages?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* This paper effectively addresses a significant challenge in the field, presenting meaningful improvements and contributions.\n\n*The experimental design is comprehensive and well-executed, providing robust results.\n\n*The presentation is clear, concise, and well-organized, facilitating easy understanding of the research.", "weaknesses": "* As authors point out in limitations, efficiency of the method", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nLiKOjuvJf", "reviewer": "Reviewer_oGf7", "summary": "The paper proposes DelTA, a document-level translation agent that can translate documents or long context text with better translation quality than other methods. The approach consists of a series of prompts that are used to encode information about proper nouns, immediate context, larger context, summaries in the source and target language. Experiments are carried out in the IWSLT2017 benchmark for 4 language pairs and evaluated for consistency of translation terms and translation quality using reference-based translation evaluation methods.", "questions": "* Consider using the WMT24 shared task benchmark test sets that are going to be released. They provide document-level test sets. In 2023 the organizers released test sets only for English-German but for this year more language pairs will be made available.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The idea of the paper is simple and straightforward: provide more context about the translation task by presenting this information in the prompt of an LLM. The context is varied: sentences occurring in the immediate before the sentence being translated, context that is longer with sentences much before the current sentence, a summary of the source text, a summary of the target text and proper nouns occurring in the text with their respective translations.", "weaknesses": "The evaluation was performed only with automatic metrics such as sentence-level and document-level COMET. It would be interesting to understand if the performance reported holds with evaluation performed by translators in terms of quality and consistency as both implementations of COMET are known to be weak proxies for translation quality beyond the sentence-level, failing to capture consistency, coherence and other discourse-related linguistic dimensions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "amm3Zb4QO9", "reviewer": "Reviewer_h7o9", "summary": "This paper proposes a LLM-based, document-level translation agent (called Delta). This agent performs document-level translation on a sentence-by-sentence basis, and consists of auxiliary \"multi-level memory\" components, which are updated after translation of each sentence. These memory components include Proper Noun Records, Bilingual Summary, and Long-Term/Short-Term Memory. The information stored in all of these components is retrieved by prompting LLMs, where the prompt for each component is targeted towards its desired function (e.g., the Bilingual Summary component asks the LLM to generate a summary given a chunk of sentences). During translation of each sentence within a document, after updating all of these memory components, the Delta system aggregates their stored information into a single prompt. This process is continued iteratively until the entire document is translated. The paper shows that the Delta system outperforms other document-level MT systems on the IWSLT2017 and Guofeng datasets, and also presents ablations to understand the contribution of each memory component to the overall performance of Delta.", "questions": "1. For the summary writers, why isn't the summary for chunk m conditioned on the summary for chunk m-1 (which would allow the summary to take longer context into account)?\n2. Did you investigate the effect of adjusting the numbers (k, l) of sentences to use as short- and long-term memory throughout the translation process? It seems likely that the model would probably need more context at later stages (when more sentences have already been seen), versus at the beginning of translating a long document.\n3. In Figure 2, why does consistency drop so much for the en-xx 11-20 sentences bucket (and then increase again for longer sentence-wise distances) across all models?\n4. The Workshop on Machine Translation (WMT, https://machinetranslate.org/wmt) also has document-level test sets, which are canonically used for MT evaluation in the literature. Why were evals presented on IWSLT2017 and Guofeng (relatively unknown datasets), but not on these standard test sets? Do the test sets used require long context to produce correct translations?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Document-level translation is currently one of the most important and underexplored research areas in the field of machine translation (MT). This paper clearly presents the motivation for their work, and summarizes the importance of document-level MT.\n2. The Delta system presented in the paper is easy to implement, since it just requires prompting of off-the-shelf LLMs. The motivation for each of the modules in the Delta system is clear.\n3. The experimental setup is well-written and easy to follow. The results are comprehensive (covering multiple LLMs and language pairs), and comparisons against all essential document-level MT baselines are presented.", "weaknesses": "1. The paper mentions that inconsistency is one of the major challenges during document-level translation. However, the only type of inconsistency they measure and report is Proper Noun Inconsistency, which is a surface-level type of inconsistency (can the model consistently copy a fixed string?) that doesn't require the model to reason across long contexts. The results on Proper Noun Inconsistency are used to make strong claims about their Delta system's overall improvement in consistency relative to baselines, but these claims are not adequately supported with evidence.\n2. The paper focuses on the comparison to \"Doc2Doc\" (comparing memory costs, etc), but from the reported results, \"Context\" is consistently better than \"Doc2Doc\". There is no significance testing for the results in the paper (e.g., Tables 2 and 3), and in terms of translation quality (as measured by sCOMET and dCOMET), the performance of the \"Context\" and \"Delta\" models looks quite close, with \"Context\" outperforming \"Delta\" according to dCOMET for the Qwen2-7B-Instruct model in Table 2. In Table 3, the quality-based performance of \"Context\" and \"Delta\" is even closer. (And it is not surprising that Delta outperforms in proper noun consistency, since the proper noun extractor module in Delta provides a list of all proper nouns seen. If the Context model were enhanced with this module, it seems likely that its performance on the proper noun consistency metric would also correspondingly improve.)\n3. The Delta system is comprised of 5 auxiliary modules. The ablation to isolate the effects of each individual module in Figure 4 shows very small differences in quality (as measured by sCOMET and dCOMET) across these models (again, with no significance testing). The recipe (combination of modules) proposed by the authors works reasonably well based on their results, but the paper provides limited insight into the relative contribution of each of these modules to improving performance. For instance, does short-term or long-term memory help more? Is the Memory Retriever step for long-term memory necessary, and how much does this step improve performance over just using the (l sentences in) long-term memory?\nSee additional questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M1kmnvbARO", "reviewer": "Reviewer_gPC6", "summary": "The paper introduces DELTA, a Document-level Translation Agent that uses a multi-level memory architecture to improve translation consistency and quality in document-level machine translation. The agent employs four memory components (Proper Noun Records, Bilingual Summary, Long-Term Memory, and Short-Term Memory) that store and retrieve information across different granularities. The approach shows impressive improvements in translation consistency and quality across multiple language pairs and different LLM backbones. The work is evaluated extensively using both established metrics and novel consistency measures.", "questions": "1. How does the approach perform compared to long-context LLMs? Given recent advances in context window sizes, this seems particularly relevant.\n\n2. Will the code be released? This would be valuable for reproducibility and adoption.\n\n3. Can you provide a more detailed efficiency analysis, including:\n   - Total wall-time comparison between DELTA and Doc2Doc approaches\n   - FLOPS comparison for open-weight models\n   - Analysis of the number of LLM calls required\n\n4. How does DELTA handle context-dependent translations where meaning depends on broader document context? While pronoun translation is briefly discussed, a more comprehensive analysis (see https://aclanthology.org/2023.wmt-1.42/) would be valuable.\n\n5. Have you considered evaluating against DeepL, given its explicit support for document-level translation?\n\n6. Could you provide statistical significance tests for the reported improvements?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Originality**:\n- The multi-level memory architecture represents a novel approach to document translation. While memory architectures have been used in LLM agents before, the specific design with four complementary components (Proper Noun Records, Bilingual Summary, Long/Short-Term Memory) targeting different aspects of document translation is innovative.\n- The approach of using an LLM-based agent for document translation, while maintaining sentence-level alignment, is creative and practical.\n- The introduction of LTCR-1 and LTCR-1f metrics for evaluating translation consistency is a useful contribution to the field.\n\n**Quality**:\n- The experimental methodology is comprehensive, evaluating across 8 language pairs and multiple LLM backbones (both open and closed source).\n- The ablation studies effectively demonstrate the contribution of each memory component.\n- The memory efficiency analysis is well-done, with clear comparisons showing the advantages over Doc2Doc approaches.\n- The evaluation considers both translation quality (sCOMET, dCOMET) and consistency (LTCR metrics).\n\n**Clarity**:\n- The paper is logically structured, moving from motivation to design to evaluation.\n- The multi-level memory architecture is well-explained with clear diagrams and algorithms.\n- The experimental results are presented systematically with detailed tables and analyses.\n- The limitations and future work are honestly discussed.\n\n**Significance**:\n- The work addresses a real and important challenge in document translation - maintaining consistency while handling long documents.\n- The approach is practical and implementable, with clear memory efficiency advantages.\n- The method shows strong improvements across different languages and models, suggesting broad applicability.\n- The memory architecture could potentially be adapted for other document-level NLP tasks, as demonstrated by the query-based summarization results.", "weaknesses": "1. Efficiency analysis needs more detail:\n   - While memory efficiency is demonstrated, the paper claims cost-effectiveness without supporting evidence\n   - No comparison of total wall-time or FLOPS between approaches\n   - Limited discussion of computational overhead from multiple LLM calls\n\n2. Missing comparisons and baselines:\n   - No evaluation against long-context LLMs (e.g., Claude, Gemini) which could be particularly relevant for document translation\n   - DeepL, which explicitly supports document-level translation, would be a valuable baseline, and a more relevant choice than Google.\n   - Limited discussion of Doc2Doc approach limitations in related work\n\n3. Incomplete motivation and analysis:\n   - Does not address context-dependent translation as a key challenge (see https://aclanthology.org/2023.wmt-1.42/)\n   - Claims of \"significant\" improvements lack statistical significance testing\n   - Claims about undertranslation issues and translation quality decline (L177-178) should be supported with citations to relevant work (e.g.,  https://arxiv.org/abs/2401.06468, https://aclanthology.org/2024.acl-long.336/)\n \n4. Technical details and reproducibility:\n   - No mention of code release plans", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)