DelTA: An Online Document-Level Translation Agent Based on Multi-Level Memory

## Abstract

Abstract Large language models (LLMs) have achieved reasonable quality improvements in machine translation (MT).
However, most current research on MT-LLMs still faces significant challenges in maintaining translation consistency and accuracy when processing entire documents.
In this paper, we introduce DelTA , a D ocument-lev EL T ranslation A gent designed to overcome these limitations. DelTA features a multi-level memory structure that stores information across various granularities and spans, including Proper Noun Records, Bilingual Summary, Long-Term Memory, and Short-Term Memory, which are continuously retrieved and updated by auxiliary LLM-based components.
Experimental results indicate that DelTA significantly outperforms strong baselines in terms of translation consistency and quality across four open/closed-source LLMs and two representative document translation datasets, achieving an increase in consistency scores by up to 4.58 percentage points and in COMET scores by up to 3.16 points on average. DelTA employs a sentence-by-sentence translation strategy, ensuring no sentence omissions and offering a memory-efficient solution compared to the mainstream method.
Furthermore, DelTA improves pronoun translation accuracy, and the summary component of the agent also shows promise as a tool for query-based summarization tasks.
We release our code and data at https://github.com/YutongWang1216/DocMTAgent .

## 1 Introduction

Large language models (LLMs) such as GPT-4 (OpenAI, ) have recently demonstrated reasonable performance on the machine translation (MT) task within the natural language processing domain (Garcia & Firat, ; Hendy et al., ; Zhang et al., ; Siu, ; Jiao et al., ).
Numerous studies have been carried out to further unleash LLMs’ potential for MT (Ghazvininejad et al., ; Peng et al., ; Zeng et al., ; He et al., ; Wang et al., ).
However, the majority of these researches mainly focus on sentence-level translation, operating under the strong assumption that source sentences are independent of one another.
This isolated approach may fail to model the discourse structure and overlook the coherence in continuous document texts (Scarton & Specia, ; Bawden et al., ).

Document-level machine translation (DocMT) systems have been receiving growing focus in recent years, which involves the whole document or some part of it to capture more context information to guide the translation process (Kim et al., ; Maruf et al., ).
Researchers find that modeling discourse phenomena (Bawden et al., ) while translating the whole document helps increase the coherence and consistency in the generated translation (Maruf & Haffari, ; Wang et al., ; Zhang et al., ; Tan et al., ).
Recently, a few studies have proposed to introduce LLMs to the DocMT task, utilizing their inherent context information modeling and long text processing capabilities (Wang et al., ; Wu & Hu, ; Wu et al., ).
However, existing DocMT-LLMs still suffer from critical issues such as occasional content omissions and terminology translation inconsistency (Karpinska & Iyyer, ).
These issues seriously affect the reliability of the developed system, especially when accurate document translations are required.

LLM-based autonomous agents equipped with specially designed memory components can efficiently store and retrieve key information embedded in the environment.
These data assist the inference process of LLMs, facilitating the handling of complex tasks and environments through self-directed planning and actions (Wang et al., ; ; Park et al., ; Lee et al., ).
Inspired by this, we propose DelTA, an online Document-levEL Translation Agent based on multi-level memory components.
Specifically, we store information in four memory components: Proper Noun Records, Bilingual Summary, Long-Term Memory, and Short-term Memory, and utilize LLMs to update and retrieve them.
Proper Noun Records maintain a repository of previously encountered proper nouns and their initial translations within the document, ensuring consistency by reusing the same translation for each subsequent occurrence of the same proper noun.
The Bilingual Summary contains summaries of both the source and target texts, capturing the core meanings and genre characteristics of the documents to enhance translation coherence.
Long-Term Memory and Short-Term Memory store contextual sentences over broader and narrower scopes, respectively. Long-Term Memory is accessed by LLMs to retrieve sentences most relevant to the current source sentence, while Short-Term Memory provides instant context to support the translation process.
During translation, sentence pairs are drawn from the Long-Term and Short-Term memory as the exemplars for few-shot learning demonstration, and the proper noun translation records and bilingual summaries are also integrated into the prompt for DocMT-LLMs as auxiliary information.

Experimental results indicate that DelTA achieves improvements in both translation consistency and quality.
For translation consistency, DelTA achieves an average improvement of 4.36 percentage points across four translation directions from English and 4.58 percentage points across four directions into English.
In terms of translation quality, DelTA yields an average improvement of 3.14 COMET points for four translation directions from English and 3.16 COMET points for four directions into English.
Moreover, DelTA translate documents in a sentence-by-sentence manner (following an online approach) to avoid content omissions, ensuring sentence-level alignment of target documents with source documents.
This manner also prevents memory bloat caused by data accumulation, making it more suitable for practical application scenarios.

Our main contributions are summarized as follows:

- •
We develop DelTA, an online DocMT agent employing a multi-level memory structure, which stores information across different granularities and spans.
- •
We demonstrate that DelTA substantially improves the consistency and quality of document translations. Additionally, the summary component of DelTA can function as an independent tool for query-based summarization tasks.
- •
We certificate that the sentence-wise translation approach employed by DelTA incurs a lower memory cost compared to existing document translation methods.
- •
We observe that DelTA is particularly effective in maintaining translation consistency over expended spans. Moreover, it enhances the pronoun translation accuracy in the document.

## 2 Related Work

#### Document-Level Machine Translation

In recent years, studies on DocMT have achieved rich results (Kim et al., ; Maruf et al., ; Fernandes et al., ).
These studies can be separated into two categories.
Studies of the first group employ a document-to-sentence (Doc2Sent) approach, where the source-side context sentences are encoded to generate the current target sentence (Wang et al., ; Tan et al., ; Lyu et al., ).
However, these approaches suffer from limitations caused by separated encoding modules of the current sentences and their context (Sun et al., ; Bao et al., ), as well as the failure to utilize target-side context (Li et al., ).
Studies of the second group employ a document-to-document (Doc2Doc) approach, where the translation unit is extended from a single sentence to multiple sentences (Zhang et al., ; Liu et al., ; Lupo et al., ; Bao et al., ; Li et al., ).

#### Autonomous Agents

LLM-based autonomous agents have recently achieved remarkable performance in various NLP tasks.
Park et al. (); Wang et al. (); Lee et al. () deal with long-context understanding and processing tasks by introducing carefully designed memory and retrieval workflows.
Xu et al. (); Wang et al. (); Feng et al. () prompt LLMs to evaluate their own outputs and conduct refinement accordingly to improve the quality of the outputs.
Li et al. (); Liang et al. (); Li et al. (); Wu et al. () enhance the performance of LLMs on specific tasks by multi-agent interaction.

Our method in this paper represents a Doc2Sent approach implemented through an LLM-based automatic agent.
Instead of simply encoding source-side context to generate target sentences, our method directs LLMs to retrieve key information across varying granularities and spans, and store this data in memory components.
During document translation, relevant information is incorporated into the prompts for DocMT-LLMs to assist the translation process.

## 3 Motivation

### 3.1 Main Challenges for DocMT-LLMs

Due to the maximum context limitation inherent in LLMs, translating a lengthy document in a single pass becomes unfeasible.
A conventional strategy involves segmenting the document into smaller translation windows and translating them sequentially.
In our study, we initially leverage the GPT-3.5-turbo-0125 model to translate the IWSLT2017 En $\Rightarrow$ Zh test set, comprising 12 documents sourced from TED talks.
We employ a window of size $l$ to facilitate document translation, where $l$ source sentences are simultaneously processed to generate $l$ hypothesis sentences.
Once all source sentences are translated, they are concatenated to form the complete target document.
The primary challenges associated with DocMT-LLMs arise from the following two aspects.

**Table 1: Translation results with different translation window sizes. “#Missing Sents” represents the number of missing target sentences in the translated document.**
| Window | LTCR-1 | $\text{LTCR-1}_{\text{f}}$ | #Missing Sents | sCOMET | dCOMET |
| --- | --- | --- | --- | --- | --- |
| 1 | 75.09 | 88.24 | 0 | 84.04 | 6.62 |
| 5 | 80.49 | 88.15 | 0 | 84.30 | 6.70 |
| 10 | 79.65 | 90.81 | 2 | 84.27 | 6.65 |
| 30 | 83.08 | 95.83 | 8 | 83.88 | 6.69 |
| 50 | 86.94 | 95.90 | 10 | 83.70 | 6.66 |

#### Translation Inconsistency

Given a source document $\bm{D}_{s}=(s_{1},s_{2},\ldots,s_{N})$ and its corresponding target document $\bm{D}_{t}=(t_{1},t_{2},\ldots,t_{N})$, if there exists a proper noun $p\in P$ ($P$ denotes the set of all proper nouns in $\bm{D}_{s}$, including names of people, locations, and organizations), and $p$ appears multiple times in $\bm{D}_{s}$, we expect that all occurrences of its translation in $\bm{D}_{t}$ should be consistent.

Lyu et al. () propose the Lexical Translation Consistency Ratio (LTCR), a metric that quantifies the proportion of consistent translation pairs among all proper noun translation pairs in the target document.
However, we argue that the translations of the proper nouns are supposed to not only maintain consistency throughout the document but also align their first appearance.
This consideration is particularly important for enhancing the reading experience of audiences.
Therefore, we introduce the LTCR-1 metric for the DocMT-LLMs, which calculates the proportion of proper noun translations that are consistent with the initial translation within the document:

$$ $\displaystyle\text{LTCR-1}(\bm{D}_{s},\bm{D}_{t})=\frac{\sum_{p\in P}{\sum_{i= 2}^{k_{p}}{\mathbb{1}(\mathcal{T}_{i}(p)=\mathcal{T}_{1}(p))}}}{\sum_{p\in P}{ (k_{p}-1)}}$ (1) $$

$\mathcal{T}_{i}(p)$ represents the $i$-th translation of $p$ in $\bm{D}_{t}$, and $k_{p}$ denotes the number of occurrences of $p$ in the document.
The indicator function $\mathbb{1}(\mathcal{T}_{i}(p)=\mathcal{T}_{1}(p))$ returns $1$ if the translations $\mathcal{T}_{i}(p)$ and $\mathcal{T}_{1}(p)$ are identical, and 0 0 otherwise.
The numerator is the number of times the proper nouns appear again and their translation remains the same as their first appearance, and the denominator represents the sum of all occurrences except the first one of all proper nouns.
To compute this metric, we initially annotate all proper nouns in the source document using spaCy(^1^11[https://spacy.io/](https://spacy.io/)).
Subsequently, we utilize the token align tool awesome-align (Dou & Neubig, )(^2^22[https://github.com/neulab/awesome-align/](https://github.com/neulab/awesome-align/)) to determine the translations of these proper nouns in the target document.
To mitigate the impact of errors from the alignment tool, we introduce a fuzzy match version of this metric, where two proper noun translations are considered consistent when one is a substring of the other:

$$ $\displaystyle\text{LTCR-1}_{\text{f}}(\bm{D}_{s},\bm{D}_{t})=\frac{\sum_{p\in P }{\sum_{i=2}^{k_{p}}{\mathbb{1}(\mathcal{T}_{i}(p)\subseteq\mathcal{T}_{1}(p) \lor\mathcal{T}_{1}(p)\subseteq\mathcal{T}_{i}(p))}}}{\sum_{p\in P}{(k_{p}-1)}}$ (2) $$

As shown in Table 1, translating every sentence separately (window size = 1) causes poor translation consistency.
An example is illustrated in Appendix A.
Increasing the window size consistently leads to higher scores across all three consistency metrics.
This suggests that when more sentences are processed within a single translation pass, the LLM is better able to model discourse phenomena and maintain consistent translation of proper nouns throughout the document.
However, due to the inherent limitations in the context length of LLMs, resolving translation inconsistencies cannot be achieved solely by indefinitely expanding the window size.

#### Translation Inaccuracy

When employing a large window size for document translation, LLMs tend to process the input source sentences as cohesive documents rather than as individual sentences.
As a result, the model prioritizes maintaining the general meaning of the text and loses track of the detailed information in each sentence.
This can lead to undertranslation issues and a decline in translation quality.
We utilize two neural metrics to assess the quality of document translation.
The first is the sentence-level COMET (sCOMET) score(^3^33[https://github.com/Unbabel/COMET/](https://github.com/Unbabel/COMET/)), for which we utilize the model Unbabel/wmt22-comet-da to obtain the scores.
The second metric is the document-level COMET (dCOMET) score(^4^44[https://github.com/amazon-science/doc-mt-metrics/](https://github.com/amazon-science/doc-mt-metrics/)) proposed by Vernikos et al. (), for which we use wmt21-comet-qe-mqm(^5^55[https://unbabel-experimental-models.s3.amazonaws.com/comet/wmt21/wmt21-comet-qe-mqm.tar.gz](https://unbabel-experimental-models.s3.amazonaws.com/comet/wmt21/wmt21-comet-qe-mqm.tar.gz)) to derive reference-free scores.
In calculating this document-level metric, the model encodes previous sentences as context rather than encoding only the hypothesis, making this approach more accurate for evaluating document translations.

As illustrated in Table 1, an increase in window size correlates with a higher tendency for the LLM to omit sentences from the source document, resulting in missing translations in the target document.
An example of this undertranslation issue is presented in Appendix A.
Additionally, quality metrics such as sCOMET and dCOMET do not demonstrate a consistent improvement with larger translation windows.
Therefore, we conclude that translating documents in batches of several sentences at a time may introduce translation inaccuracy issues.
These concerns are particularly significant in contexts where precise translations are essential, such as in technical manuals or official documents.

### 3.2 Why Using A Doc2Sent Approach?

Previous experiments indicate that translating a document by processing multiple sentences at once may occasionally result in sentence omissions.
Although human translators often translate entire paragraphs simultaneously, which can also lead to occasional omissions, their underlying translation mechanism is fundamentally distinct from that of DocMT-LLMs.
DocMT-LLMs are prone to omitting source sentences due to hallucination issues or limited capabilities in handling long texts effectively.
Therefore, we argue that, at this moment, a Doc2Sent approach offers a more promising alternative for DocMT-LLMs to produce precise and high-quality document translations.
In our study, we provide LLMs with contextual information from the document while asking them to translate each source sentence separately.
Once all sentences are translated, we concatenate the target sentences to form the final target document.

## 4 DelTA : DocMT Agent Based on Multi-Level Memory

Considering the multi-granularity and multi-scale of key information in the document during translation, we introduce DelTA, an online DocMT agent. DelTA employs a multi-level memory stream that captures and preserves critical information encountered throughout the translation process.
This memory stream accommodates a wide range of perspectives, spanning from recent to historical, concrete to abstract, and coarse-grained to fine-grained details.
DelTA translate the source document in a sentence-by-sentence manner while updating its memory in real-time.
This approach addresses the context limitations of large language models and ensures the generation of a sentence-level aligned target document, thereby preserving both the quality and rigorousness of the translation.
The main framework of DelTA is illustrated in Figure 1, the algorithm of DelTA is detailed in Algorithm 1, and the prompts used for each module are given in Appendix C.

#### Proper Noun Records

The first level of the agent’s memory component we introduce is a dictionary called the Proper Noun Records $\mathcal{R}$ to store proper nouns $p$ in the document along with their translations upon first encounter $\mathcal{T}_{1}(p)$ within the document: $\mathcal{R}^{(i)}=\{(p,\mathcal{T}_{1}(p))\mid p\in s_{j},1\leq j<i\}$, where $\mathcal{R}^{(i)}$ represents the state of $\mathcal{R}$ before translating the $i$-th sentence, and the same applies to other components.
When translating the subsequent sentence $s_{i}$, the agent consults $\mathcal{R}^{(i)}$ to obtain all recorded proper nouns that are also contained in $s_{i}$: $\hat{\mathcal{R}}^{(i)}=\{(p,\mathcal{T}_{1}(p))\mid p\in s_{i},(p,\mathcal{T}
_{1}(p))\in\mathcal{R}^{(i)}\}$.

The Proper Noun Records are continuously updated by an LLM-based component of the agent known as the Proper Noun Extractor $\mathcal{L}_{\text{Extract}}$.
After each sentence is translated, the LLM is asked to extract newly encountered proper nouns from the source sentence and their corresponding translations from the target sentence $\mathcal{L}_{\text{Extract}}(s_{i},t_{i})=\{(p,\mathcal{T}_{j}(p))\mid p\in s_
{i},\mathcal{T}_{j}(p)\in t_{i},\forall(p^{\prime},\mathcal{T}_{k}(p))\in
\mathcal{R}^{(i)},p\neq p^{\prime}\}$ and add them to $\mathcal{R}$.

Figure: Figure 1: Framework of DelTA. The modules outlined with dashed lines represent the multi-level memory components, while those outlined with solid lines denote the LLM-based components. Memories closer to the top are more global, abstract, and densely packed with information. During translation, memory information is retrieved and incorporated into the translator LLM’s prompt. After the translation of each sentence, the LLM-based components extract key information from both the source and target documents and update the multi-level memory components.
Refer to caption: x1.png

#### Bilingual Summary

Unlike previous studies (Wang et al., ; Lee et al., ), our research implements a bilingual summary approach as the second level of the agent’s memory component to address the challenges of translating documents with extensive texts on both the source and target sides.
We maintain a pair of summaries throughout the translation process to enhance accuracy and fluency.
The Source-Side Summary $\mathcal{A}_{s}$ encapsulates the main content, domain, style, and tone of the previously translated sections of the document.
This summary serves to preserve a coherent understanding of the text’s overall context, thereby aiding the LLMs in producing more accurate translations.
Conversely, the Target-Side Summary $\mathcal{A}_{t}$ focuses solely on the main content of the previously translated target text.

The pair of summaries are generated by two LLM-based components of the agent: the Source Summary Writer $\mathcal{L}_{\text{WriteS}}$ and the Target Summary Writer $\mathcal{L}_{\text{WriteT}}$.
These summaries are updated every $m$ sentences through a two-step process.
Initially, the writers generate segment summaries for the last $m$ sentences from both the source and target texts: $\tilde{\mathcal{A}}_{s}^{(i+1)}=\mathcal{L}_{\text{WriteS}}(s_{i-m+1},\ldots,s
_{i})$, $\tilde{\mathcal{A}}_{t}^{(i+1)}=\mathcal{L}_{\text{WriteT}}(t_{i-m+1},\ldots,t
_{i})$.
Subsequently, these segment summaries are merged with the previous overall summaries for both sides to summary mergers to obtain new overall summaries: $\mathcal{A}_{s}^{(i+1)}=\mathcal{L}_{\text{MergeS}}(\mathcal{A}_{s}^{(i)},
\tilde{\mathcal{A}}_{s}^{(i+1)})$, $\mathcal{A}_{t}^{(i+1)}=\mathcal{L}_{\text{MergeT}}(\mathcal{A}_{t}^{(i)},
\tilde{\mathcal{A}}_{t}^{(i+1)})$.
This process is repeated iteratively until all sentences in the source document have been read.

#### Long-Term & Short-Term Memory

The last two levels of the agent’s memory component are the Long-Term Memory and the Short-Term Memory, respectively.
These two components are designed to address the requisite coherence across document-level translations.
The Short-Term Memory $\mathcal{M}$ retains the last $k$ source sentences along with their corresponding translations, where $k$ represents a relatively small number: $\mathcal{M}^{(i)}=\{(s_{i-k},t_{i-k}),\ldots,(s_{i-1},t_{i-1})\}$.
This component is specifically designed to capture immediate contextual information in adjacent sentences, which is then seamlessly integrated into the translation prompt, serving as the context for the current sentence.

Conversely, the Long-Term Memory $\mathcal{N}$ component preserves a broader range of context by maintaining a window of the last $l$ sentences from the source document, with $l$ being significantly greater than $k$: $\mathcal{N}^{(i)}=\{(s_{i-l},t_{i-l}),\ldots,(s_{i-1},t_{i-1})\}$.
This allows for the retention of extended coherent information throughout the document.
Before translating a given source sentence, an LLM-based component called the Memory Retriever $\mathcal{L}_{\text{Retrieve}}$ is asked to choose $n$ source sentences that are most relevant to the current translation query: $\hat{\mathcal{N}}^{(i)}=\mathcal{L}_{\text{Retrieve}}(s_{i},N^{(i)})$.
These $m$ sentences with their translations are subsequently incorporated into the translation prompt as demonstration exemplars.

Figure: Algorithm 1 The Overall Framework of DelTA

#### Document Translator

We utilize an LLM-based component called Document Translator $\mathcal{L}_{\text{Translate}}$ to perform the final translation process.
Information from the multi-level memory is integrated into the prompt to support the translator in producing high-quality and consistent translations: $t_{i}=\mathcal{L}_{\text{Translate}}(s_{i},\hat{\mathcal{R}}^{(i)},\hat{
\mathcal{N}}^{(i)},\mathcal{A}_{s}^{(i)},\mathcal{A}_{t}^{(i)},\mathcal{M}^{(i
)})$.
The sentence-by-sentence approach ensures that the resulting target document is consistently aligned with the source document at the sentence level, effectively minimizing the risk of missing target sentences.
Additionally, this method allows for straightforward evaluation of translation quality using sentence-level metrics such as sCOMET.

Table: Table 2: Test results on the IWSLT2017 dataset. Since the translations produced by the Doc2Doc method are not aligned at the sentence level with the source text, we do not report the sCOMET scores for this method. The highest score in each block is highlighted in bold font The results in the “Average” block represent the mean scores across the four backbone models.

## 5 Experiments

### 5.1 Settings

#### Datasets & Metrics

We conduct our experiments on the two test sets.
The first is the tst2017 test sets from the IWSLT2017 translation task(^6^66[https://wit3.fbk.eu/2017-01-d/](https://wit3.fbk.eu/2017-01-d/)) (Akiba et al., ), which consists of parallel documents sourced from TED talks, covering 12 language pairs.
Our experiments are conducted on eight language pairs: En $\Leftrightarrow$ Zh, De, Fr, and Ja.
There are 10 to 12 sentence-level aligned parallel documents with approximately 1.5K sentences for each language pair.
The second is Guofeng Webnovel(^7^77[https://github.com/longyuewangdcu/GuoFeng-Webnovel/](https://github.com/longyuewangdcu/GuoFeng-Webnovel/)) (Wang et al., ; ), a high-quality and discourse-level corpus of web fiction.
We conduct our experiments on the Guofeng V1 TEST_2 set in the Zh $\Rightarrow$ En direction.
The detailed dataset statistics are demonstrated in Appendix B.
We employ LTCR-1 and $\text{LTCR-1}_{\text{f}}$ for proper noun translation consistency evaluation and adopt sCOMET and dCOMET as translation quality metrics, which are all introduced in §3.1.

#### Models and Hyperparameters

In this work, we utilize two versions of GPT models, GPT-3.5-Turbo-0125 and GPT-4o-mini, as our base models.
We get access to these models through the official API provided by OpenAI(^8^88[https://platform.openai.com/docs/guides/text-generation/](https://platform.openai.com/docs/guides/text-generation/)).
We also introduce the open-source Qwen2-7B-Instruct(^9^99[https://huggingface.co/Qwen/Qwen2-7B-Instruct/](https://huggingface.co/Qwen/Qwen2-7B-Instruct/)) and Qwen2-72B-Instruct(^10^1010[https://huggingface.co/Qwen/Qwen2-72B-Instruct/](https://huggingface.co/Qwen/Qwen2-72B-Instruct/)) in our experiments.
The max_new_tokens is set to 2048 and other hyper-parameters remain default.
The updating window of Bilingual summary $m$ and length of Long-Term Memory $l$ are set to 20.
The number of retrieved relative sentences from Long-Term Memory $n$ is set to 2.
The length of Short-Term Memory $k$ is set to 3.

#### Baseline Methods

We include the following three approaches as our baselines.
a) Sentence: We employ the same LLMs but conduct a sentence-level translation process to obtain the baseline results. We use a simple instruction “Translate the following $\langle$src_lang$\rangle$ sentence to $\langle$tgt_lang$\rangle$” to translate one single sentence in the document at a time.
b) Context: We follow Wu et al. () to provide the LLMs with three previously obtained source-target sentence pairs as the context of the current sentence. This method integrates more contextual information and helps adapt the LLMs for the document-level translation task.
c) Doc2Doc: We reproduce the approach proposed by Wang et al. (), translating 10 sentences in a single conversation turn and processing the entire document within a single chat box, thereby leveraging the long-term modeling ability of the LLMs.
In computing the metric scores for the Doc2Doc results, we first perform sentence alignment using Bleualign(^11^1111[https://github.com/rsennrich/Bleualign/](https://github.com/rsennrich/Bleualign/)) to obtain aligned source and target documents, after which we calculate the involved metrics.
Furthermore, we also introduce the results of NLLB-3.3B (Costa-jussà et al., ) and GoogleTrans(^12^1212[https://py-googletrans.readthedocs.io/](https://py-googletrans.readthedocs.io/)) for comparison.

Table: Table 3: Test results on the Guofeng dataset.

### 5.2 Results

#### Test Results on IWSLT2017

The main experiment results on the IWSLT2017 test set are demonstrated in Table 2.
For more detailed scores, please refer to Appendix D.
It is evident that DelTA outperforms baseline approaches on LTCR-1 and $\text{LTCR-1}_{\text{f}}$ metric scores across nearly all translation directions and models.
This indicates that our approach yields significant enhancements in proper noun translation consistency for document-level translation.
Furthermore, DelTA significantly improves the overall quality of document translation, as evidenced by consistently higher sCOMET and dCOMET scores.
The superior dCOMET scores indicate that DelTA effectively captures contextual information to support the translation process.
However, DelTA fails to surpass the Doc2Doc approach when using Qwen2-7B-Instruct as the backbone model, likely due to its limited capabilities in handling the complexity of maintaining translation consistency during document translation.

Translation consistency is improved significantly in the En $\Rightarrow$ Zh direction, while gains in directions like En $\Rightarrow$ De are modest.
For instance, with GPT-3.5-Turbo, LTCR-1 improves by 6.17 percentage points (86.44 vs. 80.27) for En $\Rightarrow$ Zh, but only 1.40 points (93.46 vs. 92.06) for En $\Rightarrow$ De (see Table LABEL:tab:main_results_en_to_many of Appendix D).
This disparity stems from linguistic differences: English proper nouns require conversion into Chinese characters, posing challenges for maintaining consistency, whereas in German, they can be directly copied due to the shared alphabet.
Despite this, a reasonable LTCR-1 improvement in En $\Rightarrow$ De still demonstrates our method’s effectiveness.

#### Test Results on Guofeng

The test results on Guofeng are illustrated in Table 3.
Our approach achieves superior results across almost all metrics and backbone models, demonstrating its robustness to data of the novel domain.
Notably, stronger models, such as GPT-4o-mini and Qwen2-72B-Instruct, achieve greater improvements in translation consistency and quality metrics.
This suggests that the stronger the backbone models are, the more substantial the gains achieved by DelTA.
The Guofeng test set poses particular challenges for maintaining translation consistency due to the prevalence of proper nouns (mainly names) in the source text.
Nevertheless, DelTA demonstrates significant improvements in the relevant metrics, with an increase in LTCR-1 of up to 48.50 percentage points (85.50 vs. 37.00).
This indicates that DelTA represents a strong tool for addressing translation inconsistency issues and holds great potential for novel translation, as it effectively reduces inconsistent noun translations, thereby minimizing potential confusion for readers.

## 6 Analysis

**Table 4: Ablation Study.**
| Id | Setting | sCOMET | dCOMET | LTCR-1 | $\text{LTCR-1}_{\text{f}}$ |
| --- | --- | --- | --- | --- | --- |
| 1 | Sentence-level | 83.78 | 6.55 | 80.27 | 88.78 |
| 2 | 1 + Short-Term Memory | 84.50 | 6.68 | 77.89 | 87.41 |
| 3 | 2 + Long-Term Memory | 84.54 | 6.67 | 79.23 | 89.44 |
| 4 | 3 + Source Summary | 84.61 | 6.68 | 76.09 | 91.25 |
| 5 | 3 + Target Summary | 84.70 | 6.72 | 82.14 | 92.86 |
| 6 | 3 + Bilingual Summary | 84.72 | 6.74 | 82.49 | 93.60 |
| 7 | 6 + Record (DelTA) | 84.70 | 6.72 | 86.44 | 95.25 |

#### Ablation Study

Table 4 presents an ablation study in the En $\Rightarrow$ Zh direction using GPT-3.5-Turbo-0125 as the backbone model.
When provided with context sentences (Model 2), the model exhibits improved translation quality scores, but no significant enhancement in translation consistency is observed.
The introduction of long-term memory contributes to more consistent translations (Model 3), as evidenced by an increase of 2.22 percentage points in LTCR-1.
Incorporating bilingual summaries (Model 6) led to an increase in COMET scores as well as consistency metrics, indicating that this component not only enhances translation quality but also reinforces translation consistency.
When proper noun records are introduced (Model 7), a slight decrease in sCOMET and dCOMET scores is observed, likely due to the perturbation introduced by incorporating additional information.
However, translation consistency improves significantly, with LTCR-1 increasing by 3.95 points and $\text{LTCR-1}_{\text{f}}$ increasing by 1.65 points compared to Model 6.
Moreover, it is evident that the bilingual summary has a superior impact on both translation quality and consistency compared to using a summary on either the source side or the target side alone.
Among Models 4, 5, and 6, Bilingual Summary achieves the highest scores across all four metrics.

#### Consistency Distance

Figure: Figure 2: Proportions of consistent translations in different sentence-wise distances.
Refer to caption: x2.png

One significant challenge in document-level translation is maintaining long-term consistency.
To evaluate whether our approach addresses this challenge, we divide the sentence-wise distance between each proper noun’s translation and its first occurrence into several intervals.
We then report the proportion of consistent translations in each interval in En $\Rightarrow$ Xx (upper) and Xx $\Rightarrow$ En (lower) in Figure 2 using Qwen2-72B-Instruction.
We observe that our approach almost outperforms the Sentence and Context methods in achieving proper noun translation consistency across all distance intervals.
Notably, our approach excels when the distances exceed 50 sentences, yielding a larger proportion of consistent translations than the baseline methods.
This demonstrates the effectiveness of our approach in enhancing long-context translation consistency.

#### Pronoun Translation

**Table 5: Evaluation results of pronoun translation accuracy (APT). Data are presented as percentages.**
| Metric | Sentence | Context | Doc2Doc | DelTA |
| --- | --- | --- | --- | --- |
| APT | 59.96 | 60.84 | 56.11 | 61.07 |

We follow Miculicich et al. (); Tan et al. (); Lyu et al. () to evaluate the accuracy of pronoun translation (APT) of our system using the reference-based metric proposed by Miculicich Werlen & Popescu-Belis ().
The evaluation results of GPT-3.5-Turbo-0125 in the En $\Rightarrow$ Zh direction are demonstrated in Table 5.
DelTA can improve the performance of pronoun translation compared to the sentence-level and Context-based baselines.
This indicates that the multi-level memory in DelTA is beneficial to resolving coreference and anaphora in the document.

#### DelTA as a Summarize Writer

**Table 6: QMSum test results of ReadAgent and DelTA. “Length” denotes the word-wise length of the response.**
| System | ROUGE-L | Length |
| --- | --- | --- |
| ReadAent | 21.50 | 67.86 |
| DelTA | 23.60 | 82.28 |

Our system employs an iterative approach to document summarization, where a summary writer generates partial summaries every 20 sentences, sequentially merging them with previous summaries to produce an updated version.
To assess the effectiveness of this component, we conduct an experiment using the QMSum (Zhong et al., ) test set.
QMSum is a benchmark for query-based multi-domain meeting summarization, where systems are required to generate summaries of the meeting transcripts in response to a specified query.
In our experiment, the query is incorporated into the prompts for both summary generation and the merging process, enabling a query-based summarization approach.
We compare our results with those of Lee et al. (), who paginate the document, generate summaries for each page, and then perform a lookup process on these summaries according to the query.
The results are shown in Table 6.
Our system’s segment-by-segment, query-oriented summary generation approach enables us to effectively locate relevant portions of the document and synthesize the information in these segments through the summary merging process.
This demonstrates that the summary component of DelTA is also well-suited for general summarization tasks.

#### Memory Costs

Figure: Figure 3: Memory cost of the Doc2Doc approach and our approach.
Refer to caption: x3.png

Our agent system consumes less memory than LLM-based Doc2Doc methods, such as those described by Wang et al. ().
In their research, multiple continuous sentences are translated in a single conversational turn, and the whole document is translated within a single chat box.
However, this approach suffers from significant memory costs.
As shown in Figure 3, we compared the memory usage of the Doc2Doc method with our agent-based online approach by utilizing Qwen2-72B-Instruction to translate a document in En $\Rightarrow$ Zh on a device with 2 NVIDIA A800 80GB GPUs.
Our method demonstrates relatively slow memory growth as the number of processed sentences increases, primarily due to the increasing length of the summaries.
In contrast, while the Doc2Doc method starts with lower memory consumption, its usage increases rapidly with the number of processed sentences. Memory consumption surpasses that of our method when the document length reaches 70 sentences, and it runs out of memory when the length reaches 490 sentences.
This indicates that our approach is more memory-efficient and cost-effective for deployment on local devices.

## 7 Conclusion

In this paper, we start with analyzing two critical challenges for DocMT-LLMs, namely translation inconsistency and inaccuracy.
To tackle these issues, we design an online document-level translation agent equipped with a multi-level memory component.
This memory structure retrieves and stores key information to assist the document translation process, significantly enhancing translation consistency and quality.
Notably, the effectiveness in maintaining proper noun translation consistency is particularly pronounced in novel translation, and our approach is still able to maintain consistency even when there is a large distance between the occurrences of a proper noun pair.
The sentence-by-sentence online translation method avoids sentence omissions and reduces GPU memory consumption, in contrast to mainstream Doc2Doc approaches.
Further analysis indicates that our method improves pronoun translation accuracy, and the built-in summarizer component in our agent is also capable of the query-based summarization task.

## Limitations

In this work, we present a framework for the DocMT agent, without prioritizing its inference efficiency.
Given the complexity of DelTA’s inference process, LLMs are frequently invoked during document translation, leading to prolonged runtime.
To address this issue, we identify several potential directions for improvement.
First, by explicitly marking sentence boundaries with special boundary tags, we can enforce sentence-level alignment within generated paragraphs, allowing LLMs to process multiple sentences concurrently and thereby reduce invocation frequency.
Second, employing more precise alignment tools and scripts to extract proper nouns and their translations, rather than relying on LLMs, can further enhance the efficiency of DelTA.
Other components, such as Long-Term Retriever, could be implemented using a dense retriever rather than employing LLMs to identify related sentences.
Finally, in the summary component, reducing the summary generation to a single step by directly merging sentences within the window into an overall summary can also decrease runtime.
We consider these optimizations as future directions of our work.

## Appendix A Examples of Translation Inconsistency and Inaccuracy

Examples of translation inconsistency, undertranslation, and low translation quality issues are presented in Table A.
In the first instance, the name “Natalia Rybczynski”, which appears twice in the source document, is translated into two different forms: “{CJK}UTF8gbsn娜塔莉亚·雷布琴斯基” and “{CJK}UTF8gbsn娜塔莉亚·丽琴斯基”.
This variation leads to a notable inconsistency in the translation.
In the second instance, the second sentence in the source document is omitted.
Its corresponding translation is absent in the target document.
This exemplifies an undertranslation issue in the document translation task.
In the third instance, the translation produced by the LLM with a translation window of 50 sentences deviates significantly from the customary word order in Chinese compared to the sentence-level approach (using a window size of 1).

**Table 7: Demonstrations of proper noun translation inconsistency, undertranslation, and low translation quality issues encountered during document-level translation. In the first part, texts highlighted in red represent the same source proper noun with two different translations in the target document. In the second part, “//” indicates sentence boundaries in the document, illustrating that the translation of the second source sentence is absent from the target document. In the third part, “$\textsc{Hyp}_{n}$” represents the hypothesis generated by the LLM using a translation window of size $n$.**
| Proper Noun Translation Inconsistency |  |
| --- | --- |
| \hdashlineSrc | It’s a story about this woman, Natalia Rybczynski. |
| Hyp | {CJK}UTF8gbsn这是关于这个女人\ruby娜nà\ruby塔tǎ\ruby莉lì\ruby亚yà·\ruby雷léi\ruby布bù\ruby琴qín\ruby斯sī\ruby基jī的故事。 |
| \hdashlineSrc | Natalia Rybczynski: Yeah, I had someone call me “Dr. Dead Things.” |
| Hyp | {CJK}UTF8gbsn\ruby娜nà\ruby塔tǎ\ruby莉lì\ruby娅yà·\ruby丽lì\ruby琴qín\ruby斯sī\ruby基jī：是的，有人叫我“死物博士”。 |
| Undertranslation |  |
| \hdashlineSrc | But here’s the truth.//Here’s the epiphany that I had that changed my thinking.//From 1970 until today, the percentage of the world’s population living in starvation levels… |
| Hyp | {CJK}UTF8gbsn但事实是，// *Missing translation*  //自1970年至今，生活在饥饿水平、每天生活在一美元以下（当然要根据通货膨胀调整）的全球人口比例下降了80％。 |
| Low Translation Quality |  |
| \hdashlineSrc | And we make decisions about where to live, who to marry and even who our friends are going to be, based on what we already believe. |
| Ref | {CJK}UTF8gbsn我们做的各种决定，选择生活在何处，与谁结婚甚至和谁交朋友，都只基于我们已有的信念。 |
| $\textsc{Hyp}_{1}$ | {CJK}UTF8gbsn我们根据自己已有的信念来做决定，包括选择居住的地方，结婚对象，甚至决定谁会成为我们的朋友。 |
| $\textsc{Hyp}_{50}$ | {CJK}UTF8gbsn我们决定居住地、婚姻对象，甚至我们的朋友根据我们已经相信的事情。 |

## Appendix B Detailed Statistics of the Datasets in Our Experiments

We conduct our experiments on two test sets: IWSLT2017 and Guofeng.
The IWSLT2017 test set consists of parallel documents sourced from TED talks, covering 12 language pairs.
Our experiments are conducted on eight of these pairs, En $\Leftrightarrow$ Zh, De, Fr, and Ja.
Each language pair contains 10 to 12 sentence-level aligned parallel documents, totaling approximately 1.5K sentences, with an average of around 120 sentences per document.
Detailed statistics for each language pair are shown in the first block of Table 8.

The Guofeng Webnovel corpus is a high-quality and discourse-level corpus of web fiction, and we conduct our experiments on the Guofeng V1 TEST_2 set, which is designed in the Zh $\Rightarrow$ En language pair.
The second block of Table 8 illustrated the statistics of the test sets.

**Table 8: Statistics of the test sets used in our experiments. “$|\text{S}|$” represents the number of sentences in each test set, “$|\text{D}|$” represents the number of documents, and $|\text{S}|/|\text{D}|$ represents the average number of sentences per document.**
| Dataset | Language | $|\text{S}|$ | $|\text{D}|$ | $|\text{S}|/|\text{D}|$ |
| --- | --- | --- | --- | --- |
| IWSLT2017 | Zh $\Leftrightarrow$ En | 1459 | 12 | 122 |
| De $\Leftrightarrow$ En | 1138 | 10 | 114 |  |
| Fr $\Leftrightarrow$ En | 1455 | 12 | 121 |  |
| Ja $\Leftrightarrow$ En | 1452 | 12 | 121 |  |
| \hdashlineGuofeng V1 TEST_2 | Zh $\Rightarrow$ En | 857 | 12 | 71 |

## Appendix C Prompt Templates for LLM-Based Components in DelTA

This part details the prompts used for each module of DelTA.
The prompt template for the Proper Noun Extractor is depicted in Figure 4.
We use a prompt in the few-shot style to ensure accurate and formatted outputs.
The prompt templates of the source and target summary writers are shown in Figure 5 and Figure 6, respectively.
Note that the prompts for the summary components are written in their respective source or target language to avoid off-target issues.
The prompt template for the Memory Retriever is shown in Figure 7.
The prompt template for the Document Translator is illustrated in Figure 8.
All the retrieved information from the multi-level memory components is formatted into this prompt to assist the translation process.

## Appendix D Detailed Results of the Main Experiment

The scores for the En $\Rightarrow$ Zh, De, Fr, Ja translation directions are presented in Table LABEL:tab:main_results_en_to_many, while the scores for the Zh, De, Fr, Ja $\Rightarrow$ En are shown in Table LABEL:tab:main_results_many_to_en.

DelTA achieves improvements in both translation consistency, as indicated by the LTCR-1 and $\text{LTCR-1}_{\text{f}}$ metrics, and translation quality, as indicated by the sCOMET and dCOMET metrics, across most translation directions compared to several baselines.
The Qwen models show significant enhancements in sCOMET and dCOMET scores, demonstrating that our approach provides strong reinforcement for the document translation quality of these open-source models.

The quality improvements are most pronounced in the Ja $\Leftrightarrow$ En directions across all backbone models, likely due to their modest baseline capabilities for these language pairs, which our approach effectively enhances.
Translation consistency in the Zh $\Leftrightarrow$ En directions benefits most from our approach, as the distinct character sets of Chinese and English pose challenges in maintaining proper noun consistency, highlighting the effectiveness of our method.

When applying GPT-3.5-Turbo as the backbone model, DelTA outperforms translation-specialized baselines, such as NLLB-3.3B and GoogleTrans, across most languages, demonstrating the promising capabilities of LLM-based autonomous agents in document translation.

Figure: Figure 4: Prompt template for the Proper Noun Extractor. We provide several few-shot exemplars preceding the current input. This template is designed for the En $\Rightarrow$ Zh translation direction. For other translation directions, adjust the corresponding content to match the specific languages.

Figure: Figure 5: Prompt template for Source Summary Writer.

Figure: Figure 6: Prompt template for Target Summary Writer. We write the prompt in the target language to better align with the agent profile of the monolingual summary writer and reduce the off-target issues. For demonstration purposes, the prompts provided here are written in English.

Figure: Figure 7: Prompt template for Long-Term Memory Retriever.

Figure: Figure 8: Prompt template for Document Translator. Involved proper nouns and their corresponding translations are formatted into the “HISTORY” field. Source and target sentences from Short-Term Memory are concatenated and formatted into the “SRC_CONTEXT” and “TGT_CONTEXT” fields, respectively. Retrieved instances from Long-Term Memory are formatted into the “RELEVANT_INSTANCES” field as source-target pairs.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: * How well does this scale to extremely low-resource languages?
soundness: 3
strengths: * This paper effectively addresses a significant challenge in the field, presenting meaningful improvements and contributions.

*The experimental design is comprehensive and well-executed, providing robust results.

*The presentation is clear, concise, and well-organized, facilitating easy understanding of the research.
summary: This paper presents a methodology that improves translation consistency by using an agent-based approach. DelTA utilizes a modular, multi-level memory system, comprising Proper Noun Records, Bilingual Summary, Long-Term Memory, and Short-Term Memory, which store and manage information at different scales, with auxiliary LLM-driven components ensuring efficient information retrieval and updates.
weaknesses: * As authors point out in limitations, efficiency of the method

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: * Consider using the WMT24 shared task benchmark test sets that are going to be released. They provide document-level test sets. In 2023 the organizers released test sets only for English-German but for this year more language pairs will be made available.
soundness: 3
strengths: The idea of the paper is simple and straightforward: provide more context about the translation task by presenting this information in the prompt of an LLM. The context is varied: sentences occurring in the immediate before the sentence being translated, context that is longer with sentences much before the current sentence, a summary of the source text, a summary of the target text and proper nouns occurring in the text with their respective translations.
summary: The paper proposes DelTA, a document-level translation agent that can translate documents or long context text with better translation quality than other methods. The approach consists of a series of prompts that are used to encode information about proper nouns, immediate context, larger context, summaries in the source and target language. Experiments are carried out in the IWSLT2017 benchmark for 4 language pairs and evaluated for consistency of translation terms and translation quality using reference-based translation evaluation methods.
weaknesses: The evaluation was performed only with automatic metrics such as sentence-level and document-level COMET. It would be interesting to understand if the performance reported holds with evaluation performed by translators in terms of quality and consistency as both implementations of COMET are known to be weak proxies for translation quality beyond the sentence-level, failing to capture consistency, coherence and other discourse-related linguistic dimensions.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. For the summary writers, why isn't the summary for chunk m conditioned on the summary for chunk m-1 (which would allow the summary to take longer context into account)?
2. Did you investigate the effect of adjusting the numbers (k, l) of sentences to use as short- and long-term memory throughout the translation process? It seems likely that the model would probably need more context at later stages (when more sentences have already been seen), versus at the beginning of translating a long document.
3. In Figure 2, why does consistency drop so much for the en-xx 11-20 sentences bucket (and then increase again for longer sentence-wise distances) across all models?
4. The Workshop on Machine Translation (WMT, https://machinetranslate.org/wmt) also has document-level test sets, which are canonically used for MT evaluation in the literature. Why were evals presented on IWSLT2017 and Guofeng (relatively unknown datasets), but not on these standard test sets? Do the test sets used require long context to produce correct translations?
soundness: 3
strengths: 1. Document-level translation is currently one of the most important and underexplored research areas in the field of machine translation (MT). This paper clearly presents the motivation for their work, and summarizes the importance of document-level MT.
2. The Delta system presented in the paper is easy to implement, since it just requires prompting of off-the-shelf LLMs. The motivation for each of the modules in the Delta system is clear.
3. The experimental setup is well-written and easy to follow. The results are comprehensive (covering multiple LLMs and language pairs), and comparisons against all essential document-level MT baselines are presented.
summary: This paper proposes a LLM-based, document-level translation agent (called Delta). This agent performs document-level translation on a sentence-by-sentence basis, and consists of auxiliary "multi-level memory" components, which are updated after translation of each sentence. These memory components include Proper Noun Records, Bilingual Summary, and Long-Term/Short-Term Memory. The information stored in all of these components is retrieved by prompting LLMs, where the prompt for each component is targeted towards its desired function (e.g., the Bilingual Summary component asks the LLM to generate a summary given a chunk of sentences). During translation of each sentence within a document, after updating all of these memory components, the Delta system aggregates their stored information into a single prompt. This process is continued iteratively until the entire document is translated. The paper shows that the Delta system outperforms other document-level MT systems on the IWSLT2017 and Guofeng datasets, and also presents ablations to understand the contribution of each memory component to the overall performance of Delta.
weaknesses: 1. The paper mentions that inconsistency is one of the major challenges during document-level translation. However, the only type of inconsistency they measure and report is Proper Noun Inconsistency, which is a surface-level type of inconsistency (can the model consistently copy a fixed string?) that doesn't require the model to reason across long contexts. The results on Proper Noun Inconsistency are used to make strong claims about their Delta system's overall improvement in consistency relative to baselines, but these claims are not adequately supported with evidence.
2. The paper focuses on the comparison to "Doc2Doc" (comparing memory costs, etc), but from the reported results, "Context" is consistently better than "Doc2Doc". There is no significance testing for the results in the paper (e.g., Tables 2 and 3), and in terms of translation quality (as measured by sCOMET and dCOMET), the performance of the "Context" and "Delta" models looks quite close, with "Context" outperforming "Delta" according to dCOMET for the Qwen2-7B-Instruct model in Table 2. In Table 3, the quality-based performance of "Context" and "Delta" is even closer. (And it is not surprising that Delta outperforms in proper noun consistency, since the proper noun extractor module in Delta provides a list of all proper nouns seen. If the Context model were enhanced with this module, it seems likely that its performance on the proper noun consistency metric would also correspondingly improve.)
3. The Delta system is comprised of 5 auxiliary modules. The ablation to isolate the effects of each individual module in Figure 4 shows very small differences in quality (as measured by sCOMET and dCOMET) across these models (again, with no significance testing). The recipe (combination of modules) proposed by the authors works reasonably well based on their results, but the paper provides limited insight into the relative contribution of each of these modules to improving performance. For instance, does short-term or long-term memory help more? Is the Memory Retriever step for long-term memory necessary, and how much does this step improve performance over just using the (l sentences in) long-term memory?
See additional questions below.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. How does the approach perform compared to long-context LLMs? Given recent advances in context window sizes, this seems particularly relevant.

2. Will the code be released? This would be valuable for reproducibility and adoption.

3. Can you provide a more detailed efficiency analysis, including:
   - Total wall-time comparison between DELTA and Doc2Doc approaches
   - FLOPS comparison for open-weight models
   - Analysis of the number of LLM calls required

4. How does DELTA handle context-dependent translations where meaning depends on broader document context? While pronoun translation is briefly discussed, a more comprehensive analysis (see https://aclanthology.org/2023.wmt-1.42/) would be valuable.

5. Have you considered evaluating against DeepL, given its explicit support for document-level translation?

6. Could you provide statistical significance tests for the reported improvements?
soundness: 3
strengths: **Originality**:
- The multi-level memory architecture represents a novel approach to document translation. While memory architectures have been used in LLM agents before, the specific design with four complementary components (Proper Noun Records, Bilingual Summary, Long/Short-Term Memory) targeting different aspects of document translation is innovative.
- The approach of using an LLM-based agent for document translation, while maintaining sentence-level alignment, is creative and practical.
- The introduction of LTCR-1 and LTCR-1f metrics for evaluating translation consistency is a useful contribution to the field.

**Quality**:
- The experimental methodology is comprehensive, evaluating across 8 language pairs and multiple LLM backbones (both open and closed source).
- The ablation studies effectively demonstrate the contribution of each memory component.
- The memory efficiency analysis is well-done, with clear comparisons showing the advantages over Doc2Doc approaches.
- The evaluation considers both translation quality (sCOMET, dCOMET) and consistency (LTCR metrics).

**Clarity**:
- The paper is logically structured, moving from motivation to design to evaluation.
- The multi-level memory architecture is well-explained with clear diagrams and algorithms.
- The experimental results are presented systematically with detailed tables and analyses.
- The limitations and future work are honestly discussed.

**Significance**:
- The work addresses a real and important challenge in document translation - maintaining consistency while handling long documents.
- The approach is practical and implementable, with clear memory efficiency advantages.
- The method shows strong improvements across different languages and models, suggesting broad applicability.
- The memory architecture could potentially be adapted for other document-level NLP tasks, as demonstrated by the query-based summarization results.
summary: The paper introduces DELTA, a Document-level Translation Agent that uses a multi-level memory architecture to improve translation consistency and quality in document-level machine translation. The agent employs four memory components (Proper Noun Records, Bilingual Summary, Long-Term Memory, and Short-Term Memory) that store and retrieve information across different granularities. The approach shows impressive improvements in translation consistency and quality across multiple language pairs and different LLM backbones. The work is evaluated extensively using both established metrics and novel consistency measures.
weaknesses: 1. Efficiency analysis needs more detail:
   - While memory efficiency is demonstrated, the paper claims cost-effectiveness without supporting evidence
   - No comparison of total wall-time or FLOPS between approaches
   - Limited discussion of computational overhead from multiple LLM calls

2. Missing comparisons and baselines:
   - No evaluation against long-context LLMs (e.g., Claude, Gemini) which could be particularly relevant for document translation
   - DeepL, which explicitly supports document-level translation, would be a valuable baseline, and a more relevant choice than Google.
   - Limited discussion of Doc2Doc approach limitations in related work

3. Incomplete motivation and analysis:
   - Does not address context-dependent translation as a key challenge (see https://aclanthology.org/2023.wmt-1.42/)
   - Claims of "significant" improvements lack statistical significance testing
   - Claims about undertranslation issues and translation quality decline (L177-178) should be supported with citations to relevant work (e.g.,  https://arxiv.org/abs/2401.06468, https://aclanthology.org/2024.acl-long.336/)
 
4. Technical details and reproducibility:
   - No mention of code release plans

Meta-review: This paper introduces DelTA, a Document-level Translation Agent designed to enhance translation consistency and quality through an agent-based, multi-level memory system. DelTA uses four memory components: Proper Noun Records, Bilingual Summary, Long-Term Memory, and Short-Term Memory, to effectively store and retrieve information at varying levels of detail. By utilizing LLM-driven components, DelTA ensures efficient information management and retrieval. The methodology is tested on the IWSLT2017 benchmark across four language pairs, showing superior translation quality compared to traditional methods. Evaluation employs both standard and novel consistency metrics, demonstrating notable improvements in translation consistency and quality.

On the positive side, the paper offers a straightforward yet impactful approach to enhancing translation quality by incorporating additional context into the prompts given to large language models (LLMs). This context includes the immediate preceding sentences, longer contexts from earlier in the text, summaries of both the source and target texts, and proper nouns with their translations. The motivation for this research is well-articulated, highlighting the significance and relative underexploration of document-level machine translation (MT) in the current landscape. DelTA, the system proposed, is notably easy to implement as it primarily involves prompting existing LLMs. Each module within the DelTA system is clearly justified, contributing to the paper's clear and accessible presentation of its experimental setup. The results are thorough, covering a variety of LLMs and language pairs, and include comprehensive comparisons with key document-level MT baselines.

Areas for improvement, as highlighted by the reviewer, include the reliance on automatic metrics such as sentence-level and document-level COMET for evaluation. To gain a more comprehensive understanding of the system's performance, it would be beneficial to incorporate assessments by human translators, particularly regarding quality and consistency. Current COMET implementations are recognized as weak proxies for translation quality beyond the sentence level, often failing to adequately capture elements such as consistency, coherence, and other discourse-related linguistic aspects. Additionally, the initial evaluations lacked significance testing, a concern that the authors have since addressed and updated in the revised paper. Another concern I have regarding this paper is its narrow focus on document translation, a specific aspect of natural language processing (NLP). While the findings may be valuable to a limited segment of the scientific community, they may not have broad applicability across the field of NLP as a whole, including ICLR audiences.  Consequently, while the paper presents significant merits within the machine translation field, I am learning to recommend acceptance. 

Its impact appears to be more confined to a specialized audience rather than offering broad applicability across the wider NLP community. As such, it may be more advantageous for this paper to be presented at an NLP-focused conference where its specialized contributions would be better appreciated.

Decision: Accept (Poster)