Benchmarking and Improving Long-Text Translation with Large Language Models (2024)

Abstract: Recent studies have illuminated the promising capabilities of large language models (LLMs) in handling long texts. However, their performance in machine translation (MT) of long documents remains underexplored. This paper aims to shed light on how LLMs navigate this complex task, offering a comprehensive evaluation of their capabilities and limitations in long-text MT. First, we collect and construct an instruction-based benchmark dataset, specifically designed for the finetuning and evaluation of LLMs, encompassing multilingual, multi-domain, and document-level parallel data. Second, we conduct a comprehensive comparison between MT and LLM models concerning document-level translation. Our analysis uncovers that LLMs exhibit shortcomings in long-text domains, and their performance diminishes as document size escalates. By exploiting various extrapolation strategies, we enhance the capacity of LLMs to translate longer texts. We release data, code, and models at https://github.com/ longyuewangdcu/Document-MT-LLM.

heading: 1 Introduction
text: Recently, significant progress has been made in the field of large language models (LLMs) such as ChatGPT (Ouyang et al., 2022; OpenAI, 2023). These models have demonstrated remarkable capabilities in processing extensive texts, opening up new avenues for a variety of natural language processing (NLP) tasks (Li et al., 2023). Specifically, research on long-text LLMs primarily focuses on two aspects: 1) Maximum Capacity: how long text a model can handle (Chen et al., 2023b; Ding et al., 2023); 2) Efficient Utilization: how efficiently that model can model long texts (Liu et al., 2023). Even though LLMs can process long texts, it does not
∗ Longyue Wang, Zefeng Du and Wenxiang Jiao contribute equally. This work was completed during the internships of Zefeng Du, Jianhui Pang, and Chenyang Lyu at Tencent AI Lab.
necessarily imply they can effectively model such texts. The effectiveness and necessity of extrapolating LLMs for various NLP tasks remain ambiguous. In this paper, we predominantly delve into the use of LLMs for long-text machine translation (MT).
Diverging from other long-text tasks such as text summarization and completion, which involve either long input or output, long-text MT presents unique challenges. It necessitates not only comprehending extensive input but also preserving context and coherence over extended text sequences (Yang et al., 2019, 2020). This amalgamation of requirements intensifies the complexity of the task, positioning long-text MT as a rigorous benchmark for evaluating the potential and prowess of LLMs.
To facilitate this study, we first construct an instruction-based benchmark dataset designed for finetuning and evaluating the document-level translation capabilities LLMs. This dataset comprises a total of 410K documents with a document length 160∼54K tokens,1 spanning across seven domains (i.e. news, TED, subtitles, Parliamentary, social, Q&A, novels) and three language pairs (i.e. Chinese-English, German-English, RussianEnglish). Leveraging this dataset, we conduct a comprehensive comparison between MT and LLM models (e.g. DeepL, GPT-4, Llama). The experiments reveal that 1) all models experience a performance decline in the novels domain due to its exceedingly long nature; 2) the performance of LLMs drastically deteriorates as the document length increases from 2K to 8K. To further investigate the impact of enlarging document length on translation quality, we examine a variety of extrapolation strategies during finetuning stage. Our findings indicate that a mixture of document lengths with multi-task learning can effectively enhance LLMs’ performance. The main contributions are: • To facilitate future research on long-text transla-
1Here it means subword, referring a basic processing unit after sentencepiece.
7175
tion, we release a large instruction-based dataset. This dataset is comprehensive, covering various domains and languages, and notably includes exceptionally long texts (Section 3).
• We establish a long-text MT benchmark by evaluating advanced MT and LLM models. This not only highlights the limitations of current LLMs but also provides insights for improving their performance in translating long texts (Section 4).
• We investigate simple finetuning strategies, and identify effective methods to enhance the LLMs’ proficiency in translating long texts (Section 5).

heading: 2 Preliminary
text: Background Recent years have seen a growing interest in document-level translation by leveraging traditional neural models (Wang et al., 2017; Voita et al., 2019a; Wang et al., 2018), pre-training (Liu et al., 2020) as well as LLMs (Wang et al., 2023a; Wu et al., 2024a). Surprisingly, LLMs have shown potentials in modeling exceptionally long texts with complex discourse structures for document-level MT (Wang et al., 2023b; Pang et al., 2024a; Lyu et al., 2024). More recently, there have been sev-
eral length extrapolation methods aimed at efficiently extending the pre-trained context length of LLMs (Chen et al., 2023a,b). However, they were rarely tested on challenging generation tasks such as long-text translation. Our preliminary experiments on document translation using simple extrapolation methods (Pang et al., 2024b; Chen et al., 2023b) show a significant performance degradation on WMT2023 English-German testset (Llama-2 vs. +Extrapolation: 39.1 vs. 35.6 d-BLEU), which is consistent with the findings in multi-document QA task (Liu et al., 2024). Thus, this work mainly investigate finetuning strategies.
Pseudo-Document Given a full length of document-level parallel corpus, previous works usually concatenate N consecutive sentences into a pseudo-document (Wang et al., 2017; Voita et al., 2019b; Wu et al., 2023) until it matches either the
maximum length limitation or the actual document length. We follow this method to preprocess all types of data in our paper.
Evaluation Methods The d-BLEU (documentlevel BLEU) metric (Liu et al., 2020) has emerged as the de facto standard for evaluating documentlevel translation models such as MCN (Zheng et al., 2021), G-Trans (Bao et al., 2021), and MRDoc2Doc (Sun et al., 2022). Besides, Wang et al. (2023a) demonstrates that d-BLEU are more attuned to discourse than sentence-level BLEU. In this work, we utilize the commonly-used d-BLEU as the default instead of sentence-level metrics since all translations are at the document level. As a supplement, we also report results using human evaluation (Wang et al., 2023a) and the BlonDe automatic metric (Jiang et al., 2022) in Section 5.4
and the Appendix §A.3, respectively. Note that, evaluating document-level translation is another challenging issue, which potentially beyond the scope of this work.

heading: 3 Instruction-Based Benchmark Dataset
text: According to Taori et al. (2023), we construct instruction-based datasets for long-text translation training and testing, spanning various domains and languages. In addition to leveraging existing parallel datasets, we create two challenging testsets.

heading: 3.1 Overview and Statistics
text: Following Wang et al. (2023a), we ask GPT-4 to suggest a variety of instructions and manually refine them for the long-text translation task (as enumerated in Table 1). We then generate
instruction-based instances by integrating the original document-level sentence pairs with these instructions. As seen, each instance consists of four parts: 1) System Prompt, a predefined text to set the context or frame for the model’s response (fixed length); 2) Instruction, describing the specific task (around 8 words); 3) Input, sentences in the source language; and 4) Response, the corresponding translation in the target language. During instruction data construction, we combine N continuous sentences as a pseudo-document, of which source and target sides are respectively added into the ###Input and ###Response to form one instance of instruction-based MT data.
Table 2 outlines the languages, domains, and statistics of our dataset. We choose three language pairs: Chinese-English (Zh-En), German-English (De-En) and Russian-English (Ru-En). We select texts from seven domains, of which five are consistent across both training and testing (i.e. News, TED, Subtitle, Parliamentar and Novels), while the remaining two are exclusive to the testing set (i.e. Social and Q&A): • News. We used WMT2008∼2021 News Devsets
and WMT2022 News Commentary for training, and WMT2022 News Testset for testing.2
• TED. We collected TED talks in the IWSLT2015, 2017 and 2014 corpora for training, and the standard testsets for testing.3
• Subtitle. We collected movie subtitles in the OpenSubtitles2018 corpus for training.4 As the existing Ru-En testset contains limited length of context (Voita et al., 2019a), we employed professional translators to create a new one.
• Parliamentary. We employed the United Nations v1.0 and Europarl v7 training and testing sets.5
• Novels. The Par3 corpus is organized at the paragraph level.6 Note that, Par3 does not reflect the natural length since the publicly available data has already been processed into pseudodocuments (without orignal data). In addition, we release a copyrighted corpus: 23K continuous chapters from 152 web fictions, covering 14 genres such as fantasy science and romance. The texts are originally written in Chinese and were
2https://www.statmt.org/wmt22. 3https://wit3.fbk.eu. 4https://opus.nlpl.eu/OpenSubtitles-v2018.php. 5https://conferences.unite.un.org/uncorpus and
https://www.statmt.org/europarl. 6https://github.com/katherinethai/par3.
translated into English by professional translators. Regarding testsets, we utilize both mZPRT and our specially curated long-text. More details are discussed in Appendix §A.2.
• Q&A and Social. We used subsets in BaiduKnows and website domains from WMT2022 and mZPRT7 corpora, respectively. The average document length (|T|/|D|) can be considered as an indicator of the complexity inherent in a document-level dataset. Extended documents introduce a two-fold challenge: they not only test the model’s capacity to process extensive sequences but also necessitate the understanding and generation of nuanced discourse properties across extended contexts. As seen, the document lengths of our constructed testsets are 16K and 54K, currently making them the longest document-level translation test sets available.

heading: 3.2 Analysis and Discussion
text: Data Distribution Figure 1 provides a visualization of the benchmark datasets, illustrating the natural distribution of data size and document length across various domains. As seen, 1) the data scale across different domains is imbalanced. For instance, there is an abundance of data in the Parliamentary, while spoken language domains such as TED are scarce; 2) the natural length of documents varies across different domains. For example, Novels and Subtitles tend to be quite lengthy, whereas News are typically shorter; 3) The distribution between the training and testing sets is not exactly identical, which mirrors real-world challenges.
Data Contamination There exists a risk of data contamination as publicly available data are often leveraged during different stages of LLM training (e.g., pre-training, supervised fine-tuning (SFT), or reinforcement learning from human feedback (RLHF)). To mitigate this risk, we have compiled three new datasets for training and testing models:
• Novels Zh-En training set. We crawl monolingual web novels and manually conduct crosslingual alignment. The collected Par3 dataset is organized at the paragraph level, while ours is more extensive, organized at the chapter level.
• Novels Zh-En testing set. We collect latest Chinese web novels and then employ professional translators to translate them into English. We
7https://github.com/longyuewangdcu/mZPRT.
combine 20 continuous chapters into one document, resulting in a super-long document.
• Subtitle Ru-En testing set. We collect latest movie subtitles in English and engage professional translators to translate them into Russian.

heading: 4 Benchmark Results
text: We provide a comprehensive view of how MT and LLM systems handle the complexities of long-text translation across various languages and domains.

heading: 4.1 Experimental Setup
text: Models We utilize testing sets in the benchmark dataset. We conduct a systematic comparison of: • Commercial MT Systems. The commercial trans-
lation products always perform best in translation performance. However, they typically preprocess documents into individual sentences and then perform translation on a sentence-by-sentence basis. We mainly compare Google Translate,8 DeepL Translate,9 and Tencent Translate,10 as adopted in (Jiao et al., 2023b).
• Commercial LLM Systems. ChatGPT are known for their extensive context modeling capabilities and can achieve state-of-the-art results across various languages and tasks. Between August 1st
8https://translate.google.com. 9https://www.deepl.com.
10https://transmart.qq.com.
and 30th, 2023, we used GPT-3.5 (4K) and GPT4 (8K) APIs to translate documents.11 In our preliminary experiments, we employed GPT-4 (128K) but observed frequent instances of omitted translations.
• Open-sourced LLM Models. Llama and Llama2-chat are widely-used, open-source LLMs, pretrained and fine-tuned with a maximum length of 4,096 tokens (Touvron et al., 2023a,b). This enables to handle a relatively longer context window size, making it suitable for document-level general tasks. About Llama-mt, we enhance Llama on document-level MT by using the training set of our benchmark for SFT. We randomly select 5K instances from each subset in Table 2, resulting in approximately 200K instances.
Pre-processing Pseudo-Document In testing commercial and open-source LLM systems, we concatenate N consecutive sentences into a pseudodocument (Wang et al., 2017; Voita et al., 2019b; Wu et al., 2023) until it matches either the maximum LLM input or the actual document length. Assuming a 1:1.5 input to output ratio, we can calculate the maximum size of the maximum LLM input based on the total length ∈ {2K, 4K, 8K}. We transform these pseudo-document parallel data into instruction formats as Jiao et al. (2023a).
11API: gpt-3.5-turbo-0125 and gpt-4-1106-preview.
Finetuning LLMs for Document Translation We finetune Llama-7B for 3 epochs with a batch size of 128. Our experiments are conducted on 16 NVIDIA A100 GPUs with DeepSpeed ZeRO stage 3 facilitating model parallelism.12 To conserve GPU memory, we integrate Flash Attention mechanism (Dao et al., 2022). The SFT task is akin to the casual language modeling task, where the loss is computed solely based on the output portion of the input sequence.
Evaluation According to Wang et al. (2023a), the document-level sacreBLEU (d-BLEU) (Liu et al., 2020) are consistent with performances of discourse-specific phenomena (e.g. consistency of terminology translation and accuracy zero pronoun translation). We employ case-sensitive d-BLEU as our metric, which is computed by matching ngrams in the whole document.

heading: 4.2 Main Results
text: Table 3 presents a comprehensive comparison of various MT and LLM models on benchmark testing set across different domains and languages. When evaluating individual testsets, we notice discernible differences between models. Taking Zh-En testsets for example, Tencent Translate excels in Novels, while GPT-4 is more adept with web-based content like Social and Q&A. Concerning language pairs, Google and DeepL models show proficiency in tasks targeting non-English languages, whereas GPT-4 predominates in tasks with English as the
12https://github.com/microsoft/DeepSpeed.
target. These variances can largely be attributed to differences in training data. Consequently, we will draw our conclusions holistically rather than relying solely on individual test sets.
On average across all testsets, GPT-4 outperforms all other models (i.e. 31.36 BLEU), particularly excelling in the Zh-En News, Social, Q&A, and En-De TED. The performance of GPT-3.5 is comparable to that of sentence-level MT systems, such as Tencent Translate (e.g. GPT-3.5 30.29 vs. Tencent Translate 30.48 BLEU). In contrast, Llama2-chat registers a significantly lower score of 9.4 BLEU, attributable to its limited MT data during SFT and coverage of other languages (e.g. Russian). Addressing the need for long-text MT, we developed Llama-mt models by finetuning Llama using our benchmark training data. This adaptation enhances the translation quality of Llama by almost 200% (Llama-mt-2K 27.9 vs. Llama-2chat 9.4 BLEU), placing it on par with commercial sentence-level MT systems (Llama-mt-2K 27.9 vs. DeepL 28.9 BLEU). Upon closer comparison between Llama-mt 2K and 4K, we observed a significant drop in performance, particularly on the Novel-XL testset, with a decrease of -3.3 BLEU. We cannot enhance the long-text translation capability of Llama by simply finetuning and decoding it with longer texts. This highlights the complexities and challenges inherent in attempting to bolster LLMs’ proficiency in long-text translation.
When comparing the performance of ChatGPT reported in Wang et al. (2023a), we notice a general upward trend across the overlapping testsets.
While this might indicate performance enhancements over time with model evolution, it’s crucial to acknowledge the different experimental parameters: 1) Their evaluations were conducted in March 2023, whereas ours took place in August 2023. 2) They accessed ChatGPT via its web interface, whereas we used its API. 3) Their methodology involved pseudo-documents of predetermined lengths (4 or 10 sentences), taking into account multi-turn contexts. Conversely, we employed the maximum lengths allowed by the API (2K or 4K) and disregarded multi-turn contexts.

heading: 4.3 Effects of Document Lengths
text: We further investigate the impacts of directly extending document length at decoding phase on the Nov.XL dataset (more analysis are discussed in Appendix §A.1). For comparison, we select two representative systems: DeepL and Llama-2-chat with 4K length of texts in pre-trained and SFT. Figure 2 shows the translation performance as the document length increases ∈ {256, 1K, 2K, 4K, 8K} during the decoding phase. While the performance of sentence-level DeepL remains unaffected by decoding length, Llama-2-chat paints a different picture. It initially achieves its peak d-BLEU score at a document length of 1K but sees a sharp decline from 2K to 8K. This underscores that even though LLMs can process long texts, it does not necessarily imply they can effectively model such texts. This motivates us to exploit extrapolation methods to enhance the ability of LLMs to translate texts that extend beyond their pretraining length. Another interesting observation is that the MT performance of Llama-2-chat does not reach its zenith at 4K, diverging from the lengths used in pre-training and
SFT. We suspect that this may be caused by various training signals at SFT phase. To focus specifically on the MT task, without being influenced by signals from other tasks, we primarily employ the Llamamt settings for further experiments (in Section 5).

heading: 5 Context Extrapolation on Translation
text: In practice, it is unrealistic to keep increasing the context window of LLMs in the training process due to resource limits and unsatisfied performance (Chen et al., 2023b). Thus, it is interesting to investigate the potential of LLMs in translating documents that are noticeably longer than the predefined context window. In this section, we focus on curating the training data of long texts and explore various strategies for context extrapolation.
We employ Llama (7B) as testbed, which is pre-trained and fine-tuned with a maximum length of 2,048 token (Touvron et al., 2023a). Following the Llama-mt configurations detailed in Section 4.1, we pre-process each training subset into pseudo-documents with defined length. Then we randomly select 5K instances from each training subset (consider language directions), culminating ∼200K instances for SFT phase. About evaluation, we utilize Chinese⇒English WMT2022 News and mZPRT Novels as representative datasets and report d-BLEU score. We have examined different methods for utilizing data to improve the effective-
ness of modeling text at the document level. We maintain the overall bilingual token count used during instruction finetuning. Note that three methods in Section 5.1, 5.2, and 5.3 build upon each other.

heading: 5.1 Effects of Scaling Document Length
text: First of all, we present how LLMs perform when directly scaling the document size for training and decoding. For comparison, we also include the performance of the sentence-to-sentence approach like typical MT systems. At the SFT phase (FT), we employ two context windows (2048 and 4096), which are used to split the full documents with sliding block techniques. In the decoding phase (DEC), we assess LLMs on a variety of document lengths: shorter than the context window, matching the context window, and exceeding it. For example, when finetuning LLMs with the context window of 2048, we test on documents with context windows of 1024, 2048, and 4096, respectively. Here, 2048 corresponds to the context window employed during Llama’s pre-training and finetuning, while 4096 represents an extrapolated context window.
Table 4 lists the results. Compared with the sentence-by-sentence approach (SENT-sent), increasing the context window during training can lead to much better translation results, which corroborates the findings of paragraph-level translation demonstrated in Karpinska and Iyyer (2023). Both approaches highlight the importance of
context-aware translation techniques in achieving better results for long texts. Besides, we have two specific observations concerning the context windows: (1) With a context window of 2048 for SFT, the test performance on documents with the context window of 4096 degrades significantly, indicating the poor ability of LLMs in context extrapolation. (2) When conducting SFT with a context window of 4096, which is larger than that used in llama pre-training, we have not observed further improvements over the context window of 2048. Specifically, both DOC(4096)-doc(2048) and DOC(4096)-doc(4096) underperform DOC(2048)doc(2048), especially on Novels, which indicates the ineffectiveness of merely increasing context window. Therefore, it becomes crucial to explore other methods for context extrapolation.

heading: 5.2 Effects of Content Overlapping and Mixture of Document Lengths
text: As an attempt, we explore the compositional generalization ability of LLMs in learning from short documents and generalizing to long documents. The default method for constructing training data has two problems that may limit the generalization ability of LLMs: (1) The full documents are split into blocks with no overlap; (2) The length of input documents is fixed rather than following the length distribution in practice. For the first problems, we propose shift block to allow the content overlap between adjacent document blocks (CO, Content Overlapping). With a sliding window size of N, the overlap is N/2, indicating that the window advances by N/2 sentences to extract the next document. For the second, we propose to mix document blocks with varied sizes (MDS, Mixture of Document Sizes). For example, if the base N is 2048, we randomly select one of the numbers from the set 256, 512, 1024, 2048 (all predefined values smaller than 2048 are considered) to serve as N for extracting the current pseudo-document. The process is then repeated in subsequent cycles.
We present the results in Table 5. Unexpectedly, content overlapping leads to worse performance which may result from the less unique tokens during finetuning. In contrast, the model trained with the mixture of document sizes achieves noticeable improvement, compared to that with a fixed document size. The improvement is especially enlarged when we adopt a context window exceeding the one used in training, e.g., 5120 vs. 4096. It demon-
Scaling +TE News Novels DOC doc
strates that learning on various sizes of documents indeed improves the generalization of LLMs to longer documents.

heading: 5.3 Effects of Multi-task Finetuning
text: Unlike sentence-level translation, document-level translation tasks suffer from the scarcity issue of bilingual parallel corpora severely, which may limit the performance of long-text finetuning. We exploit the advantages of monolingual novel data and combine it with our document-level translation corpora in the form of text completion (TE). Similar to Section 3, we construct an instruction-based dataset for TE by combining instructions (in Table 7) with Chinese and English monolingual Novels (different from parallel data in Table 2). We employ the data split by MDS (in Section 5.2) and the scale of the data is equivalent to that of the Chinese-English document translation corpus.
As shown in Table 6, incorporating the monolingual text completion tasks can improve the performance considerably. However, it can also degrade the translation performance slightly when extrapolating the context, making more hallucinations due to the relatively free generation style of text completion. Nonetheless, monolingual text completion tasks can improve the modeling ability of Llama models in non-English languages, and also ameliorate the language imbalance issue of Englishcentric parallel data. We will explore this direction more extensively in the future.

heading: 5.4 Results of Human Evaluation
text: We randomly selected a paragraph of Novels subset (around 50 sentences) from three system outputs: Table 4 DOC (2048)-doc (2048), Table 5 DOC (4096)-doc (4096) +MDS, Table 6 DOC (4096)- doc (4096) +TE. We follow Wang et al. (2023a)’s
general guideline to conduct a human evaluation. The results are 1.6, 2.1, 2.8, respectively (d-BLEU scores are 16.6, 18.1, 19.2), which confirms our findings based on d-BLEU scores. However, it is still an open question whether human and automatic evaluation metrics are com- plementary or mutually exclusive in measuring the documentlevel translation quality. We will explore more human evalution methods in the future.

heading: 6 Conclusion
text: We provided valuable insights into the performance of LLMs in long-text translation. By creating a comprehensive, multi-domain, and multilingual document-level corpus for training and testing, we have been able to analyze the impact of various extrapolation methods on translation performance and propose a simple yet effective approach. By releasing data, code, and models, we aim to significantly promote further research in this field, ultimately improving the performance of LLMs and MT systems in handling long translation tasks.
Limitations
We list the main limitations of this work as follows: 1. Extrapolation Methods: This study focuses on
enhancing long-text translation through data manipulation and training strategies. Nonetheless, a wide array of alternative approaches merit exploration for boosting model performance. Delving into these methodologies may reveal more efficacious means of bolstering the robustness and precision of predictive models.
2. Evaluation Methods: In this paper, we mainly use the commonly-used automatic metric. The d-BLEU allows for a more nuanced assessment of translation quality, reflecting the unique challenges and requirements of long-text translation. However, it is still an open question whether human and automatic evaluation metrics are complementary or mutually exclusive in measuring the document-level translation quality (Wang et al., 2020, 2023a; Wu et al., 2024b).
Ethics Statement
We take ethical considerations very seriously, and strictly adhere to the ACL Ethics Policy. Most datasets used in this paper are publicly available. Besides, we are the copyright owners of the newly proposed dataset. We ensure that the findings and conclusions of this paper are reported accurately and objectively.

heading: Acknowledgements
text: This work was supported in part by the Science and Technology Development Fund, Macau SAR (Grant Nos. FDCT/060/2022/AFJ, FDCT/0070/2022/AMJ), National Natural Science Foundation of China (Grant No. 62261160648), Ministry of Science and Technology of China (Grant No. 2022YFE0204900), the Multi-year Research Grant from the University of Macau (Grant No. MYRG-GRG2023-00006-FST-UMDF), and the Tencent AI Lab Rhino-Bird Gift Fund (Grant No. EF2023-00151-FST).

heading: A Appendix
text: A.1 Effects of Different Domains and Languages
As shown in Figure 3, the Social and Q&A domains exhibit a more informal structure and length, leading to varied performance across models. Besides, most models consistently excel in the News and Parliamentary domains. However, they underperform in the Novel and Subtitle domains, which are characterized by its extended context. This underscores the challenges LLMs face in long-text translation and motivates our pursuit to further explore translations of longer texts. Concerning languages, most models exhibit consistent performance: they shine in En-De translation while showing relatively subdued results in Zh-En and En-Ru. Therefore, we will delve deeper into the impact of document lengths using the Zh-En Novels dataset.
A.2 About Novels Dataset The novels were translated from one language to another by professional translators, with each novel chapter serving as the translation unit. Due to the absence of sentence-level alignment information—having only chapter-level data—we employed a method that combines automatic alignment with manual proofreading to achieve highquality sentence-by-sentence alignment. For more details, please check GuoFeng Webnovel13 and mZPRT14.
A.3 Results of BlonDe Metric We report results of different methods in Section 5 using the BlonDe automatic metric (Jiang et al., 2022) in Table 8, Table 9, Table 10. BlonDe takes discourse coherence into consideration by categorizing discourse-related spans and calculating the similarity-based F1 measure of categorized spans. There is a certain degree of discrepancy discrepancy between d-BLEU and BlonDe metrics, which potentially provide complementary reference points when measuring the quality of document-level translations.
13https://github.com/longyuewangdcu/ GuoFeng-Webnovel.
14https://github.com/longyuewangdcu/mZPRT.

heading: Future Work
text: We will explore more human evalution methods in the future