Scaling Law for Document Neural Machine Translation
Authors: Zhang Zhuocheng
Shuhao Gu
Min zhang
Yang Feng
Venue: EMNLP 2023 Findings (2023)
Keywords: Document Machine Translation
NMT
Scaling

Abstract: The scaling laws of language models have played a significant role in advancing large language models. In order to promote the development of document translation, we systematically examine the scaling laws in this field. In this paper, we carry out an in-depth analysis of the influence of three factors on translation quality: model scale, data scale, and sequence length. Our findings reveal that increasing sequence length effectively enhances model performance when model size is limited. However, sequence length cannot be infinitely extended; it must be suitably aligned with the model scale and corpus volume. Further research shows that providing adequate context can effectively enhance the translation quality of a document's initial portion. Nonetheless, exposure bias remains the primary factor hindering further improvement in translation quality for the latter half of the document.

Scaling Law for Document-Level Neural Machine Translation
Zhuocheng Zhang 1,2 Shuhao Gu Min Zhang', Yang Feng!.
Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS)
University of Chinese Academy of Sciences, Beijing, China School of Future Science and Engineering, Soochow University, China
zhangzhiochengg222eiccc..ccc
zhangminmtt@hottaiil. com fengyangeict. cn
Abstract
Sample Efficiency
Corpus Size
The scaling laws of language models have played significant role advancing large language models. However, the scaling law document-level neural machine translation remains unclear. order promote the de- velopment of document-level neural machine translation, we systematically examine the seal- ing laws in this field. In this paper, carry out an in depth analysis of the influence among three factors on translation quality: model scale, data scale, and maximum sequence length. Our results indicate that all three factors have significant impact on model performance. In particular, increasing the maximum sequence length effectively reduces the context-relatee errors and improves the overall translation qual- ity. Nevertheless, the sequence length cannot be increased indefinitll,, as the number of pa- rameters limits the optimal sequence length. Specifically, we propose formula describing the empirical scaling law between the model size and the optimal sequence length. Our fur- ther analysis shows that the emor accumulation problem the primary factor that hindering further improvement in translation quality for the document-lewt translation by extending the sequence length.
Number
Translation Quality
Parameters
Optimal
Sequence
Length
Length
Figure I: The key factors affecting the translation qual- ity of document-level neural machine translation.
by leveraging contextual information beyond the sentence being translated. Different from previ- ous sentence-by-sentence DNMT methods, which employ an additional context encoder to capture the discourse information (Chen et al., 2021; Mi- culicich et al., 2018), recent studies have shown that the Transformer (Vaswani et al., 2017) is able to translate multiple sentences directly (Liu et al.. 2020; Bao al., 2021; Sun al., 2022), referred document-byy byy-nentt---dooumentt translation method. These works demonstrated that increasing the gran- ularity translation from sentence to document is able to further reduce the -ontext-dependent errors encountered by sentence-level translation methods, such as misinterpretation of pronouns (Müller et al., 201 8) and lexical inconsistency (Jiang et al., 2021) 1).
1 Introduction
Although has been widely explored, the mecha- nism of DNMT has been fully investigated. For example, previous document- couument-by-documentt DNMT methods are trained on a fixed maximum sequence length, i.e. the document longer than the maximum sequence will be split into smaller segments, and whether further increasing the maximum sequence length can continue improve the performance of the model still remains to be investigated. Addition- ally, the sequence length grows, the difficulty of modeling the contextual information increases further, and whether more training data is needed remains to be explored. Furthermore, the scale
Neural machine translation (NMT) (Bahdanau al., 2014) has made great progress in recent years (Barrault al., 2020; Guo et al., 2022). However, the input text exceeds single sen- tence, sentence-level NMT methods will fail to capture discourse phenomena, such as pronomi- nal anaphora, lexical consistency, and document coherence. In contrast, document-level neural ma- chine translation (DNMT) (Maruf et al., 2021) aims to improve translation consistency and coherence
"Corresponding author.

---

model parameters among the most crucial fac- tors affecting the quality of sentence-level NMT methods and widely studied by the scaling law of neural machine translation (Ghorbani et al., 2021; Ott et al., 2018), but how this factor affects the performance of the DNMT model remains unclear. Finally, it remains open question whether these factors will interact with each other.
posed hierarchical attention mechanism cap- ture the discourse information while Maruf et al. (2019) employed a selective attention module to select the most relevant information in the context. Lupo et al. (2021) further improved these methods by splitting the sentence into smaller segments to overcome the training sparsity problem. Recently, Tiedemann and Scherrer (2017); Ma et al. (2020) suggest that Transformer has the ability trans- late multiple sentences directly, and this document- by-document paradigm further reduce the context related errors. However, the DNMT model may fail to converge the corpus size limited and the sequence length is long (Liu et al., 2020). Bao et al. (2021); Zhang et al. (2020) proposed to employ attention masks to prevent the overfitting problem, while Sun al. (2022) proposed multi- multi-resolution training strategy smooth the training difficulty of the DNMT model. However, these methods are all based on a fixed sequence length. Whether further increasing the maximum sequence length can con- tinue to improve the translation quality still remains to be investigated.
To answer the questions above, we conduct mas- sive experiments, specifically investigating the im- pacts of three key factors: corpus size, model scale, and maximum sequence length. In addition, we explore the joint effect of these factors. Our find- ings suggest that all three factors are closely related to the performance of the DNMT model and en- larging the maximum sequence length effectively improves the ability of the DNMT model to capture the contextual information. However, the sequence length can not be increased infinitely as the trans- lation quality begins to drop when the sequence length exceeds a certain limit. Surprisingly. our experimental result shows that this limitation is pro- portional to the logarithm of the number of model parameters. Our further analysis demonstrates that determining the optimal maximum sequence length requires striking a balance between capturing ad- ditional contextual information and managing ac- cumulated errors during auto-regressive decoding. This finding highlights the importance of future research in exploring effective solutions to mitigate this challenge. Furthermore, we find that although the DNMT model is initialized using pretrained sentence-level translation model, increasing the document-level corpus size can further improve the translation quality. Notably, this improvement increases as the number of model parameters in- creases. As Figure concludes, our study indicates that the number of model parameters affects the translation quality of DNMT by affecting two key factors: sample efficiency and optimal sequence length. As the number of model parameters in- creases, both the sample efficiency and the optimal sequence length grow, leading the performance improvement of the DNMT model.
2.2 Scaling Law
The Scaling law of language models has been widely studied, and have played a significant role in advancing large language models. Kaplan et al. (2020) proposed that the perplexity of the language model is inversely proportional to the logarithm of the size of the training data and the number of the model parameters. Hoffmann et al. (2022) fur- ther optimized this scaling law by taking larger corpus size into consideration. Recently, Touvron et al. (2023) proposed that the inference overhead should also be considered by the scaling law. In the field of neural machine translation, Ott et al. (2018) proposed that enlarging the data size further improve the translation quality of NMT models, while Ghorbani al. (2021) further pro- posed that the cross-entropy loss is also governed by the power law model size and data size. These works have greatly accelerated the development of large language models and neural machine trans- lation (Wang and Komatsuzaki, 2021; Scao et al., 2022; Costa-jussà et al., 2022). However, the ex- isting scaling law of machine translation is based on the cross-entropy loss of the model, which is not directly related to the translation quality. In addition, the effect of contextual information has not been considered in the scaling law machine
2 Related Work
2.1 Document-level Neural Machine Translation
Document-level neural machine translation has wit- nessed significant advancements in recent years (Maruf et al., 2021). Miculicich et al. (2018) pro-

---

Architecture
Params
Params w/o Embedding
MACs/ Token
MACs/ / Token
Transformer-small
48.32M
31.54M
48.7M 78.7M
67.2M
Transformer-baa base
60.92M
44.14M
61.5M
79.8M
Transformer-big
209.92M
176.37M
210.9M
247.5M
Transformer-base-3
38.85M
22.07M
39.0M
48.2M
Transformer-base-9
82.99M
66.21M
83.7M
111.2M
Transformer-basee 105.06M
88.28M
105.9M
142.6M
Table 1:
The statistics of the used model. The first column is the model architecture. The second and the third columns show the total number of parameters and the number of parameters without embedding. The fourth and fifth columns show the number of MACs per token at sequence length equal to and 1024, respectively.
translation. Therefore, it is necessary to explore the scaling law of DNMT.
Subtitles (Lison and Tiedemann, 2016), Europarl (Koehn, 2005), NewsCommentary, and Tilde- MODEL (Rozis and Skadinš, 2017), from OPUS?. The only pre-processing step take is tokenizing the dataset using SentencePicce' with joint 32k vocabulary. To perform early stopping and deter- mine the hyperparameters, we randomly extracted 100 documents from the dataset as the validation set. For the test set, we utilize newstest2020 from the WMT competition. The statistics of the com- bined dataset are shown in Table 2. In order to further investigate the impact of data scale the performance of DNMT, randomly sampled sub- sets of different sizes from the combined dataset to serve as our training sets. The details of the sampled subsets are shown in Appendix A. In our experiments, we segment the document into dif- ferent lengths, ranging from 32 to 1024 tokens, to investigate the impact of maximum sequence length on the performance of DNMT. The length distribution of segmented datasets is shown in Appendix B.
3 Experimental setup
3.1 Architecture
We the Transformer (Vaswani al., 2017) ar- chitecture as our DNMT models. To bring our work closer to cutting-edge research, we leverage several improvements on the standard Transformer archi- tecture, including the pre-norm layer normalization (Xiong et al., 2020) and the Gaussian Error Lin- ear Unit (GELU) activation function (Hendrycks and Gimpel, 2016). To further investigate the im- pact of different model structures the quality of document-level translation, we experimented with various model widths and depths. Specifically, the small, base, and big models have layers in both the encoder and decoder. terms of model width, the small and base models employ an embedding dimension of 512, whereas the large model has an embedding dimension of 1024. Additionally, we employ dimensions of 1024, 2048, and 4096 for the small, base, and big models, respectively, in the feed-forward network. To investigate the model depth on translation quality, we vary the number layers in the base model from 3 to 12. We use deepspeed to profile the number of parameters and MACs per token for our model. The statistics of our models are shown in Table 1.
Subset
#Words
#Sents
#Does
EUbookshop
172M
9.3M
14k
OpenSubtitles
136M
23M
29k
Europarl
42M
1.8M
10k
NowsCommentary
8.8M
0.4M
9.7k
TildeMODEL
23M
1.6M
52k
Combined 382M
36M
115k
3.2 Dataset
Compared to sentence-level bilingual corpus, document-level bilingual datasets limited in size (Chen al., 2021). To maximize the cor- pus size, we combine data from 5 high-quality document-level English-German datasets, includ- ing EUbookshop (Skadin' et al., 2014), Open-
Table 2: The statistical results of the datasets used in our experiments in terms of word count, sentence count and document count.
hhtps://opus.nlll.ee

---

3.3 Training
Previous researches indicate that training a document-level model based on sentence-level model yields better results compared to training a document-level model from scratch (Bao et al.. 2021; Miculicich et al., 2018). Therefore, adopt two-step training strategy. First, train a sentence-level model by splitting the documents in the corpus into sentences. Then, we finetune the sentence-level model into a DNMT model using the document-level corpus. During training, we use the Adam optimizer (Kingma and Ba, 2014) 10 and
33
@
31
ambed
29.
w/o embed
29
20
10
Parameters
(a) Effect model parameters
(h) Effeett of model shape
Figure 2: The effect of model shape on the performance of DNMT.
with 31 = 0.9, A2 0.98, the learning rate is scheduled using the inverse square root method (Vaswani al., 2017) with warmup steps of 4000. We use the label smoothed cross-cntropy loss (Szegedy et al., 2016) with the smoothing value 0.1. We conduct experi- ments on NVIDIA A100 GPUs with a maximum of 4096 tokens per GPU. We use half-precision float (fpl6) to accelerate the training process (Ott et al., 2018). The dropout and the attention dropout are tuned for every model from {0.1, 0.2, 0.3} and (0.0, 0.1, 0.2, 0.3), respectively. We conduct experiments using the fairseq toolkit (Ott al.. 2019).
4.1 Effect of Single Factor
We first investigated the independent impact of two factors, the model and the data, on the performance of DNMT.
Effect of model size. To figure out the most im- portant factor affecting translation quality, train DNMT systems with different model sizes on the combined dataset with a maximum sequence length of 5 122. As shown Figure 2, the performance of DNMT increases as the model size increases. How- ever, the d-BLEU score does not increase linearly with the model depth or the model width. Similar to the previous study (Kaplan et al., 2020), observe approximately linear relationship between the model's performance and the logarithm of the pa- rameter count. Furthermore, this linearity is more closely related the number of non-embedding pa- rameters rather than the total number of parameters. These findings suggest that compared to the model width and depth, non-embedding parameters better reflects the translation quality of the DNMT model.
3.4 Evaluation
In order to make the number of generated sentences matches that of the source document, we add an EOS token at the end of each sentence. We use beam-search with beam size 5 for all our experi- ments. Following previous study (Liu et al., 2020; Bao et al., 2021). we employ document BLEU (d-BLEU) as our document-level metries, while sentence BLEU (s-BLEU) as our sentence-level metries. BLu (Post, 2018) was used to calculate the above metrics. In addition, we also employ COMET-22 (Rei et al., 2022) and chrl (Popović, 2015) to evaluate the quality of gener- ated sentences. We conduct each experiment three times with different random seeds, and present the average score for analysis.
Effect of Data Scale. To investigate the effect of the data scale on the performance of DNMT, we conduct experiments different sized subsets of the combined dataset with a maximum sequence length of 512. As shown in Figure 3, the perfor- mance of DNMT increases as the data scale in- creases. Surprisingly, the evaluation shows that regardless of how the model changes, when the number of sentence pairs exceeds 4 million, the performance of the DNMT model will surpass the corresponding sentence-level model. This finding gives us new understanding of the relationship between data scale and model performance. In ad- dition, the overall trend is consistent with previous studies that there is a power law between model performance and data size (Ghorbani et al., 2021).
4 Experiments
In this section we present the results of our experi- ments. First, analyze the effect of single factor the translation quality of DNMT in Section 4.1. Then investigate the joint effect of multiple fac- tors in Section 4.2.

---

34
33
32
0-8LEU
30
31
30
28
2. 2.8e-05
6.9e+06
4.9e+05
1.16+07
0.
0.0
28
1.4e-06
3.3e+
2. 0.
sent
27
small
big
base.
4.0c-06
26
base
baiee_3
bare
0.6
1.2
1.4
200
400
600
800
1000
Params
Sequence Length
Figure 3: Document BLEU the number of pa- rameters. The solid lines represent the performance of document-level models at different data scales, which is measured by the number of sentence pairs, while the black dotted line represents the performance of the sentence-level model.
Figure 4: The joint effect of maximum sequence length and model shape on the performance of DNMT. Differ- ent colors represent different model shapes, while the marker represents the estimated optimal sequence length for different model shapes.
differs from the previous studies (Press et al., 2021; Beltagy et al., 2020), as they demonstrate that the model's performance consistently increases with an expanding context. We proposed that this is due to the model being influenced by erroneous context in the generation process, i.e. the error accumulation problem, and we further analyze this phenomenon in Section 5.2. addition, we find that the rela- tionship between optimal sequence length L and the parameters of the model can be expressed by the following equation:
4.2 Effect of Multiple Factors
To study the interactions between factors, we keep the data scale, model scale, and sequence length fixed separately and investigate the joint impact of the other two factors on translation quality. Specifically, we used the entire dataset to exam- ine the relationship between model scale and max- imum sequence length, fixed the model scale as the Transformer-bas scale and maximum sequence length on translation quality, and held the maximum sequence length at 512 to investigate the relationship between model scale and data scale.
study the impact of data
L (Ng(N)= +
where N the number of parameters, a and are constants. We estimate the value a and b by fitting the data in Figure 4. The estimated value 111, while the estimated value of b -1399. This indicates that for every increase of 77 tokens in the optimal sequence length, the model parameter size needs to be doubled. This finding is consistent with the previous studies, indicating that suitable sequence length for the transformer-base model is approximately 512.
Joint Effect of Maximum Sequence Length and Model Scale. As shown in Figure 4, the BLEU score generally increases with the sequence length for various model shapes. However, we observe that there is upper limit to this performance im- provement as the length increases. For example, when the sequence length is 512, the BLEU score the transformer-base 31.55. However, when the sequence length reaches 1024, the BLEU score decreases to 30.59. This indicates that there ex- ists an optimal maximum sequence length for the DNMT model. In addition, we find that this opti- mal sequence length closely related to the model size. To illustrate this relationship more clearly, estimate the location of the optimal sequence length for different model sizes. As the star marker shown in Figure 4, the optimal sequence length and the model scale are positively correlated. This finding
Joint Effect of Maximum Sequence Length and Data Scale. To figure out whether the data scale affects the ability of the transformer model to han- dle long text, conduct experiments on different subsets of the combined dataset with different max- imum sequence lengths. As shown in Figure 5, under different data scales, the performance of the model generally increases and then decreases with the increase in sequence length. Similar to our find-

---

small
base_3
31
base
base_ 9
30
big
base.
29
d-BLEU
2.8e005
6.96-06
26
8.4
2.0e-07
25
1.4e+06
3.3e-07
2.c+066
sent
24
4.0e108
200
600
800
1000
Segment Longth
Corpus Size
Figure 5: The joint effect maximum sequence length and data scale on the performance of DNMT. The solid lines represent the performance document-level mod- els different maximum sequence length, while the black dotted line represents the performance of the sentence-level model.
Figure 6: The joint effect of data scale and model shape on the performance of DNMT. The solid lines represent the performance document-level models different data scales, while the dotted line represents the perfor- mance of the sentence- model.
tence pairs exceeds 4 million, the performance of the DNMT model outperforms the corresponding sentence-level model, which is consistent with our findings in Section 4.1. In contrast, when the data scale limited, the models with more layers out- perform the models with wider widths. For ex- ample, the 12-layer transformer-ban model per- forms better than the transformer-large model with layers when the sentence pairs are less than 4 million, even though the number of parameters in the transformer- model is twice that of the 12-layer transformer-base model. We believe that this is because when the document-level data is limited, the performance of the DNMT model de- pends more on the parameter initialization, which dominated by the sentence-level model. This is consistent with previous studies that sentence-level NMT models are more dependent on the model depth rather than width (Mehta et al., 2020; Wang et al., 2022). However, when the data is sufficient, model with larger number of parameters will benefit more than model with more layers. This indicates that the model's ability to capture context also related to the model width.
ings in Section 4.1, when the data scale large enough, the performance of DNMT will surpass the corresponding sentence-level model. However, although the DNMT models are initialized using pretrained sentence-level model, the d-BLEU score is lower than the sentence-level NMT model if the data size is limited. addition, we find that the d-BLEU score is below the sentence-level baseline when the maximum sequence length 64, even with an adequate amount of data. This is due to the early stopping mechanism terminating the training prematurely when the sequence length is too short. When the maximum sequence length is equal to 128, the BLEU score rises. Surprisingly, the max- imum sequence length of 256 is turning point, where the BLEU score reaches the maximum when the data scale is limited. This indicates that when the document-level corpus size scarce, we can set the maximum sequence length to 256 in order to maximize the translation quality. In addition, when the maximum sequence length exceeds 512, we observed that the BLEU score shows declin- ing trend even with further increases in the data scale. This finding further confirms our previous conclusion that there exists an optimal maximum sequence length for given model.
Discussion
5.1 The Correlation Between BLEU Score and Cross-Entropy Loss
Joint Effect of Data Scale and Model Scale. As shown in Figure we train DNMT models with different data scales and model sizes with max- imum sequence length of 512. The performance DNMT increases consistently as the data scale increases. addition, when the number of sen-
As mentioned by Post and Junczys-Dowment (2023), The quality assessment of document trans- lation is challenging. In this paper, we demonstrate that the cross-entropy (CE) loss DNMT model

---

position in the document, Then we evaluate each subsets via COMET-22 (Rei al., 2022). The distribution of the number of sentences differ- ent locations is shown in Figure 10. As shown Figure 8, with the help of contextual information, the translation quality of DNMT model is slightly better than that of the sentence-level model at the beginning of each documents. However, this advan- tage soon diminishes the translation proceeds. From the fifth sentence on wards, there has been growing gap translation quality between the DNMT model and the sentence-level translation model, which clearly indicates the error accumu- lation process. This finding demonstrate that the optimal sequence length is actually the trade-off between extra contextual information and the ac- cumulated errors. Interestingly, the gap between the sentence-level model and the document-level model is larger when the model size smaller, in- dicating that larger model has a stronger ability to resist the error accumulation problem.
Inference Length: 64 R'=0.39
38
Interence Length: 82-0.34
A
Inference Length: 1024 R²=0.26
36-
d-8LEU
3.1
3.2
3.3
3.5
3.7
Cross Entropy Loss
Figure 7: The correlation between cross-entropy loss and d-BLEU score of DNMT model on the validation set. Different colors represent different maximum se- quence length, and the solid line represents the linear regression of this correlation. The coefficient of deter- mination are reported in the legend.
does not adequately capture the quality of trans- lation. To further investigate the relationship be- tween the CE loss and the translation quality, we plot the CE loss and the d-BLEU score DNMT model on the validation set. As shown in Figure 7, the CE loss and the d-BLEU score of DNMT model are weakly correlated, indicating that the loss alone fails to fully depict the translation quality. This finding is inconsistent with the previous study (Ghorbani al., 2021), which has indicated a sub- stantial association between CE loss and translation quality. In addition, we calculate the coefficient of determination (R²) of the linear regression model. The R.2 decreases as the maximum sequence length grows. We suppose that this is because document- by-document DNMT requires generating longer text, which is prone to the eиror accumulation problem, which will be discussed in Section 5.2.
Context Length
Accuracy
0
50.06
60
67.75
120
81.89
250
82.26
500
82.38
750
82,49
1000
81.96
Table 3: The accuracy DNMT model on ContraPro test suite. Context length is the number of tokens in the context. 7ero context length indicates that the result is from the sentence-level model.
5.3 The Contrastive Experiment
In addition evaluating the overall translation quality, we also conduct experiments on ContraPro (Müller et al., 2018), a large contrastive test suite extracted from OpenSubtitles 2018 (Lison and Tiedemann, 2016), to evaluate the accuracy of translating the English word "it". As shown in Table 3, all the DNMT models outperform their sentence-leve counterparts, clearly indicating that the DNMT model is able to capture contextual in- formation. As the available context increases, the accuracy of the test suite increases in fluctuation. When the context length exceeds 500, the accuracy of the DNMT model stops increasing, and remains at a high level. This tendency is not completely con-
5.2 The Error Accumulation Problem
To gain a more comprehensive understanding the underlying reasons for the existence the optimal maximum sequence length and the failure of CE loss to adequately reflect the translation quality, we analyze the translation quality of sentences in differ- ent positions in the documents. We selectively ex- tract subset from the validation set, where DNMT models under-perform the sentence-level counter- parts. Then we employ both sentence-level mod- els and document-level models translate those documents. We split the translated segments into sentences, and gather them into a subset by there

---

big
0.85
0.80
COMET-22
0.70
0.65
sent model doc model
0.60
40 Sentence Position
0
40 Sentence Position
60
Sentence Position
Figure 8: The COMET score of the translated sentences at different positions the document. The solid lines represent the estimated trend line of the COMET score. As the model size grown, the gap between the DNMT model and the sentence-level model becomes smaller.
sequence length when finetuned document-level corpus. Thus it possible apply DNMT model to both document-level and sentence-level trans- lation tasks. However, when the inference length exceeds the training length, the performance expe- riences a sharp drop. We considered that this is because vanilla Transformer lacks length extrapo- lation ability (Press al., 2021), I), which needs to be further investigated in the field of document-level translation.
30
G-BLEU
15
Training Length:
Length:
Training Length: 256
Training Lengeh: 512
Training Length: 768
0
800
1000
Decoding Length
5.5
The computation overhead of DNMT
As the quadratic computational complexity of the attention mechanisms, the computational require- ments of the DNMT models are higher than those of sentence-level models. We report the MACs Operations) per token re- quired by the DNMT models in Table 1. Interest- ingly, we find that even though the sequence length increases from 32 to 1024, which is much longer than the optimal sequence length of transformer- big, the MACs per token increase by only 40 per- cent. This finding indicates that the computation overhead of the DNMT models during inference not much higher than the sentence-level models, but the errors encounter by sentence-level transla- tion reduced significantly. Thus it is efficient to employ the DNMT model in practice. We have placed a more detailed derivation in Appendix C.
Figure 9: The generalization ability of DNMT model. The models are trained with different maximum se- quence length.
sistent with the trend of d-BLEU score, showing that the overall translation quality is also influenced by factors other than the discourse context.
5.4 The Generalization Ability
In practice, the lengths of input and output sen- tences may differ from those encountered during the training phase. Therefore, proficient DNMT model should be capable of handling documents various lengths and produce accurate translation results. To investigate this, we segment the doc- uments in the test set into different lengths, and evaluate the DNMT models on the segmented test sets. According to Figure 9, when the sequence length during inference is shorter than that during training, the d-BLEU score of DNMT model de- creases mildly. This observation shows that Trans- former (Vaswani et al., 2017) is capable of handling documents various lengths within the maximum
Conclusion
In this paper, explore the scaling law of document-level neural machine translation. Our findings suggest that the corpus size, model scale and the maximum sequence length all have sig- nificant impacts on the translation quality of the

---

model. Different from the scaling law of sentence- level neural machine translation, we find that their exists an optimal maximum sequence length for a given DNMT model, and the optimal maximum sequence length is strongly correlated with the num- ber of non-embedding parameters of the DNMT model. We further formulate this correlation with simple power law. Our additional analysis indicates that the error accumulation problem is the primary factor that hindering further improvement in trans- lation quality for the document-level translation. In future work, we will investigate how to overcome the error accumulation problem, and extend the op- timal sequence length of document-by-doeunee translation.
Iz Beltagy, Matthew Peters, and Arman Cohan, 2020. Longformer: The long- documentt transformer.
Linging Chen, Junhui Li, Zhengxian Gong, Boxing Chen, Weihua Luo, Min Zhang. and Guodong Zhou. 2021. Breaking the corpus bottleneck for context- neural machine translation with cross-task pre- training. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 1ith International Joint Conference Nutu- ral Language Processing (Volume I: Long Papers), pages 2851-2861.
Marta R Costa-jusat jussä, James Cross, Onur Çelebi, Maha Elbayad. Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. numan-centered machine translation. arXiv preprint arXiv:2207.04672.
language left behind: Scaling
7 Acknowledgement
Behrooz Ghorbani, Orhan Firat, Markus Freitag, Ankur Bapna, Maxim Krikun, Xavier Garcia, Ciprian Chelba, and Colin Cherry. 2021. Scaling laws for neural machine translation. arXiv preprint arXiv:2109.07740.
Our research requires a large number of experi- ments. Thus, we would like to express our gratitude to the ICT computing platform and the technical service team for providing GPU resources.
Dengji Guo, Zhengrui Ma, Min Zhang, and Yang Feng. 2022. Prediction difference regularization against perturbation neural machine translation. In Pro- ceedings of the 60th Annual Meeting of the Associa- tion for Computational Linguistics (Volume I: Long Papers), pages 7665-7675
Furthermore, thank the anonymous reviewers for their thorough review and valuable feedback.
Limitations
Although we have conduct massive experiments to explore the scaling law of document-leve neural machine translation, there still some limitations in our work: (1) over four hundred models were trained, resulting in high energy consumption; (2) due to the limitation the data size, we are unable to conduct larger scale experiments.
Dan Hendrycks and Kevin Gimpel. 2016. sian error linear units (gelus). arXiv preprint 16Xiv:160.008475.
Gaus-
Jordan Hoffmann, Sebastian Borgeaud, Arthur Men- sch, Elena Buchatskaya, Trevor Cai, Eliza Ruther- ford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbi, Aidan Clark, al. 2022. Train- ing compute-optimal large language models. arXiv preprint ArXiv 15556.
References
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben- gio. 2014. Neural machine translation by jointly learning align and translate. arXiv preprint arXiv:1409.0473.
Yuchen Elcanor Jiang, Tianyu Liu, Shuming Ma, Dong- dong Zhang, Jian Yang, Haoyang Huang, Rico Sen- nrich, Ryan Cotterell, Mrinmaya Sachan, and Ming Zhou. 2021. Blonde: An automatic evaluation met- ric for document-level machine translation. arXiv preprint arXiv: 103. 11878.
Guangsheng Bao, Yue Zhang, Zhiyang Teng, Boxing Chen, and Weihua Luo. 2021. G-transformer for document-level machine translation. arXiv preprint
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray. Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint 200v02001..8861.
arXiv:2105.1476.
Loic Barrault, Magdalena Biesialska, Ondrej Bojar, Marta Ruiz Costa-jussà, Christian Federmann, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Matthias Huck, Eric Joanis, Tom Koemi, Philipp Kochn, Chi kiu Lo. Nikola Ljubesic, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshi- aki Nakazawa, Santanu Pal, Matt Post, and Marcos Zampieri. 2020. Findings of the 2020 conference on machine translation (wmt20). Conference on Machine Translation.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint
arXiv: 1412.6980.
Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. Proceedings of machine trunslation summit x: papers, pages 79-86.

---

Pierre Lison and Jörg Tiedemann. 2016. Opensub- titles2016: Extracting large parallel corpora from movie and tv subtitles.
Machine Translation: Research Papers. pages 186- 191, Belgium, Brussels. Association for Computa- tional Linguistics.
Yinhan Liu, Jiatao Gu. Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre- training for neural machine translation. Transac- fions of the Association for Computational Linguis- fics, 8:726-742.
Matt Post and Marcin Junczys- Doworyy..DDmuttt... 2023. Escap- ing the sentence-level paradigm in machine transla- tion. ArXiv:2304. 12959999999999 [cs].
Ofir Press, Noah A. Smith, and Mike Lewis. 2021. Train short, test long: Attention with linear biases enables input length extrapolation.
Lorenzo Lupo. Marco Dinarelli, and Laurent Besacier. 2021. Divide and rule: Effective pre-training for context-aware multi-encoder translation models. arXiv preprint 103.::1033.7755
Ricardo Rei, José GC De Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André FT Martins. 2022. Comet-22: Unbabel-ist 2022 submission for the metrics shared task. In Proceedings of the Sev- enth Conference Machine Translation (WMT), pages 578-585.
Shuming Ma, Dongdong Zhang, and Ming Zhou. 2020. A simple and effective unified encoder for document- level machine translation. In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 2505-3511.
Roberts Rozis and Raivis Skadinš. 2017. Tilde model- multilingual open data for eu languages. In Proceed- ings of the 21 Nordic Conference on Compatational Linguistics, pages 263-265.
Sameen Maruf, André FT Martins, and Gholamreza Haffari. 2019. Selective attention for context- aware neural machine translation. arXiv preprint arXiv:1903.087
Teven Le Scao, Angela Fan, Christopher Akiki, El- lie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: 176b- parameter open-access multilingual language model. arXiv preprint 2xXh:2211.005000
Sameen Maruf, Fahimeh Saleh, and Gholamreza Haffari. 2021. A survey on document-level neural machine translation: Methods and evaluation, ACM Comput- ing Surveys (CSUR), 14(2):1-36.
Raivis Skadinš, Jörg Tiedemann, Roberts Rozis, and Daiga Deksne. 2014. Billions of parallel words for free: Building and using the eu bookshop corpus. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14), pages 1850-1855.
Sachin Mehta, Marjan Ghazvininejad, Srinivasan Iyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2020.
Delight: Deep and leight transformer. arXiv preprint 2xXv:2008.00622.
Lesly Miculicich, Dhananjay Ram, Nikolaos Pappas. and James Henderson. 2018. Document-level neu- ral machine translation with hierarchical attention networks. arxiv preprint 100.0000.
Zewei Sun, Mingxuan Wang, Hao Zhou, Chenggi Zhao, Shujian Huang. Jiajun Chen, and Lei Li. 2022. Re- thinking document-level neural machine translation. In Findings of the Association for Computational Linguistics: ACL 2022, pages 3537-3548.
Mathias Müller, Annette Rios, Elena Voita, and Rico Sennrich. 2018. A Large-Scale Test Set for the Evalu- ation -owae Pronoun Translation in Neu- ral Machine Translation. In WMT 2018, Brussels, Belgium. Association for Computational Linguistics.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. In Proceedings of the IEFE conference computer vision and pattern recognition, pages 2818-2826.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonsirations.
Jörg Tiedemann and Yves Scherrer. 2017. Neural machine translation with extended context. arXiv preprint arXiv: 1708.0593.
Myle Oit, Sergey Edunov, David Grangier, and Michael Auli. 2018. Scaling neural machine translation. ArXiv, abs/ 08//1806.00187.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet. Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. arXiv preprint ar arXiv: 13971.
Maja Popović. 2015. chrf: character n-gram f-score for automatic mt evaluation. In Proceedings of the tenth workshop on statistical machine translation, pages 392-395.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is
Matt Post. 2018 A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on

---

All You Need. arXiv:1706.03762 [es]. ArXiv: 1706.03762.
Ben Wang and Aran Komatsuzaki. 2021. GPT-J- 6B: A 6 Billion Parameter Autoregressive Lan- guage Model. https: //g thub. lol/ mesh-transformer--D
Hongyu Wang. Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, and Furu Wei. 2022. Deepnet: Scaling transformers 1,000 layers. arXiv preprint
Ruibin Xiong, Yunchang Yang. Di Hc, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. 2020. layer normalization in the transformer architecture. In In- ternational Conference Machine Learning, pages 10524-10533. PMLR.
Pei Zhang, Boxing Chen, Niyu Ge, and Kai Fan. 2020. Long-short term masking transformer: A simple but effective baseline for seuuumnt--leeel neural machine translation. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 1081-1087, Online. Association for Computational Linguistics.

---

A The Statistics of Sampled Datasets
mension is 512. There are 8 attention heads, and each attention head has dimension of 64. The feed-forward layer dimension 2048, and there are encoder layers and 6 decoder layers. We as- sume that the sequence length of the source and target document is both n. and the vocabulary size is 32768. Then we can estimate the computational cost of the model in Table 5.
To explore the relationship between translation quality and the data size in Section 4, we sampled the combined dataset with different sizes. The size of the subset is determined by the number of doc- uments, and ensure experiment reproducibility, each size was sampled three times for averaging the results. The statistics of the sampled datasets are shown Table 4.
As shown in the Table 5, the total computational cost of the Transformer base is n² 18432 6081 1456, and the coefficient of the first-order term regarding n is much larger than the coefficient of the second-order term. Therefore, the growth of computational cost is approximately linear with respect to when n is small. When substituting sequence lengths 32 and n 1024 into the above equation, the total computational cost and per-token computational cost are shown in Table 6. As demonstrated in Table 6, although the se- quence length has increased significantly, the com- putational cost per token have not increased signifi- cantly. Furthermore, for larger-scale models, due to their expanded hidden state dimensions, the con- tribution of the linear term will be more significant, which further reduce the impact of sequence length the computational cost per token.
150
100
50
0
20
40
60
Position
Figure 10: The distribution of the number of sentences different locations.
B The Length Distribution of The Split Dataset
D
The Experimental results on Chinese to English dataset.
To conduct experiments with different maximum sequence length, we split the documents into seg- ments with different segment lengths. However, due to the document boundary restrictions, we can- not ensure that all segments have the same length. As shown in Figure 11, the length distribution of the maxi-
To further validate our experimental findings, we collected over 10 million parallel sentences from Chinese English and conducted a repeat of our experiments on this dataset. The composition and scale of this dataset are presented in Table The experimental results, as shown in Figure 12, demon- strate that the overall trends in Chinese-Engliie dataset are consistent with the conclusions we ob- tained in the English-German dataset. However, there is one significant difference: the optimal quence length in this dataset is considerably shorter than in the Anglish-German dataset. We believe that the optimal sequence length also correlated with the data quality, language characteristics, and domain of the dataset. We will leave this issue for future research.
the training set becomes more mum sequence length increases, which allows the model translate the sentences of arbitrary length within the maximum sequence length.
C Estimation of Computational Complexity for Transformer Models
To gain a clearer understanding of how sequence length affects the computational cost of the model, we take the Transformer base as example and provide detailed derivation of the computational requirements for its forward propagation process. To simplify the derivation, we ignore the compu- tational cost of Layer Norm, residual connection, bias, and the softmax. The embedding dimension of the base model 512 and the hidden state di-

---

#Lines
#Tokens
Subset Number
2
1
2
3
0.28
0.29
0.27
4.29
4.27
4.08
2
0.52
0.49
0.48
8.13
8.05
7.62
3
0.82
0.86
0.85
13.12
13.92
12.85
1.39
1.42
1.40
21.77
22.28
21.48
2.37
2.41
2.30
37.70
39.21
36.30
4.00
4.06
3,99
62.64
64.93
62.83
6.81
6.88
6.87
109.17
111.99
109.54
8
11.47
11.39
11.51
183.84
181.83
182.84
9
19.53
19.66
19.58
312.61
313.62
310.31
Table 4: The statistics of the sampled datasets in millions. Subset number indicates the number subsets in the sampled dataset. #I ines and #Tokens indicate the number of lines and tokens the sampled dataset, respectively.
Module
Compatational Cost
Number of the Module
Corresponding Operation
Embeediing
a 512
The embedding and positional encoding layers both encoder and decoder ourput projection layer for generating The feed forward layers both encoder and decader layers
Ourput Projection
512 32768
outpor prohabilities
Feed Forward
512 2048
12
Attention Project
a 5112 64 8
18
The query, key, value, and output projection in the multi head attention
Dox Product
The doe product between key and query as well as attention weight and value computation
18432 60819456
Transformer
Table 5: The estimation of computational complexity for Transformer base model. We assume that the sequence length of the source and target document is both n and the vocabulary size is 32768.
Computational Cost
Computational Cost token
32
1965096960
61409280
1024
$1606475776
79693824
Table 6: The computational cost for Transformer base model when n 32 and 1024.
Subset
#Words
#Sents
#Does
UNPC
417.2M
17.4M
94.7k
NewsCommentary
5.3M
0.1M
7.8k
Combined
422.5M
17.5M
102.5k
Table 7: The statistical results of the Chinese Chinese-+Englitt datasets used our experiments in terms of word count, sentence count and document count.

---

Maximum Sequence:
Maximum Sequence: 128
60
Percent
Percent
20
Maximum Sequence: 256
Maximum Sequence: 512
12.5
10.0
15
Percent
Percent
7.5
10
2.5
0.0
Maximum Sequence: 768
Maximum Sequence: 1024
6-
Percent
Percent
4-
4
200
1000
200
1000
Figure 11: The length distribution of the segmented dataset.
12
Lengit
Copaa Size
The joint effect of maximum sc- (b) joint effect maximum (c) The joint effect of data scale and quence length model shape.
quence length and data scale. model shape.
Figure 12: The experimental result on Chinese-+English dataset.


Reviews:
[{"review_id": "s7e1QUDSog", "reviewer": "Reviewer_38v3", "summary": "", "questions": "A: How was the model for the optimal maximum sequence length estimated (Equation 1) and are there any metrics that confirm its reliability?\n\nB: Concerning the analysis in Section 5.2: What maximum sequence length did you use for these experiments? Did you check the number of training examples per sentence position (i.e. how many training examples actually consist of x sentences) and if yes, how does the distribution look like?\n\nC: Concerning the analysis in Section 5.3: Why does the accuracy decrease from context length 60 to 120 and from 120 to 250? \n \nD: I’m surprised about the finding in Section 5.5. What is the setup of this experiment? How did you make sure that the numbers between maximum sequence length 32 and 1024 are comparable?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thorough review and constructive feedback. In response to the concern raised about our paper, we summarize our response as follows.\n\n## Q1: The Scaling behavior of the document-level machine translation system is not crucially different from the sentence-level machine translation system, which has already been studied.\n\nThis paper provides a systematic investigation of the scaling laws in document-level machine translation. In comparison to the scaling laws observed in sentence-level translation models, we have derived several significant and distinctive findings:\n1. We have identified **the limitations of employing Perplexity (PPL) as an evaluation metric for machine translation tasks**, particularly in the document-level machine translation scenario. Our analysis highlights that PPL-based evaluation metrics fail to accurately reflect translation quality. Furthermore, we conducted an in-depth examination of the correlations among different evaluation metrics. The experimental results indicate that PPL gradually deviates significantly from other metrics as sequence length increases;\n2. Prior work on document-level machine translation has presented two viewpoints. Some studies have indicated that the effective information for improving translation quality is concentrated in the context of approximately 1-3 sentences around the sentence to be translated. On the other hand, another perspective suggests that the model can benefit from contexts of arbitrary lengths. In this paper, we systematically investigate this issue. Through experiments conducted on the ContraPro dataset, we find that the Doc2Doc document-level machine translation model is indeed capable of capturing relevant information from a broader context. **However, when evaluating the model using conventional translation metrics, such as BLEU, we identified an optimal sequence length that corresponds to the best translation quality. Deviating from this optimal length, whether longer or shorter, leads to a degradation in translation quality.** This paper makes the pioneering observation of this phenomenon in document-level machine translation and provides a method, based on model scale, to estimate the optimal context length, which holds significant implications for document-level translation tasks;\n3. We further explore the underlying reasons behind the optimal context length of document-level machine translation models. It is not the case that more context information is always better, nor is the crucial information solely confined to the preceding 1-3 sentences. Instead, as the available context length increases, the model gains enhanced disambiguation capabilities. **However, due to the limited generation capacity of the model, translation errors inevitably arise, subsequently causing the model to be influenced by erroneous context and thus diminishing the translation quality. This problem is called error accumulation.** As the model scale increases, the translation errors progressively decrease, empowering the model with greater resilience against the accumulation of errors. Consequently, the model can benefit from longer context sequences.\nBased on the above points, we believe that our work is innovative and holds significant significance for the advancement of the field of discourse translation. In addition, we will adjust the structure of the final version of the paper to further highlight our innovation.\n\n\n## Q2: Figure 1 in the paper is confusing and the casual relationship is not well supported by the experiments.\nFigure 1 illustrates a crucial observation in our experiments, which is that the model scale determines the final translation quality by influencing both sample efficiency and the optimal sequence length. We apologize for any confusion caused by the image in our paper, and we intend to refine this figure in the final version to prevent possible ambiguity.\nIn addition, our belief in a causal relationship, rather than a mere correlation, between model scale, optimal sequence length, and sample efficiency is grounded in the following considerations:\n1. Throughout our experiments, while keeping parameters other than model scale constant, we manipulated the model scale and observed changes in the optimal sequence length and sample efficiency.\n2. Among model scale, sample efficiency, and optimal sequence length, only the model scale can be predetermined, while the latter two are attributes observable after the model is established.\nHence, we contend that it is the model scale that determines the optimal sequence length and sample efficiency, rather than solely a correlation.\n\n## Q3: How the function for the optimal sequence length was estimated (Equation 1) and how reliable we expect this model to be.\nThe optimal sequence length used in our experiments is $L=alog(N)+b$, where $a$ and $b$ are parameters to be estimated. This function represents a linear relationship between $L$ and $logN$, thus $a$ and $b$ could be estimated using the least squares method. \nTo verify the reliable of our proposed empirical formula, we have conduct experiments from \n\n## Q4: What is the definition of error accumulation problem?\nIn this paper, error accumulation refers to the negative impact on the subsequent text generation due to possible errors in the previous text generation which act as context to the subsequent text generation. Thank you for your suggestion, we will add the definition of error accumulation in the final paper to avoid potential misunderstanding.\n\n## Q5: What is the maximum sequence length used in Section 5.2 (Error Accumulation Problem)?\nThe maximum sequence length is 1024 in this experiment, and the model we use is Transformer base. Thanks for your insightful feedback and we will include this information in our final paper.\n\n## Q6: What is the length distribution of the dataset (samples per bin) used in Section 5.2?\nIn this section, we do not train a new model; instead, we evaluated the performance on a model trained with a maximum sequence length of 1024. During evaluation, we also set the maximum sequence length to 1024. Subsequently, we split the translated segments into sentences and allocated them to different bins based on their positions within the segments. The number of sentences in each bin is illustrated in the table below. \n\n| sentence position | sentence number |\n| ----------------- | --------------- |\n| 0-20              | 12885           |\n| 20-40             | 7088            |\n| 40-60             | 4858            |\n| 60-80             | 1830            |\n\nOnce again, we appreciate your valuable feedback, and we apologize for not including this information in the submitted paper. We will incorporate this information in the form of a distribution graph into Figure 8.\n\n## Q7: The error accumulation may be related to tendencies of the Transformer to overfit to length statistics of the training data.\nIt is true that transformer has the tendency of overfitting to length statistics of the training data. However, it is important to note that the maximum sequence length used for both training and testing remains consistent in our experiment in Section 5.2. Therefore, the observed error accumulation problem here is unlikely to be caused by the length bias problem.\n\n## Q8: Why increasing the maximum sequence length from 60 to 128 hurt the performance on ContraPro?\nThank you for your careful review of our paper. We also observed this unusual phenomenon during our experiments. We speculate that the fluctuations in accuracy might be attributed to the inherent randomness in the system. To address this, we will repeat these experiments several times and average the results. However, as the limited time, we plan to provide this result during discussion period. \n\n## Q9: It would be helpful to also provide the confidence intervals to strengthen the conclusions.\nThank you for your constructive advice. We will incorporate the confidence interval ranges in the final version of the paper. However, due to the complexity of some figures with numerous lines, adding confidence interval ranges to certain figures might potentially impact their readability. As a result, we will carefully consider the trade-off while taking into account the practical aspects.\n\n## Q10: The BLEU score alone is insufficient to demonstrate that perplexity (PPL) is an inadequate evaluation metrics. \nWe greatly appreciate your insights regarding the evaluation metrics. Therefore, we further complement our experiments by calculating the coefficient of determination $R^2$ at different sequence lengths. As shown in the table below, the experimental results demonstrate a significant disparity between PPL and any other translation-specific metrics, confirming that PPL does not effectively reflect translation quality. Therefore, our conclusion remains consistent: PPL is not a truly meaningful measure of translation quality. Due to time constraints, we are unable to include more translation metrics in this analytical experiment. Therefore, we plan to include more evaluation metrics such as COMET and doc-COMET in the final version of our paper. \n\n| $R^2$  | 64   | 512  | 1024 |\n| ------ | ---- | ---- | ---- |\n| d-BLEU | 0.39 | 0.34 | 0.26 |\n| d-chrF | 0.49 | 0.52 | 0.48 |\n\n## Q11: The factor that is held constant is not reported in Section 4.\nThank you for your detailed examination of our paper. In Section 4.1, we maintained the model as a Transformer-base as controlled conditions. In Section 4.2, the entirety of the dataset was employed to analyze the joint impact of maximum sequence length and model scale. For the section titled \"Joint Effect of Maximum Sequence Length and Data Scale,\" we again utilized a Transformer-base model. We sincerely appreciate your observation regarding this matter, and in the final version of the paper, we will provide a more explicit description of the variables we have controlled.\n\n## Q12: The method employed in previous work differs from that used in this paper, thus the conclusion \"model performance improves with a larger context\" can be different.\nWe are grateful to your thoughtful comments on our work. We acknowledge the inaccuracy in the citations here and will make necessary revisions to the final version of our paper.\n\n## Q13: A lot of the details on training and inference configurations can be moved to the appendix.\nThank you for your constructive advice. We will further optimize the structure of the paper, highlighting the key points and relocating minor portions to the appendix.\n\n## Q14: How the MACs per token evaluated?\nWe first constructed virtual texts with sequence lengths of 32 or 1024. Subsequently, we set the batch size to 1, and utilizing deepspeed to measured the MACs (Multiply-Accumulate operations) required for both forward and backward passes of a batch. We then divided this value by the sequence length of the text (tokens per batch) to obtain MACs per token. Since this value has been normalized with respect to the number of tokens in each batch, it is comparable between any maximum sequence length.\n\n## Q15: The grammar errors, typos and not explained abbreviation.\nThanks for your careful examination of our submitted paper. We apologize for the presence of these errors in the paper and will correct these errors in the final version of our paper. In addition, we will examine the paper more meticulously to prevent such errors. \n\n# Conclusion\nWe acknowledge your thoughtful feedback and criticisms of our work. We have carefully reviewed your comments and have provided detailed responses to each of the points. If you find our response satisfactory, we would greatly appreciate an increase in our score. If you have any concerns or doubts about our response, please feel free to contact us during the discussion period.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper investigates the maximum sequence length for document-level machine translation, a hyperparameter that isn’t studied well yet even though it is crucial for training performant document-level MT systems in practice.", "reasons_to_reject": "Based on the analyses presented in the paper, the scaling behavior of document-level machine translation models (in terms of number of parameters and number of data points) is not crucially different from the scaling behavior of sentence-level machine translation systems, which has already been studied by [Ghorbani et al. (2021)](https://arxiv.org/pdf/2109.07740.pdf).\n\nIn my opinion, the experiments (or their reporting) could be more thorough in many places to back up the claims made by the authors, e.g.\n\n- Figure 1 indicates a causal relationship between the number of parameters and translation quality by affecting sample efficiency and optimal sequence length. While the graphic is confusing to me (e.g. it looks to me as if the number of parameters is affecting the corpus size), I don’t think that the intended claim (as reiterated at the end of the “Introduction” section) is supported by the results. None of the experiments actually show causation but rather association between the variables.\n\n- There is no information on how the function for the optimal sequence length was estimated (Equation 1) and how reliable we expect this model to be.\n\n- In Section 5.2, the authors should define what they mean by “error accumulation” explicitly. Is the problem that the models base the translation of later sentences made on erroneous translation of earlier sentences? If that is what the authors meant by “error accumulation”, it would need more analysis to verify such a phenomenon. Personally, I’d be interested in details like the maximum sequence length used for these experiments and the number of training examples per bin (which are not reported yet), as the observations made by the authors could be related to tendencies of the Transformer to overfit to length statistics of the training data, see [Varis and Bojar (2021)](https://aclanthology.org/2021.emnlp-main.650.pdf).\n\n- The results presented in Section 5.3 look rather noisy to me as the accuracy on ContraPro decreases from context length 60 to 120 and from 120 to 250 while it improves considerably for larger context lengths. Intuitively, a substantial context length increase (e.g. from 60 to 250 tokens) should not hurt the accuracy. The authors do not make an attempt to explain this trend.\n\n- It would be helpful to also provide the confidence intervals to strengthen the conclusions from the experiments with one or multiple factors.\n\n- The authors mention in Section 5.1 that the cross entropy loss “fails to fully depict the translation quality”. I don’t think that this conclusion is valid based on the authors' experiments. The authors are measuring translation quality in terms of (d-)BLEU, i.e. in terms of a metric that has many limitations, especially for document-level MT, see [Kocmi et al. (2021)](https://aclanthology.org/2021.wmt-1.57.pdf); [Post and Junczys-Dowmunt (2023)](https://arxiv.org/pdf/2304.12959.pdf). If general translation quality is not properly reflected by the metric, then it can’t really be determined whether the loss is a good indicator of general translation quality. \n\nSome minor comments are: \n- For most of the experiments in Section 4, the factor that is held constant is not reported, e.g. for the experiment of the joint effect of maximum sequence length and data scale (in Section 4.2), I couldn’t find the model size.\n- The authors mention that previous work, in particular, Beltagy et al. (2020) and Press et al. (2021) demonstrate that model performance improves with a larger context. Let me note here that their methods are different from the ones presented in this paper and thus conclusions can be different for a number of reasons.\n- In my opinion, a lot of the details on training and inference configurations can be moved to the appendix.\n- The abbreviation “MAC” is used in line 194 already but only explained later in the paper.", "questions_to_authors": "A: How was the model for the optimal maximum sequence length estimated (Equation 1) and are there any metrics that confirm its reliability?\n\nB: Concerning the analysis in Section 5.2: What maximum sequence length did you use for these experiments? Did you check the number of training examples per sentence position (i.e. how many training examples actually consist of x sentences) and if yes, how does the distribution look like?\n\nC: Concerning the analysis in Section 5.3: Why does the accuracy decrease from context length 60 to 120 and from 120 to 250? \n \nD: I’m surprised about the finding in Section 5.5. What is the setup of this experiment? How did you make sure that the numbers between maximum sequence length 32 and 1024 are comparable?", "paper_topic_and_main_contributions": "This paper explores the relationship of the number of parameters, the number of training data points and the maximum sequence length with the performance of transformer-based document-level machine translation models. Additionally, it explores the question why performance of these models drops once the maximum sequence length reaches a critical value.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 45: “the” missing \nLine 55: “it” missing\nLine 168: “necessary” seems quite a strong word\nLine 496: Figure 5.3 is mentioned, which doesn’t exist"}, "extra_text": {"Missing_References": "A recent paper that highlights important difficulties with document-level MT models (e.g. their evaluation, as mentioned above) is [Post and Junczys-Dowmunt (2023)](https://arxiv.org/pdf/2304.12959.pdf)."}}, {"review_id": "8PN0SVDSdx", "reviewer": "Reviewer_4ECh", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Reviewer 2\nYour feedback is greatly appreciated, and we would like to address your main points of concern to provide a clearer understanding of our work.\n\n\n## Q1: Can these conclusions be extended to other languages?\nSure! We are conducting new experiments on Chinese-English. However, due to time constraints, we are currently unable to present the experimental results in this round of response. We anticipate being able to complete the experiments in a few days and showcase our results during the discussion phase.\n\n## Q2: Statistics in Table 4 are not clear.\nThanks for your careful examination of our submitted paper. We sincerely apologize for the oversight in including units while creating the appendix. In Table 4, the counts for sentences and tokens are both reported in units of millions. We will include this information in the final version of our paper.\n\n## Conclusion\nWe acknowledge your thoughtful feedback and criticisms of our work. We have carefully reviewed your comments and have provided detailed responses to each of the points. If you find our response satisfactory, we would greatly appreciate an increase in our score. If you have any concerns or doubts about our response, please feel free to contact us during the discussion period.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Experiments are extensive in all three scaling factors.", "reasons_to_reject": "The study is conducted on one direction English to German. I am wondering if these conclusions can be extended to other languages.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper examines the impact of scaling model and data sizes and maximum sequence length on document-level neural machine translation. The study is conducted on English to German translation and based on the standard Transformer architecture with improvements. Analysis of the three factors reveled that they are all have impact the translation quality. However, the maximum sequence length is correlated with the number of non-embedding parameters.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Statistics in Table 4 is not clear to me including #Lines and #Tokens (1-3)."}, "extra_text": {}}, {"review_id": "MClRn7KhjU", "reviewer": "Reviewer_zccM", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Your feedback is greatly appreciated, and we would like to address your main points of concern to provide a clearer understanding of our work.\n\n## Q1: The novelty is considered limited.\nThis paper provides a systematic investigation of the scaling laws in document-level machine translation. In comparison to the scaling laws observed in sentence-level translation models, we have derived several significant and distinctive findings:\n1. We have identified the limitations of employing Perplexity (PPL) as an evaluation metric for machine translation tasks, particularly in the document-level machine translation scenario. Our analysis highlights that PPL-based evaluation metrics fail to accurately reflect translation quality. Furthermore, we conducted an in-depth examination of the correlations among different evaluation metrics. The experimental results indicate that PPL gradually deviates significantly from other metrics as sequence length increases;\n2. Prior work on document-level machine translation has presented two viewpoints. Some studies have indicated that the effective information for improving translation quality is concentrated in the context of approximately 1-3 sentences around the sentence to be translated. On the other hand, another perspective suggests that the model can benefit from contexts of arbitrary lengths. In this paper, we systematically investigate this issue. Through experiments conducted on the ContraPro dataset, we find that the Doc2Doc document-level machine translation model is indeed capable of capturing relevant information from a broader context. **However, when evaluating the model using conventional translation metrics, such as BLEU, we identified an optimal sequence length that corresponds to the best translation quality. Deviating from this optimal length, whether longer or shorter, leads to a degradation in translation quality.** This paper makes the pioneering observation of this phenomenon in document-level machine translation and provides a method, based on model scale, to estimate the optimal context length, which holds significant implications for document-level translation tasks;\n3. We further explore the underlying reasons behind the optimal context length of document-level machine translation models. It is not the case that more context information is always better, nor is the crucial information solely confined to the preceding 1-3 sentences. Instead, as the available context length increases, the model gains enhanced disambiguation capabilities. **However, due to the limited generation capacity of the model, translation errors inevitably arise, subsequently causing the model to be influenced by erroneous context and thus diminishing the translation quality. This problem is called error accumulation.** As the model scale increases, the translation errors progressively decrease, empowering the model with greater resilience against the accumulation of errors. Consequently, the model can benefit from longer context sequences.\nBased on the above points, we believe that our work is innovative and holds significant significance for the advancement of the field of discourse translation. In addition, we will adjust the structure of the final version of the paper to further highlight our innovation.\n\n\n## Q2: The writing needs to be polished.\nThanks for your carefully examination of our submitted paper. We will try our best to enhance the quality and refinement of the writing.\n\n## Conclusion\nWe acknowledge your thoughtful feedback and criticisms of our work. We have carefully reviewed your comments and have provided detailed responses to each of the points. If you find our response satisfactory, we would greatly appreciate an increase in our score. If you have any concerns or doubts about our response, please feel free to contact us during the discussion period.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Numerous experiments are conducted, and these results are sufficient to support their claims. Interestingly, they even estimated the constant parameters in their Equation (1), thus, providing not only a qualitative but also a quantitative understanding of these phenomena.", "reasons_to_reject": "I believe most of these findings, as also mentioned by the authors themselves, are not first discovered by them, but are further validated. Therefore, the novelty is considered limited. Besides, the writing needs to be polished. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper provides experiments on analyzing the most important three factors in the scaling law of document-level machine translations. Their results show: 1) the number of non-embedding parameters is related to the performance, 2) data size is related to the performance, 3) the optimal sentence length is logarithmically related to the number of parameters 4) the optimal sentence length still exists when increasing the data size 5) model width gets more critical than model depth when the data size gets larger.\n\nThey also give further discussions and try to explain these findings in the following sections, by associating it with the error accumulation problem.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates how the number of parameters, the amount of training data, and the maximum sequence length affect the performance of transformer-based document-level machine translation models. It also explores why the performance of these models decreases when the maximum sequence length exceeds a certain threshold.
This paper studies the maximum length of text sequences that can be translated effectively by document-level machine translation (MT) models. This hyperparameter, or tuning knob, is important for training high-performance document-level MT systems in practice, but it has not been well-studied yet. The authors run many experiments that are sufficient to support all claims. However, all experiments are on a single language pair. Nevertheless, the author promised to include Chinese-English results in the camera ready version.
Decision: Accept-Findings