Exploring Discourse Structure in Document-level Machine Translation
Authors: Xinyu Hu
Xiaojun Wan
Venue: EMNLP 2023 Main (2023)
Keywords: Document-level Machine Translation
Discourse Structure
RST Parsing
TL;DR: We propose a more sound paragraph-to-paragraph mode and RST-Att model with a multi-granularity attention mechanism based on RST parsing for DocMT.

Abstract: Neural machine translation has achieved great success in the past few years with the help of transformer architectures and large-scale bilingual corpora. However, when the source text gradually grows into an entire document, the performance of current methods for document-level machine translation (DocMT) is less satisfactory. Although the context is beneficial to the translation in general, it is difficult for traditional methods to utilize such long-range information. Previous studies on DocMT have concentrated on extra contents such as multiple surrounding sentences and input instances divided by a fixed length. We suppose that they ignore the structure inside the source text, which leads to under-utilization of the context. In this paper, we present a more sound paragraph-to-paragraph translation mode and explore whether discourse structure can improve DocMT. We introduce several methods from different perspectives, among which our RST-Att model with a multi-granularity attention mechanism based on the RST parsing tree works best. The experiments show that our method indeed utilizes discourse information and performs better than previous work.

Exploring Discourse Structure in Document-level Machine Translation
Xinyu Hu and Xiaojun Wan
Wangxuan Institute of Computer Technology, Peking University (huxinyu, Janxia.jjun@ekk. edu edu. cn
Abstract
they utilize additional contextual information, in- cluding the surrounding sentences (Zhang et al., 2018; Miculicich et al., 2018; Kang et al., 2020; Zhang et al., 2020b, 2021a), document contex- tual representation (Jiang et al.. 2020; Ma et al.. 2020) and memory units (Feng et al., 2022). In recent years, many researches have turned to trans- lating multiple sentences the entire document at once (Tan al., 2019; Bao et al., 2021; Sun al., 2022). However, previous work (Zhang et al., 2018; Liu et al., 2020; Sun et al., 2022) has demonstrated that direct Doc2Doc translation may cause the model not to converge. Therefore, those methods adopt pertinent measures such as data augmentation, text truncation, and spe- cific frameworks for sentence alignment.
Neural machine translation has achieved great success the past few years with the help of transformer architectures and large-scale bilin- gual corpora. However, when the source text gradually grows into an entire document, performance of current methods for document- level machine translation (DocMT) is less sat- isfactory. Although the context is beneficial to the translation general, it is difficult for traditional methods to utilize such long-range information. Previous studies on DocMT have concentrated extra contents such as multi- ple surrounding sentences and input instances divided by fixed length. We suppose that they ignore the structure inside the source text, which leads to under-utilization
et al., 2022;
the con- text. In this paper, present more sound saragraph-to-paragaap translation mode and explore whether discourse structure can im- prove DocMT. We introduce several methods from different perspectives, among which our RST-Aut model with multi-granularity atten- tion mechanism based on the RST parsing tree works best. The experiments show that our method indeed utilizes discourse information and performs better than previous work.
Despite the effectiveness of previous efforts on DocMT, some research, such Kim al. (2019), suggested that existing methods may not fully uti- lize the context and the improvement may come from regularization. Kang et al. (2020) also indi- cated that dynamic context selected from the sur- rounding sentences can improve translation more efficiently. The additional contexts used in most previous work are simply based on the distance from the current sentence or the length of the text, which somewhat arbitrary. In this paper, aim to explore more reasonable way encode context through the discourse structure.
1 Introduction
Transformer (Vaswani et al., 2017) based ap- proaches, together with adequate bilingual datasets, have led significant progress on machine trans- lation (MT). However, the performance of MT models usually drops dramatically when process- ing long texts. Although document-level machine translation (DocMT) can be solved with sentence- level MT by translating each sentence separately, the potential information in the long-range context may be ignored. To address these problems, many methods in DocMT have been proposed better utilize the contextual information and improve the overall translation quality of the document.
Discourse structure refers to how elementary text units are organized to form a discourse and logically linked to one another. Early studies have demonstrated that discourse parsing can ben- efit various downstream NLP tasks, including sentiment analysis (Bhatia et al., 2015), relation extraction (Wang et al., 2021), text summariza- tion (Gerani al., 2014; Xu al., 2020), machine translation evaluation (Guzmán al., 2014; Joty et al., 2014, 2017; Bawden et al., 2018) 8) and so on. RST parsing, based on Rhetorical Structure Theory (Mann and Thompson, 1987), one
Among these methods, the dominant approaches still adhere to the sentence- seeteenc--y--eeeeee sentence mode, but

---

the most influential parsing methods in discourse analysis. According to RST, a text is segmented into several clause-like units (EDUs) as leaves of the corresponding parsing tree. Through certain rhetorical relations among adjacent spans, underly- ing EDUs or larger text spans are recursively linked and merged to form their parent nodes, representing the concatenation of them.
parsing model improve document-level ma- chine translation.
3) Our RST-Att model achieves superior perfor- mance on three widely used datasets of DocMT and further linguistic evaluation, compared with existing works.
Methodology
Although RST parsing would be better con- ducted on an integral text to maintain discourse structure, existing models perform poorly on long texts. As result, we present new paragraph-to- paragraph translation mode, where the original doc- ument is divided into several shorter paragraphs. Our paragraph segmentation is generated by the TextTiling tool (Hearst, 1997) based subtopic shifts and discourse cues, since frequently-use datasets of DocMT do not contain paragraph align- ment tags. We suppose the discourse analysis of paragraphs is proper compromise and more sound than previous partitioning methods, And it is more labor-saving when dealing with other multilingual corpora equipped with paragraph alignments.
In this section, we will elaborate on the relevant steps to obtain the discourse structure information and explore its utilization in DocMT. In Section 2.1, we compare two text segmentation methods and determine the TextTiling tool for our paragraph segmentation according to the distribution of re- sults. In Section 2.2, we introduce the brief train- ing process of the RST parsing model and how to linearize the corresponding RST tree. Section 2.3 describes the construction and details of our pro- posed RST sequence integration method, as well the RST-Att model with the RST-specific attention mechanism of multiple levels of granularity.
We employ the end-to-end method of Hu and Wan (2023) to train our RST parsing model. The parsing task reformulated into a Seq2Seq task through a linearization process and then trained based on pretrained language model. Several at- tempts have been made from different perspectives to investigate how to utilize the discourse struc- ture, including the integration of the RST sequence, RST-specific attention mechanism, and graph learning based on the RST tree. Among them, our RST-Att model, designed with a multi-granulee granularity attention mechamism to inject the RST tree into the translation encoder, achieves the best performance. Specifically, each token at the first encoder layer can only notice other tokens from the EDU it be- longs to, when calculating self-attention scores. As the encoder layer moves backward, the range of attention continuously expands to complete con- text according to the structure the RST tree. We believe such progressive pattern can reduce the difficulty of modeling long-range context.
2.1 Paragraph Segmentation
In this step, we consider two distinct approaches for paragraph segmentation. The first is the traditional TextTiling method, which detects topic and lexical co-occurrences. The other a neural segmentation model proposed in recent years by Lukasik et al. (2020), based on pretrained language model and cross-segment attention.
TextTiling Algorithm TextTiling is an algo- rithm for tokenizing document into multi- paragraph units that represent subtopics, proposed by Hearst (1997). The algorithm identifies subtopic shifts based on the discourse cues of lexical co- occurrence and distribution patterns. It first divides the whole document into several parts, each which contains multiple unbroken sentences. Then the similarity scores are calculated between sen- tences, and the peak differences between them are found to become the final segmentation boundaries after the normalization.
Overall, our main contributions are as follows:
Neural Segmentation Model (2020) proposed a neural model with transformer- based architectures. They represented each candi- date break using its left and right local contexts and then turned to the pretrained language model
Lukasik et al.
I) We generate the paragraph segmentation and introduce a more sound ceatrraphttt-prraaggee translation mode than traditional text partition.
2) We explore several methods to take advantage of the discourse structure predicted by the RST
'Codes and data are available at https//giitua..co//eem/////SSAA

---

60,0%
2.2 RST Parsing
50.0%
Previous studies have proposed many methods for RST parsing (Lin et al., 2019; Zhang et al., 2020a; Kobayashi et al., 2020; Nguyen et al., 2021). How- ever, most of them split the parsing process into steps: EDU segmentation and RST tree prediction. for which the gold EDU labels are often required. Considering that the datasets of DocMT are not equipped with such information, we follow Hu and Wan (2023) to train an end-to-end RST parser from scratch through a Seq2Seq reformulation method. The training data comes from the standard RST Discourse TreeBank (Carlson al., 2001) and has been processed with similar length distribution our paragraph segmentation.
40.0%
Tropurten
30.0%
20.0%
10.0%
(0.32)
[32, 64)
(64, 128)
256)
[256, 512)
[512) m)
Nomber ofloans
I Noural Segmontation Madel
TestTiling Algovihm
Figure The distribution the number of tokens in each divided paragraph.
18.0%
16.0%
14.0%
Based on the priority of brackets, represent hierarchical architecture by nesting several pairs of brackets. The linearization is car- ried out from the bottom up, according to post- order traversal. We replace each leaf that represents a single EDU with a sequence comprised of a left bracket, text content, a right bracket, and its nucle- arity and rhetorical relation labels. The same pro- cess is performed for other nodes, except that the concatenation of new representations of two child nodes serves the text content. The linearized sequence is designed contain the complete orig- inal input text for better performance, according to the observation of Paolini et al. (2021). More details can be found in Hu and Wan (2023), and an example is shown in Figure 3(d).
Linearization
12.0%
Proprtonn
10.0%
8.0%
6.0%
4.0%
0.00%
Number of sentences
Neural Segmottaionn
TextTiling Algorithm
Figure 2: The distribution of the number of sentences in each divided paragraph.
for judging whether the candidate break was rea- sonable. The LM was finetuned on a large-scale dataset, Wiki-727K (Koshorek et al., 2018).
Seq2Seq Training Since the input and new out- put of the task are both sequences, we have trained our RST parsing model on the pretrained Seq2Seq model T5-base (Raffel et al., 2020). The related latent knowledge may be transferred and utilized during training since the reformulated sequences close to natural language text, which aligns with the pretraining of T5. Moreover, modify and align the output predicted by the model with the format we design before to obtain the final RST parsing tree through a recursive algorithm.
We apply the two methods above to the most commonly used dataset document-level machine translation and obtain the respective paragraph seg- mentation. We take the dataset News as an example, and the distributions of the number of tokens and sentences contained in each paragraph shown in Figure and Figure The vertical axis of the legends shows the proportion of the corresponding category in all samples. The paragraphs from the TextTiling method have a more reasonable distri- bution, and most of them contain moderate num- bers of tokens and sentences. On the other hand, a considerable part of the paragraphs obtained by the neural segmentation model only contain a few sentences. which not conducive to subsequent discourse parsing. Therefore, we finally choose the results of the TextTiling method for the following experiments, and the statistical details of paragraph segmentation can found in Table
2.3 Utilizing RST Structure
2.3.1 RST Sequence Integration
We first attempt simple method that directly in- tegrates the discourse parsing tree into the inputs of the model, called RST-Seq. The source input during training replaced with the correspond- ing linearized RST tree, and the target output is

---

(a) Input Text
(c)
RST Parsing Tree
Govemment lending was not intended to be way to obfuscate
Root
spending figures, hide fraudulent activity, or provide large subsidies.
(Nucleus, Span)
(Satellite, Elaburation)
(b) EDU Segmentation
EDU;
Government lending was not intended to be way
(Nucleus, joint)
(Nucleus, joint)
& obfuscate spending figures,
E hide fraudulent activity,
(Nucleus, joint)
(Nuclcus,
EDU, provide large subsidies.
(d) Linearization
EDU;
EDU;
EDU,
EDC,
[[ Government lending was not intended to way Nucleus span [[ obfuscate spending figures, ] Nucleus joint [[ [hide fruudulent activity. J Nuclees joint or provide large subsidies. Nucleus Nucleus joint Satellite eluboration
Figure 3: A brief example from RST Discourse TreeBank, with the EDU segmentation (part b), the RST parsing tree of the input text (part c), and the corresponding sequence of the linearized RST tree (part d).
kept the same. The experiments have been made on the well-trained mBART25 (Liu et al., 2020), similar to the previous work. We expect that the pretrained language model can encode the informa- tion of discourse structure together with the context through latent knowledge. The results of our RST- Seq method and the baseline without the discourse structure will be shown in Section 4.
the set will change. It can be simply proved that the text ranges represented by the nodes of finally obtained set Š perfectly cover the entire paragraph
content. We assume that S = (r1, P2, (2,
where the node r' covers the token range from the position of Abegi to kiend in the paragraph.
In the encoder layer 1 of the original transformer model, the multihead self-attention is computed
2.3.2 Multi-Granularity Attention
QiKj Vd ))
A' MultiHcad(Sfftmax
To further take advantage of the discourse struc- ture, we propose the RST-Att model with a multi- granularity attention mechanism based on the RST tree, inspired by Wu et al. (2018). From the first layer of the encoder to the last, the range that each token attend continuously expands accord- ing the bottom-ut up nodes of the RST parsing tree. According to Beltagy et al. (2020), context may be better integrated the higher layer, whereas the lower layer tends to encode local information. We suppose the model will better understand the source context in progressive way under the instruction of discourse structure.
where Qi and Kj are the query and key matrices with the vector dimension of in the / th layer. We then modify the calculation of the attention matrix Al to A' according to the text ranges in Sy:
QKT Vd
À MultiHead(SSftmmx
+ M*))
3rk E St, kibegin S j S kend
My
clsc
where the matrix M has the size of N N and N presents the number of tokens in the paragraph. Through the modified attention mechanism, dif- ferent granularities for context modeling can be implemented by different encoder layers with spe- cific attention ranges of tokens. Specifically, at the first layer, each token can only notice other tokens from the EDU that it belongs to. As for the last layer, each token can attend to all tokens in the paragraph, which turns back the original full at- tention mode. We simulate as many levels of granu- larity as possible, and our method illustrated with a simple example in Figure 4. The RST-Att model
Specifically, we first transform the RST pars- ing tree, combining the leaf nodes and their parent nodes. Then each node in the tree is assigned its height, which refers to the number of edges on its longest path to leaf node. We ignore the label information and replace each node with the range of EDUs it represents, shown in Figure 4. Obvi- ously, each leaf node has height of and only the root has the largest height. We construct the initial node set S, which consists of all nodes with heights no more than l. Then we delete the node from S it is descendant node of another node in Sj, until

---

Transformer Encoder
RST Parsing Tree
Attention matrix A
EDU; - , h-3
Add Norm
Feed Forward
EDU, h-
EDU; EDU, h
Nx
Add
Multi-Hadd
EDU, h=0
EDU; EDU, h-1
Attention
Layer 0 to Layer 3
EDU,
EDU, h=
Layer 0:
Government lending was intended way J EDU, obfuscate spending figures, RDU; hide
fraudulent activity, EDU; or provide large subsidies. EDU,
Layer 1:
Government lending was not intended to be way I EDu; to obfuscate spending ligures, EDU, hidd
fraudulent activity, provide large subsidies. EDUS
Luyer 2: Govermment lending was not intended to be way EDU, obfuscate spending figures, hide fraudulent activity, or provide large subsidies. EDUU-RDU
Layer 3: GGvvenment lending was not intended to be way to obfuscate spending figures, hide fraudulent activity, or provide large subsidies. EEDU,
Figure 4: The illustration of our RST-Att model with a simple example whose RST parsing tree has a height of 3, mapping to only four encoder layers. The multi-head attention vanilla transformer encoder (Vaswani et al., 2017) is modified for different layers.
does not introduce additional parameters; instead, it theoretically reduces computational overhead and improves efficiency with incomplete attention.
Europarl
The corpus extracted from Europarl v7, which is split into the training, test, and valid sets, as mentioned in Maruf et al. (2019).
Moreover, it should be mentioned that the en- coder contains a fixed number of layers, while the heights of RST trees vary. So should construct mapping guarantee each encoder layer is linked a certain node To keep the mapping uniform, the new set 5, calculated as follows:
The detailed statistics these datasets and their paragraph segmentation are displayed in Table 1. Similar to the previous work, Moses (Koehn et al., 2007) is used for data processing and sentence true- case. We apply BPE (Sennrich et al., 2016) with 32K merge operations both sides for all datasets. Following previous work (Bao et al., 2021; et al., 2022; Sun et al., 2022; Feng et al., 2022), we apply sentence-level BLEU score (s-BLEU) and document-level BLEU score (d-BLEU) as the metrics of evaluation. Since our methods are fo- cused on the DocMT and do not involve sentence alignments, the d-BLEU score our major metric, which matches n-grams in the whole document.
= S
L
where i 0, 1.... --1 and L, H denote the number of encoder layers and the height of the RST tree respectively. And m, refers the old layer index to be mapped.
3 Datasets and Settings
We evaluate our models on three widely used datasets for document-level machine translation for English to German, from Maruf et al. (2019).
Since our method can be directly applied to pre- trained language models that have been commonly employed in current research, we conduct our ex- periments based on mBART25 (Liuu et al., 2020). It is strong multilingual model with the transformer architecture and contains about 610M parameters. The setting aligns with the previous state-of-the- art model G-Transformer (Bao et al., 2021), which serves as our primary comparison objective. More experiment details are described Appendix A.
TED The corpus includes TED talks and corre- sponding translations from IWSLT 2017. 6-t2016- 2017 used as the test and the rest as valid.
News The corpus comes from News Commen- tary vl1. 6ewstest2016 is used as the test set and 5 as valid.

---

Dataset
#Sentences
#Documents
#Paragraphs
Avg #Tokens/Para
Avg #Sents/Para
TED
0.21M/9K/2.3K
1.7K//3/23
23K/966/230
217/217/232
9.0/9.3/9.9
News
0.24M/2K/3K
6.1K/71/155
35K/296/429 318K/691/988
212/200/189
6.7/7.3/7.0
Europarl
1.6 1.67M/3.6K55.IK
118K/240/600
169/170/170
5.3/5.2/5.2
Table I: The detailed statistics of the datasets used for DocMT in the form of train/vaild/test.
TED
News S-BLEU
Europarl
Model
S-BLEU
d-BLEU
d-BLEU
S-BLEU
d-BLEU
Sent2Sent (Sun et al., 2022)
25.19
29.16
24.98
27.03
31.70
33.83
MCN (Zheng al., 2020)
25.10
29.09
24.91
26.97
30.40
32.63
G-Transformer (Bao et al., 2021)
25.12
27.17
25.52
27.11
32.39
34.08
MR Doc2Sent (Sun et al., 2022)
25.24
29.20
25.00
26.70
32.11
34.18
Doc2Doc (Sun et al., 2022)*
29.27
26.71
34.48
Recurrent Memory (Feng et al., 2022)
25.62
29.47
25.73
27.78
31.41
33.50
P-Transformer (Li al., 2022)
25.67
27.94
25.93
27.67
32.62
34.49
Based on pretrained language models
Flat- Tat-Transformr (BERT) (Ma et al., 2020)
26.61
24.52
31.99
SDoc2SDoc (BART) (Bao et al., 2021)
28.29
30.49
34.00
G- G-Transformer (BART) (Bao et al., 2021)
28.06
30.03
30.34
31.71
32.74
34.31
Para2Para Baseline (BART)
30.35
31.43
34.19
RST-Seq (BART)
30.47
31.62
34.22
RST-Att (BART)
31.10
32.28
34.55
Table 2: Results of models and previous work document-level machine translation. Allhhough Sun al. (2022) achieved better results with additional sentence corpus, we do not consider them here for fairness.
4 Experiments and Analysis
based on a specified maximum length, since stan- dard Doc2Doc translation was pointed out be prone to training failures.
We compare the experimental results of our meth- ods with previous approaches to document-level machine translation on TED, News, and Europarl, as shown Table We choose the best result of Sent2Sent here from Sun et al. (2022), which even surpasses some DocMT models on d-BLEU, indi- cating the underuse of context in their work. More- over, some earlier studies are ignored since they only reported B-BLEU scores and performed much worse than recent work such G-Transformer. And we categorize all methods based on whether they are based on pretrained language models.
The results show that our Para2Para baseline out- performs the traditional SDoc2SDoc with same mBART25 model, proving the improvement our new translation mode. It can be attributed to the fact that simple SDoc2SDoc mode, without any other optimization, may arbitrarily ignore the inter- actions between strongly related sentences. In trast, our proposed
poaggraph-to-ppaagraa method takes them into consideration and is expected make better use of contextual information.
Furthermore, we introduce discourse informa- tion and propose our RST-Seq and RST-Att models. Both of them follow the Para2Para translation mode based mBART25, and contain the same number of parameters as the baselines mentioned above. Al- though RST- Seq performs better than the Para2Para baseline, the improvements suppose that much longer inputs of linearized RST trees may increase the difficulty of training. More- over, it may be difficult for mBART25 to directly adapt to the format of the RST sequence since it
Above all, we introduce our Para2Para baseline, which finetuned on the original mBART25 model. It simply performs paragraph-te ceterraphhto--paaggge paragraph trans- lation based on the segmented paragraphs with- out utilizing any RST discourse information. For better comparison, also present the results of the traditional sub-document baseline SDoc2SDoc from Bao et al. (2021). As described in Section many studies on DocMT have adopted splitting each document into dub-mmnunnts for translation
prominent. We

---

Model
TED
News
Model
deixis
lex.c
ell.infl
ell.
RST-Aut
31.10
32.28
Sent2Sent
50.0
45.9
53.0
28.9
Six-level granularity
30.80 (-0.30)
32.04 (-0.24)
Concat
83.5
47.5
76.6
76.2
Three-level granularity
30.75 (-0.35)
31.78 (-0.50)
CADec
81.6
58.1
72.2
80.0
Equal division
30.31 (-0.79) 31.55 (-0.73)
TTMM-Tranfforer
91.0
46.9
82,2
78.2
MR-Doc2Doc
64.7
46.3
65.9
53.0
Table 3: d-BLEU scores the experiments in granularity discussion with different settings.
G- G-Tranffommer
89.9
84.8
82,4
ChatGPT
57.9
44.4
75.0
71.6
GPT-4
85.9
72.4
69.8
81.4
RST-Att (ours)
87.2
81.7
85.2
87.2
has not been finetuned for such parsing tasks and there are not enough hints in the translation task. In addition, results and improves d-BLEU scores by 1.07, 0.57, and 0.06 on TED, News, and Europarl, respec- tively, showing the effectiveness of our proposed attention mechanism. Its superiority over the RST- Seq demonstrates that discourse knowledge may be applied to the model structure more effectively than the input data. Furthermore, Table 1 shows that the data scales of TED, News, and Europarl are increasing while the improvements on the cor- responding dataset are decreasing. We believe that our RST-specific attention module can alleviate the negative impact of small datasets to a certain ex- tent. When the training data is sufficient, like in the case of Europarl, the performance gaps between all existing models are not significant.
Table 4: Accuracy [%] of translation prediction for spe- cific discourse phenomena (deixis, lexical consistency, ellipsis of morphological inflection, and VP ellipsis) among different models on the contrastive test sets.
RST-Att model achieves the best
equally divided tree. Specifically, each range of content is divided into two equal ones, layer by layer, until the current content contains no more than three tokens.
The results shown in Table 3. The drops in performance become larger from the model with six-level granularity the model with three-level granularity, proving that more elaborate granularity levels contribute to the improvement of the RST- Att model. On the other hand, despite also con- taining multiple granularity, the model constructed through arbitrarily equal division gets even worse results, which further demonstrates the crucial and important role of discourse structure in our method.
4.1 Granularity Discussion
To demonstrate the effectiveness of our introduc- tion of the RST discourse structure, we evaluate the RST-Att model with different settings of the involved multi-granularrty attention mechanism.
4.2 Linguistic Evaluation
Furthermore, we have conducted linguistic eval- uations about the performance of models when dealing with discourse phenomena, based on the frequently used contrastive test sets (Voita et al., 2019) in existing works. They were designed for targeted evaluation of several discourse phenom- ena: deixis, ellipsis (two test sets for VP ellipsis and morphological inflection), and lexical cohe- sion. Each test instance consists of a true exam- ple and several contrastive translations that dif- fer from the true one only in the considered dis- course aspect. The translation system needs to select the one that it considers to be true, which is evaluated by accuracy. We compared the sults of Sent2Sent, Concat and CADec from Voita et al. (2019), LSTM-Transffrmer from Zhang et al. (2020b), G- T-aanssformer, and MR-Doc2Doc, Chat-
Different Number of Levels We first verify whether using as much granularity as possible can improve the performance of the model. The node sets are instead mapped to just six encoder layers, corresponding to half of the original levels of gran- ularity. We make a copy of each mapped layer to fill the whole encoder progressively. Furthermore, we attempt the case of much less granularity in clearer manner of partitioning, including three levels of EDU, sentence and paragraph. And sim- ilarly, each layer of these three levels is repeated four times after mapping.
Equally Divided Tree We also pay attention to the impact of discourse guidance in our multi- granularity attention module. To exclude the dis- course information, we conduct more direct im- plementation of multiple granularity based on the
Smmaller text units will cause the failure of training accord- ing to our experiments.

---

GPT and GPT-4 from Wang et al. (2023) with our RST-Aut model, as shown in Table
16.
130
Our RST-Att model has achieved superior perfor- mance, with the highest accuracy in three aspects: lexical cohesion, VP ellipsis, and morphological in- flection, while having lower performance in deixis compared to SOTA. And the improvement in terms of lexical cohesion over previous methods is no- tably significant. We suppose may be because the RST discourse structure that pays attention to the relationship between text units can promote the model to better handle lexical cohesion. More- over, our approach outperforms LLMs comprehen- sively, including ChatGPT and GPT-4, and more discussions with LLM are described in Appendix B. These results indicate the enhanced ability of model adapt to contextual information and han- dle discourse phenomena, which shows the promis- ing way to introduce discourse information into document- machine translation.
30.0
27.0
240
21.0
Numbrred tokers
Boseline
- RST-AR
Figure The p-BLEU scores of the baseline and our RST-Att model different input lengths.
tree a graph, and the connectivity and the path to other nodes with label sequences can be calcu- lated for each EDU. These features are encoded to update the representation each EDU with the architecture of GraphFormer (Yang et al., 2021). Then we integrate the learned representation into the calculation of each token the EDU contains.
4.3 Context Length
Although this method surpasses the baseline, there is no significant improvement over the RST- Att model. We suppose that the path with label information can represent the relationship between nodes at both ends, but it may be too complicated for the model to encode such knowledge. On the other hand, there are still many errors in the pre- dicted labels on account of the limitations of cur- rent RST parsing research, which may mislead the model during training. We hope our exploration can inspire future research come up with more effective approaches to utilizing RST labels.
Next, we compare our RST-Att model with the baseline with respect to different lengths of input context. Since our translation is based on para- graphs, we follow the d-BLEU to calculate the P-BLEU score, which matches the n-grams in each paragraph. Figure 5 depicts the results of two mod- els on News. Surprisingly, the baseline, whose structure is still the vanilla transformer, does not fail on long contexts, contradicting the findings Li et al. (2022). We consider that the difference may be due to the knowledge of the well-trained LM. Moreover, the result of our RST-Aut model exceeds the baseline at each length, which is more distinct as the length increases. And maintains relatively stable performance when the input length is more than 256, indicating the advancement of our model in dealing with long inputs.
Related Works
Document-level Machine Translation Al-
though early work has achieved great success on machine translation, document is often processed by translating each sentence separately. As result, the information included in context is ignored. Recently, document-level machine translation has attracted more attention and many methods have been proposed to better utilize the contextual information to improve translation quality.
4.4 Label Utilization
Since the RST-Att model ignores nuclearity and relations on the RST parsing tree, we have further explored whether the in-depth utilization of label information can lead to more improvements. We apply graph learning to enhance the comprehen- sion of the model for each EDU. RST parsing trees are transformed into dependency trees according Hirao al. (2013), so that each node will rep- resent different EDU. We serve the dependency
Most early attempts still followed sentence-to- sentence translation, but they applied various frame- works utilize the context during training. Zhang al. (2018); Miculicich et al. (2018): Zhang et al. (2020b); Zheng al. (2020); Zhang al. (2021 a) utilized surrounding sentences and integrated the contextual information into encoding of the current
Phtps://chat.ppemai.com

---

sentence. Kang et al. (2020) indicated dynamic context selected from the surrounding sentences can improve the quality of the translation more effectively. Jiang et al. (2020); Ma et al. (2020) de- signed the specific module to encode the document into a contextual representation. Feng et al. (2022) introduced recurrent memory unit to remember the information thorough the whole document.
Conclusions
In this paper, we explore the role of discourse struc- ture in document-level machine translation. We introduce a more sound paragraph to to-paragraph translation mode than the several surrounding sen- tences or fixed length of texts used in previous stud- ies. To better take advantage of the RST parsing tree, propose the RST-Att model with multi- granularity attention mechanism depending on the tree structure. The experiment results prove the superiority of our method, and further evaluation indicates that both the guidance of discourse struc- ture and more levels of granularity contribute to the improvement. And the more effective utilization of RST labels for DocMT is left to future research.
Although Zhang et al. (2018); Liu al. (2020) have shown that direct Doc2Doc translation may cause the failure of training, many recent studies have focused on translating multiple sentences or the entire document at once (Tan et al., 2019; Bao et al., 2021; Sun et al., 2022; Li et al., 2022). They used various specific methods to avoid this problem and achieved more advancement DoeMT.
7 Limitations
Discourse Parsing Discourse parsing describes the hierarchical tree structure of text and can be used quality evaluations like coherence and other downstream applications. RST parsing is the most important role of discourse parsing, and the existing approaches can be mainly divided into two classes: top-down and bottom-up paradigms.
Some limitations exist in our research and may be able be solved in future research. Firstly, cur- rent studies on discourse parsing have not achieved great success due to insufficient labeled data. How- ever, the recent improvements this domain are significant, and the parsing model we employed will directly benefit from the advancement of pre- trained language models. We also believe that bet- ter discourse understanding and dealing with mul- tilingual issues would be more beneficial, and we intend to dig into this field in future research and attract more rescarch attention. On the other hand, despite the progressive attention module reducing the calculation, our model does not significantly op- timize the cost of time consumed in training. Since we utilize mBART25 which contains quite few parameters, the process of finetuning may have a minor environmental impact. In future research, we will also further explore how take better advan- tage of RST labels in the parsing tree, which may be useful for document-level machine translation.
Bottom-up methods have been first proposed since band-engineered features were suitable for representing local information. Models with CKY- like algorithms (Hernault et al., 2010; Joty et al., 2013; Feng and Hirst, 2014; Li et al., 2014) utilized diverse features to learn the scores for candidate trees and selected the most possible one. Another common bottom-up method is the transition- based parser with actions of shift and reduce (Ji and Eisen- stein, 2014; Wang et al., 2017: Yu et al., 2018). Recent advancements in neural methods made global representation more effective, which pro- moted top-down parsers. Lin et al. (2019) first presented a Seq2Seq model based on pointer net- works (Vinyals et al., 2015) and Liu et al. (2019) im- proved it with hierarchical structure. Then Zhang al. (2020a) extended their methods to document- level RST parsing. Kobayashi et al. (2020) con- structed subtrees with three levels of granularity and merged them together.
Acknowledgements
This work was supported by National Key R&D Program of China (2021 YFF090015))),, National Science Foundation of China (No. 62161 160339), State Key Laboratory of Media Convergence Pro- duction Technology and Systems and Key Lab- oratory of Science, Technology and Standard in Press Industry (Key Laboratory of Intelligent Press Media Technology). We appreciate the anonymous reviewers for their helpful comments, and everyone who has provided assistance in this work. Xiaojun Wan is the corresponding author.
Despite the better performance of top-down mod- els, most of them still need gold EDU segmen- tation and drop a lot in performance when us- ing automatic segmenters. To address the prob- Jem, Nguyen al. (2021) introduced an end-to-end parsing model, relying on specific frameworks for different tasks. Zhang et al. (2021b) proposed a system with rerankers to improve the performance.

---

References
meeting of SIGDAT, a Special Interest Group of the ACL, pages 1602-1613. ACL.
Guangsheng Bao, Yue Zhang, Zhiyang Teng, Boxing Chen, and Weihua Luo. 2021. G-transformer for Proceedings
Shahriar Golchin and Mihai Surdeanu. 2023. Time travel in Ilms: Tracing data contamination in large language models. CoRR, abs/2308. 08843.
document-level machine translation. of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Juint Conference on Natural Language Processing. ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages tion for Computational Linguisties.
Francisco Guzmán, Shafiq R. Joty, Lluis Märquez, and Preslav Nakov. 2014, Using discourse structure im- proves machine translation evaluation. In Proceed- ings of the 52nd Annual Meeting of the Association for Computational Linguistics, ACL 2014, June 22- 27, 2014, Baltimore, MD, USA, Volume I: Long Pa- pers. pages 687-698, The Association for Computer Linguistics.
Associa-
Rachel Bawden, Rico Sennrich, Alexandra Birch, and Barry Haddow. 2018. Evaluating discourse phenom- ena neural machine translation. Proceedings of the 2018 Conference of the North American Chap- ter of the Association for Computational Linguistics:
Marti A. Hearst. 1997. Texttiling: Segmenting text into multi multi-paragraph subtopic passages. Comput. Linguis- tics, 23(1):33 64.
Human Language Technologies, NAACL-HLI 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Vol- ume I (Long Papers), pages 1304-1313, Association for Computational Linguistics.
Hugo Hernaut, Helmut Prendinger, David A. du Verle, and Mitsuru Ishizuka. 2010. HILDA: A discourse parser using support vector machine classification. Dialogue Discourse, 133)1-333.
1z Beltagy, Matthew E. Peters, and Arman Cohan, 2020. Longformer: The long-document transformer. CoRR, abs/2004.05150.
Tsutomu Hirao, Yasuhisa Yoshida, Masaaki Nishino, Norihito Yasuda, and Masaaki Nagata. 2013. Single- document summarization as tree knapsack problem. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. EMNLP 2013, 18-21 October 2013, Grand Hyatt Seattle, Seat- tle, Washington, USA, meeting of SIGDAT, a Spe- cial Interest Group of the ACI, pages 1515-1520. ACL.
Parminder Bhatia, Yangfeng Ji, and Jacob Eisenstein. 2015. Better document-level sentiment analysis from RST discourse parsing. Proceedings of the 2015 Conference on Empirical Methods in Natural Lan- guage Processing, EMNLP 2015, Lishon, Portugal, September 17-21, 2015. pages 2212-2218. The As- sociation for Computational Linguistics.
Lynn Carlson, Daniel Marcu, and Mary Ellen Okurovsky. 2001. Building a discourse-tagged cor- pus in the framework of rhetorical structure theory. In Proceedings of the SIGDIAL 2001 Workshop, The 2nd Annual Meeting of the Special Interest Group on Discourse and Dialogue, Saturday, September I, 2001 to Sunday, September 2, 2001, Aalborg, Den- mark. The Association for Computer Linguistics.
Xinyu Hu and Xiaojun Wan. 2023. RST discourse pars- ing as text-to-text generation. IEEE ACM Trans. Au- dio Speech Lang. Process., 31:327833289
Yangfeng Ji and Jacob Eisenstein. 2014, Representation learning for text-level discourse parsing. In Proceed- ings of the 52nd Annual Meeting of the Association for Computational Linguistics, ACL 2014, June 22- 27, 2014, Baltimore, MD, USA, Volume I: Long Papers, pages 13-24. The Association for Computer Linguistics.
Vanessa Wei Feng and Graeme Hirst. 2014. A linear- time bottom-ur up discourse parser with constraints and post-editing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Lin- guistics, ACL 2014, June 22-27, 2014, Baltimore, MD, USA, Volume 1: Long Pupers, pages 511-521. The Association for Computer Linguistics.
Shu Jiang, Hai Zhao, Zuchao Li, and Bao-Liang Lu. 2020. Document-leved neural machine translation with document embeddings. CoRR, 000/200990875.....
Shafiq R. Joty, Giuseppe Carenini, Raymond T. Ng, and Yashar Mehdad. 2013. Combining intra- and multi-sentential rhetorical parsing for document-level discourse analysis, In Proceedings of the S1st An- nual Meeting of the Association for Computational Linguisticx, ACL, 2013, 4-9 August 2013, Sofia, Bul- garia, Volume 1: Long Papers, pages 486-496. The Association for Computer Linguistics.
Yukun Feng, Feng Li, Ziang Song, Boyuan Zheng, and Philipp Koehn. 2022. Learn to remember: Trans- former with recurrent memory for document-level machine translation. In Findings of the Association for Computational Linguistics: NAACL 2022. Seattle. WA, United States, July 10-15, 2022, pages I 1409- 1420. Association for Computational Linguistics.
Shafiq R. Joty, Francisco Guzmán, Lluis Märquez, and Preslav Nakov. 2014. Discotk: Using discourse struc- ture for machine translation evaluation. In Proceed- ings of the Ninth Workshop on Statistical Machine Translation, WMT@ACL 2014, June 26-27, 2014. 44 Baltimore, Maryland. USA, pages 402-400 408. The As- sociation for Computer Linguistics.
Shima Gerani, Yashar Mehdad, Giuseppe Carenini, Ray- mond T. Ng. and Bita Nejat. 2014. Abstractive summarization of product reviews using discourse structure. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing. EMNLP 2014, October 25-29, 2014, Doha, Qatar, A

---

Shafiq R. Joty, Francisco Guzmán, Lluis Mårquez, and Preslav Nakov, 2017. Discourse structure in machine translation evaluation. Comput. Linguistics, 43(4).
Xiang Lin, Shafiq R. Joty, Prathyusha Jwalapuram, and Saiful Bari. 2019. A unified linear-time framework for sentence-level discourse parsing. In Proceedings of the 57th Conference of the Association for Compu- tational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume I: Long Papers, pages 4190-4200. Association for Computational Linguis- tics.
Xiaomian Kang. Yang Zhao, Jiajun Zhang, and Chengging Zong. 2020. Dynamic context selection for document-level neural machine translation via reinforeement learning. In Proceedings of the 2020 Conference on Empirical Methods Natural Lan- guage Processing, EMNLP 2020, Online, November 16-20, 2020, pages 2242-2254, Association for Com- putational Linguisties.
Linlin Liu, Xiang Lin. Shafiq R. Joty, Simeng Han, and Lidong Bing. 2019. Hierarchical pointer net parsing. In Proceedings of the 2019 Conference on Empiri- cal Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNIPP 2019, Hong Kong, China, November 3-7, 2019, pages 1007-1017. Association for Computational Linguisties.
Yunsu Kim, Due Thanh Tran, and Hermann Ney. 2019, When and why is document-level context useful in neural machine translation? In Proceedings of the Fourth Workshop on Discourse in Machine Transla- fion. DiscoMT@ @ EMNLP 2019, Hong Kong. China, November 3. 2019, pages 24-34. Association for Computational Linguistics.
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre- training for neural machine translation. Trans. Assoc. Comput. Linguistics, 8:726-742.
Naoki Kobayashi, Tsutomu Hirao, Hidetaka Kamigaito, Manabu Okumura, and Masaaki Nagata. 2020. Top- down RST parsing utilizing granularity levels in doc- uments. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirly-Second Innovative Applications of Artificial Intelligence Con- ference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, FAAI 2020, New York, NY. USA, February 7-12, 2020. pages 8099-81008 1199. AAAI Press.
Michal Lukasik, Boris Dadachev, Kishore Papineni, and Gonçalo Simões. 2020. Text segmentation by cross segment attention. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 4707-4716. Association for Computa- tional Linguistics.
Philipp Koehn, Hicu Hoang, Alexandra Birch, Chris Callison-Burch Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In ACI. 2007, Proceedings of the 45th Annual Meet- ing ef the Association for Computational Linguistics, June 23-30, 2007, Prague, Czech Republic. The As- sociation for Computational Linguistics.
Shuming Ma, Dongdong Zhang, and Ming Zhou. 2020. A simple and effective unified encoder for document- level machine translation. In Proceedings of the 58th Annual Meeting of the Association for Com- putational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 3505-3511, Association for Computa- tional Linguisties.
William C Mann and Sandra A Thompson. 1987. Rhetorical structure theory: A theory of text organi- zation.
Omri Koshorek, Adir Cohen, Noam Mor, Michael Rot- man, and Jonathan Berant. 2018. Text segmentation as a supervised learning task. In Proceedings of the 2018 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 2 (Short Papers), pages 469-473. Association for Computational Linguistics.
Sameen Maruf, André F. T. Martins, and Gholamreza Haffari. 2019, Selective attention for context-awaid neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers). pages 3092-3100 Associa- tion for Computational Linguistics.
Jiwei Li, Rumeng Li, and Eduard H. Hovy. 2014. Recur- sive deep models for discourse parsing. In Proceed- ings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, Octo- ber 25-29, 2014, Doha, Qutar, A meeting of SIGDAT, a Special Interest Group of the ACL., pages 2061 2069. ACL.
Lesly Miculicich, Dhananjay Ram, Nikolaos Pappas. and James Henderson. 2018 Document-level neural machine translation with hicrarchical attention net- works. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 November 4, 2018, pages 2947-2955 Association for Computational Linguistics.
Yachao Li, Junhui Li, Jing Jiang, Shimin Tao, Hao Yang, and Min Zhang. 2022. P-transformer: Towards better secument--o--ocumae neural machine translation. CoRR, abs/2212.000 1.005300.
Thanh- Tung Nguyen, Xuan-Phi Nguyen, Shafiq R. Joty. and Xiaoli Li. 2021. RST parsing from scratch. In

---

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional L Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pages 1613-1625. Association for Computational Linguis- tics.
lexical cohesion. In Proceedings of the 57th Con- ference of the Association for Computational Lin- guistics, ACL. 2019, Florence, Italy, July 28- August 2. 2019, Volume I: Long Papers, pages 1198-1212. Association for Computational Linguisties.
Hailin Wang, Ke Qin, Guoming Lu, Jin Yin, Rufai Yusuf Zakari, and Jim Wilson Owusu. 2021. Document- level relation extraction using evidence reasoning on RST-GRAPH. Knowl. Based Syst. 107:0477
Giovanni Paolini, Ben Athiwaratkun, Jason Krone, Jie Ma. Alessandro Achille, Rishita Anubhai, Cícero Nogueira dos Santos, Bing Xiang, and Ste- fano Soatto. 2021. Structured prediction as transla- tion between augmented natural languages. In 9th International Conference on Learning Representa- tions, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net,
Longyue Wang, Chenyang Lyu, Tianbo Ji, Zhirui Zhang, Dian Yu, Shuming Shi, and Zhaopeng Tu. 2023, Document-level machine translation with large lan- guage models. CoRR, abs/2304.02210.
Yizhong Wang, Sujian Li, and Houfeng Wang. 2017. A two-stage parsing method for text-level discourse analysis. In Proceedings of the 55th Annual Meet- ing of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 August 4, Volume 2: Short Papers, pages 184-188. Association for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roherts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. J. Mach. Learn. Res., 21:140:1-140:61
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Lin- guistics, ACI. 2016, August 7-12, 2016, Berlin, Ger- many, Volume 1: Long Papers. The Association for Computer Linguisties.
Wei Wu, Houfeng Wang. Tianyu Liu, and Shuming Ma. 2018. Phrase-level self-attention networks for universal sentence encoding. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium. October 31 November 4, 2018, pages 3729-3735 Association for Computational Linguistics.
Zewei Sun, Mingxuan Wang, Hao Zhou, Chengqi Zhao, Shujian Huang, Jiajun Chen, and Lei Li. 2022. Re- thinking decument-level neural machine translation. In Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 3537-3548. Association for Computa- tional Linguisties.
Jiacheng Xu, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. Discourse-awand aware neural extractive text sum- marization. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, ACL 2020, Online, 5-10, 2020, pages 5021-5031. Association for Computational Linguisties.
Xin Tan, Longyin Zhang, Deyi Xiong, and Guodong Zhou. 2019. Hierarchical modeling of global con- text for document-level neural machine translation. In Proceedings of the 2019 Conference on Empiri- cal Methods in Natural Lunguage Processing and the 9th International Joint Conference on Natural Language Processing, SMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 1576-1585 Association for Computational Linguisties.
Junhan Yang, Zheng Liu, Shitao Xiao, Chaozhuo Li, Defu Lian, Sanjay Agrawal, Amit Singh, Guangzhong Sun, and Xing Xie. 2021. Graphform- ers: Gnn-nested transformers for representation learn- ing on textual graph. In Advances in Neural In- formation Processing Systems 34: Annual Confer- ence on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 28798-28810.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Mia Polosukhin. 2017. Attention all you need. Advances in neural information processing systems, 30.
Nan Yu, Meishan Zhang, and Guohong Fu. 2018. Transition- neural RST parsing with implicit syntax features. In Proceedings of the 27th Inter- national Conference on Computational Linguistics, COLING 2018, Santa Fe, New Mexico, USA, August 20-26, 2018, pages 359-570. Association for Com- putational Linguistics.
Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly, 2015. Pointer networks. In Advances in Neural Information Processing Systems 28: Annual Confer- ence on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 2692-2700.
Jiacheng Zhang. Huanbo Luan, Maosong Sun, Feifei Zhai, Jingfang Xu, Min Zhang, and Yang Liu. 2018. Improving the transformer translation model with document-level context. Proceedings of the 2018 Conference on Empirical Methods in Natural Lan- guage Processing, Brussels, Belgium, October 31 November 4. 2018, pages 533-542. Association for Computational Linguistics.
Elena Voita, Rico Sennrich, and Ivan Titov. 2019, When a good translation is wrong in context: Context-aware machine translation improves on deixis, ellipsis, and

---

Long Zhang, Tong Zhang, Haibo Zhang, Baosong Yang. Wei Ye, and Shikun Zhang. 2021a. Multi-hop trans- former for document-level machine translation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- fional Linguislics: Human Language Technologies, NAACL-HILT 2021, Online, June 6- 6-11, 2021, pages 3953-3963. Association for Computational Linguis- tics.
Model
TED
News
Europarl
ChatGPT
33.60
39.40
30.40
RST-Att
31.10
32.28
34.55
Table 5: The comparison on the three prominent Doc- MT datasets between ChatGPT and our RST-Att model. The results of ChatGPT come from Wang et al. (2023).
Training time
Inference time seconds per set
Longyin Zhang, Yuqing Xing, Fang Kong, Peifeng Li, and Guodong Zhou, 2020a. A top-down neural archi- tecture towards text-level parsing of discourse thetor- ical structure. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, ACL 2020, Online, July 5-10, 2020, pages 6386-6395. Association for Computational Linguis- tics.
Model
seconds per epoch
Sent2Sent
906
Para2Para
1055
112
RST-Att
1127
115
Table 6: Time consumed in the training and inference processes of different models.
Pei Zhang, Boxing Chen, Niyu Ge, and Kai Fan. 2020b. Long-shon term masking transformer: A simple but effective baseline for document-level neural machine translation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing, EMNIPP 2020, Online, November 16-20, 2020, pages 1081-1087. Association for Computational Linguistics.
and 2500 on Europarl. The beam size during infer- ence is 5, and the BLEU score is calculated in a maximum order of 4 after removing BPE tokens.
B Discussion with LLM
There have been some studies assessing the perfor- mance of LLMs on prominent Doc-MT datasets, using the prompt that translates the entire document at once. The results of the popular LLM ChatGPT compared
Ying Zhang, Hidetaka Kamigaito, and Manabu Oku- mura. 2021b. A language model- based generative classifier for sentence-level discourse parsing. In Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, EMNLP 2021, Virtual Event/ Punta Cana, Dominican Republic, 7- 11 November, 2021, pages 2432-2446, Association for Computational Linguistics.
RST-Att model are shown in Table with the d-BLEU metric. ChatGPT outperforms our RST-Att model on the TED and News datasets, while it lags behind on the Eurparl dataset. We speculate it may be due to the massive training data of ChatGPT, And the domain of Europarl is not as commonly used compared to the TED and News datasets. Furthermore, we cannot verify whether these datasets, particularly the test sets, have al- ready been seen by ChatGPT, they are publicly available and have been released early. Therefore, debatable whether these results are compara- ble. Recent research (Golchin and Surdeanu, 2023) has also focused on this problem and highlighted concerns regarding data contamination in the test sets of downstream tasks.
Zaixiang Zheng, Xiang Yue, Shujian Huang, Jiajun Chen, and Alexandra Birch. 2020. Towards making the most of context in neural machine translation. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, pages 3983 33899. ijcai.org.
A Detailed Experiment Settings
All of the datasets and pretrained language mod- els were obtained from publicly available sources under the MIT license, which can be used in aca- demic research. We conduct our experiments on mBART25 (Liu et al., 2020), which involves 12 layers, 16 heads, 1024-dimension outputs, and an intermediate size of 4096 in both the encoder and decoder. The hyperparameters in our work were determined according to the performance the valid set during the grid search, which started from the hyperparameters in Liu et al. (2020). We use 0.3 dropout, 0.2 label smoothing, and the Adam optimizer with a learning rate of 3c-5 for training. The warm-up steps are 2000 on TED and News,
C Time Efficiency
We have compared the translation efficiency of the training and inference processes with the same settings, such GPUs, batch size, etc. In Table 6, we present the relevant results of our RST-Aut model with Sent2Sent and Para2Para baselines on the NC-2016 dataset, and the performance on other datasets is similar. The training time consumed by

---

Para2Para2 is about 1.16 times that of Sent2Sent, and the inference time is about 1.3 times. We suppose it is because a paragraph is considerably longer than a sentence, and the computational com- plexity of the transformer framework is quadratic with respect to input length. Our RST-Att model also adopts does not introduce new training parameters. But it requires some computation for multi-granularity attention organization, its time consumed is bit greater than the Para2Para baseline. Considering the theoretically lower computation the atten- tion module, the further efficiency optimization of the implementation of our method is left to future work.
saserrappttt-praaggpee translation and


Reviews:
[{"review_id": "6f1Eiyf3X9", "reviewer": "Reviewer_2yPU", "summary": "", "questions": "1. How much slower is the decoding efficiency compared to translating the entire sentence? It is hoped that the authors can provide some experimental results related to translation efficiency;\n\n2. This is still a traditional approach applied to translation models. How does it compare with the currently prevalent large language models (LLMs)? I would like to see comparative experimental results.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and efforts, as well as the insightful comments, and we address your concerns below:\n\n**Q1:  There is a need for experimental results on relevant datasets to demonstrate the efficacy of the approach.**\n\nThank you for your constructive suggestions. In fact, we designed several baseline experiments to compare with our final RST-Att model. And we also conducted ablation studies on the involved multi-granularity attention mechanism to demonstrate the efficacy of our approach to introducing RST discourse structure. The relevant results are presented in Table 2 and Table 3 of our paper.\n\nFurthermore, we have supplemented an additional evaluation experiment based on the most frequently used contrastive test sets [1] in existing works. They were designed for targeted evaluation of several discourse phenomena: deixis, ellipsis (two test sets for VP ellipsis and morphological inflection), and lexical cohesion. Each test instance consists of a true example and several contrastive translations that differ from the true one only in the considered discourse aspect. The translation system needs to select the one that it considers to be true, which is evaluated by accuracy. We compared the results of existing research on document-level machine translation and popular large language models (LLMs) with our RST-Att model, as shown in the table below.\n\n| Model                  |  deixis  | lexical cohesion | ellipsis.infl | ellipsis.VP |\n| ---------------------- | :------: | :--------------: | :-----------: | :---------: |\n| Sent2Sent Baseline [1] |   50.0   |       45.9       |     53.0      |    28.9     |\n| Concat Baseline [1]    |   83.5   |       47.5       |     76.6      |    76.2     |\n| CADec [1]              |   81.6   |       58.1       |     72.2      |    80.0     |\n| LSTM-Transformer [2]   | **91.0** |       46.9       |     82.2      |    78.2     |\n| MR-Doc2Doc [3]         |   64.7   |       46.3       |     65.9      |    53.0     |\n| G-Transformer [4]      |   89.9   |        /         |     84.8      |    82.4     |\n| ChatGPT [5]            |   57.9   |       44.4       |     75.0      |    71.6     |\n| GPT4 [5]               |   85.9   |       72.4       |     69.8      |    81.4     |\n| RST-Att (ours)         |   87.2   |     **81.7**     |   **85.2**    |  **87.2**   |\n\nAlthough the time of rebuttal period is limited and our experiment is possibly not optimized to the best extent, our RST-Att model has achieved superior performance. It demonstrates the highest accuracy in three aspects: lexical cohesion, VP ellipsis, and morphological inflection, while having lower performance in deixis compared to SOTA. And the improvement in terms of lexical cohesion over previous methods is notably significant. Moreover, our approach outperforms LLMs comprehensively, including ChatGPT and GPT-4. It indicates the enhanced ability of our model to adapt to contextual information and handle discourse phenomena, which proves the promising way to introduce discourse information into document-level machine translation. If our paper is accepted, we will add the evaluation experiment mentioned above along with further relevant analysis to our paper.\n\n**Q2: How much slower is the decoding efficiency compared to translating the entire sentence? It is hoped that the authors can provide some experimental results related to translation efficiency.**\n\nThank you for another significant suggestion that helps to improve the quality of our work. Due to the limited time, we tested the sentence-to-sentence (Sent2Sent) and paragraph-to-paragraph (Para2Para) baselines, along with our RST-Att model. Previous works did not show the results in this regard, and we did not have enough time to reproduce them. We have compared the translation efficiency of the training and inference processes with the same settings, such as a single NVIDIA A40 GPU, the Fairseq framework, batch size, etc. In the table below, we present the relevant results on the NC-2016 dataset, and the performance on other datasets is similar.\n\n| Model              | Training time (seconds per epoch) | Inference time (seconds per epoch) |\n| ------------------ | :-------------------------------: | :--------------------------------: |\n| Sent2Sent Baseline |                906                |                 86                 |\n| Para2Para Baseline |               1055                |                112                 |\n| RST-Att            |               1127                |                115                 |\n\nThe training time consumed by the Para2Para2 baseline is about 1.16 times that of the Sent2Sent baseline, and the inference time is about 1.3 times. We suppose it is because a paragraph is considerably longer than a sentence, and the computational complexity of the transformer framework is quadratic with respect to input length. Our RST-Att model also adopts the input of paragraphs and does not introduce new training parameters. But it requires some computation for multi-granularity attention organization, so its time consumed is a bit greater than the Para2Para baseline.\n\nDespite slightly lower translation efficiency, our approach has improved the quality of document-level translation by better utilizing contextual content and discourse information. In addition, our multi-granularity attention mechanism actually can reduce computational overhead, as illustrated in Figure 4. Therefore, there is much room for further efficiency optimization, which was not primarily focused on in our work before. If our paper is accepted, we will add the relevant results and discussions, along with improvements to our model implementation.\n\n**Q3: This is still a traditional approach applied to translation models. How does it compare with the currently prevalent large language models (LLMs)? I would like to see comparative experimental results.**\n\nThere have been some studies assessing the performance of LLMs on prominent document-level machine translation datasets, using the prompt that translates the entire document at once. The results of the popular LLM ChatGPT compared to our RST-Att model are as follows, with the d-BLEU metric for evaluation.\n\n| Model          |  TED  | News  | Europarl |\n| -------------- | :---: | :---: | :------: |\n| ChatGPT [5]    | 33.6  | 39.4  |   30.4   |\n| RST-Att (ours) | 31.10 | 32.28 |  34.55   |\n\nChatGPT outperforms our RST-Att model on the TED and News datasets, while it lags behind on the Eurparl dataset. We speculate it may be due to the massive training data of ChatGPT. In fact, we cannot verify whether these datasets, particularly the test sets, have already been seen by ChatGPT, as they are publicly available and have been released early. Therefore, it is debatable whether these results are comparable. Recent research [6] has also focused on this problem and highlighted concerns regarding data contamination in downstream task test sets.\n\n**References:**\n\n[1] Elena Voita, Rico Sennrich, and Ivan Titov. 2019b. When a good translation is wrong in context: Context-aware machine translation improves on deixis, ellipsis, and lexical cohesion. In *Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics*, pages 1198–1212, Florence, Italy. Association for Computational Linguistics.\n\n[2] Pei Zhang, Boxing Chen, Niyu Ge, and Kai Fan. 2020. Long-short term masking transformer: A simple but effective baseline for document-level neural machine translation. In *Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)*, pages 1081–1087, Online. Association for Computational Linguistics.\n\n[3] Zewei Sun, Mingxuan Wang, Hao Zhou, Chengqi Zhao, Shujian Huang, Jiajun Chen, and Lei Li. 2022. Rethinking document-level neural machine translation. In *Findings of the Association for Computational Linguistics: ACL 2022*, pages 3537–3548.\n\n[4] Guangsheng Bao, Yue Zhang, Zhiyang Teng, Boxing Chen, and Weihua Luo. 2021. G-transformer for document-level machine translation. In *Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)*, pages 3442–3455.\n\n[5] Longyue Wang, Chenyang Lyu, Tianbo Ji, Zhirui Zhang, Dian Yu, Shuming Shi, Zhaopeng Tu. 2023. Document-level machine translation with large language models[J]. *arXiv preprint arXiv:2304.02210*.\n\n[6] Shahriar Golchin and Mihai Surdeanu. 2023. Time Travel in LLMs: Tracing Data Contamination in Large Language Models. *arXiv preprint arXiv:2308.08493*.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper addresses a significant challenge in the field of neural machine translation - the limitation of existing methods for document-level machine translation (DocMT) when processing lengthy texts. The proposed approach, which employs discourse structure analysis to enhance context utilization, is novel and well-motivated. The experimental results demonstrate the superiority of the proposed RST-Att model over existing approaches, showcasing its potential to advance the state-of-the-art in DocMT. The paper's clarity, solid theoretical foundation, and empirical validation make it a valuable contribution to the NLP community.", "reasons_to_reject": "While the topic of enhancing document-level machine translation through discourse structure analysis is intriguing, the paper lacks empirical validation and practical implementation. The proposed approach and models are described in detail, but there is a need for experimental results on relevant datasets to demonstrate the efficacy of the approach. Additionally, the paper could benefit from a more thorough comparison with existing methods in terms of performance and computational efficiency. As it stands, the manuscript lacks the necessary depth to make a meaningful contribution to the field.", "questions_to_authors": "1. How much slower is the decoding efficiency compared to translating the entire sentence? It is hoped that the authors can provide some experimental results related to translation efficiency;\n\n2. This is still a traditional approach applied to translation models. How does it compare with the currently prevalent large language models (LLMs)? I would like to see comparative experimental results.", "paper_topic_and_main_contributions": "1. Introducing a robust paragraph-to-paragraph translation mode that enhances context utilization compared to traditional text partition methods;\n\n2. Exploring diverse strategies to leverage discourse structure predicted by our RST parsing model, resulting in improved DocMT performance;\n\n3. The proposed RST-Att model outperforms existing methods on three prominent DocMT datasets. The authors will release the code and data to foster community engagement.\n\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "oZIMhMECSr", "reviewer": "Reviewer_JzyP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and efforts, as well as the insightful comments, and we address your concerns below:\n\n**Q1: Relies entirely on automatic RST parsing, which remains error-prone. Better discourse understanding would likely improve benefits.**\n\nThank you for your constructive suggestions. The incorporation of discourse information into document-level machine translation is the motivation and novelty of our work. And we primarily considered the RST discourse structure, so it is indeed necessary to rely on automatic RST parsing methods. In fact, the current performance of RST parsing is not unsatisfactory, and the parsing model we have trained demonstrates an F1 score of over 88% on Span in our evaluation. \n\nFurthermore, our approach has achieved improvements with our obtained RST discourse structural information. Given the possibility of further advancement on the RST parser (e.g., fine-tuned on powerful large language models (LLMs)), our work shows a promising direction to promote document-level translation with discourse parsing. And our approach can directly benefit from the corresponding advancements. We also agree that other better discourse understanding would likely improve benefits, and we intend to dig into this field in future research and attract more research attention. If our paper is accepted, we will add relevant explanations as above.\n\n**Q2: It seems that such RST parsers are mainly available for English (as such, the submission also mainly conducted evaluation from English to German), making it hard to generalize to other languages.**\n\nThank you for your concerns. The three English-to-German datasets in our experiments are the most prominent and most commonly used for document-level machine translation tasks, consistent with many existing works. Moreover, we agree that multilingual issues remain a challenge in the field of discourse parsing, primarily due to the scarcity of well-annotated multilingual corpora. However, with the advancement of cross-lingual discourse parsing research, including the use of LLMs in data augmentation, we believe that relevant parsing tools will soon be available. And they can simliarly be integrated into our approach, thereby improving document-level translation for other languages. We also consider the aforementioned multilingual issues as potential directions for future research.\n\n**Q3: Using external TextTiling tool may limit end-to-end learning of document boundaries.**\n\nOur method does not involve changing the original document boundaries. Instead, we use the TextTiling tool to split each document into several paragraphs with strong internal correlations. In fact, previous studies on document-level machine translation did not conduct end-to-end document boundary learning, namely standard document-to-document translation. This approach has been shown in some works to be prone to training failures. So many studies adopted a method of splitting each document into sub-documents based on a specified maximum length for better training.\n\nAs we pointed out in the introduction section, we think this common mode may arbitrarily ignore the interactions between strongly related sentences in the document. In contrast, our proposed paragraph-to-paragraph method is more reasonable and is expected to make better use of contextual information. We provide a comparison between a sub-document (SDoc2SDoc) translation baseline and our Para2Para baseline, both without changes in model frameworks, as shown in the table below. Our paragraph-to-paragraph translation mode itself demonstrates better performance than the traditional SDoc2SDoc baseline before introducing discourse information, which proves the effectiveness of our new translation mode.\n\n| Model                  |  TED  | News  | Europarl |\n| ---------------------- | :---: | :---: | :------: |\n| SDoc2SDoc Baseline [1] | 28.29 | 30.49 |  34.00   |\n| Para2Para Baseline     | 30.35 | 31.43 |  34.19   |\n\n**Q4: Limited linguistic analysis of what discourse phenomena the model has learned to handle.**\n\nThank you for your constructive suggestions. We have supplemented an additional evaluation experiment based on the most frequently used contrastive test sets [2] in existing works. They were designed for targeted evaluation of several discourse phenomena: deixis, ellipsis (two test sets for VP ellipsis and morphological inflection), and lexical cohesion. Each test instance consists of a true example and several contrastive translations that differ from the true one only in the considered discourse aspect. The translation system needs to select the one that it considers to be true, which is evaluated by accuracy. We compared the results of existing research on document-level machine translation and popular large language models (LLMs) with our RST-Att model, as shown in the table below.\n\n| Model                  |  deixis  | lexical cohesion | ellipsis.infl | ellipsis.VP |\n| ---------------------- | :------: | :--------------: | :-----------: | :---------: |\n| Sent2Sent Baseline [2] |   50.0   |       45.9       |     53.0      |    28.9     |\n| Concat Baseline [2]    |   83.5   |       47.5       |     76.6      |    76.2     |\n| CADec [2]              |   81.6   |       58.1       |     72.2      |    80.0     |\n| LSTM-Transformer [3]   | **91.0** |       46.9       |     82.2      |    78.2     |\n| MR-Doc2Doc [4]         |   64.7   |       46.3       |     65.9      |    53.0     |\n| G-Transformer [1]      |   89.9   |        /         |     84.8      |    82.4     |\n| ChatGPT [5]            |   57.9   |       44.4       |     75.0      |    71.6     |\n| GPT4 [5]               |   85.9   |       72.4       |     69.8      |    81.4     |\n| RST-Att (ours)         |   87.2   |     **81.7**     |   **85.2**    |  **87.2**   |\n\nAlthough the time of rebuttal period is limited and our experiment is possibly not optimized to the best extent, our RST-Att model has achieved superior performance. It demonstrates the highest accuracy in three aspects: lexical cohesion, VP ellipsis, and morphological inflection, while having lower performance in deixis compared to SOTA. And the improvement in terms of lexical cohesion over previous methods is notably significant. Moreover, our approach outperforms LLMs comprehensively, including ChatGPT and GPT-4. It indicates the enhanced ability of our model to adapt to contextual information and handle discourse phenomena, which proves the promising way to introduce discourse information into document-level machine translation. If our paper is accepted, we will add the evaluation experiment mentioned above along with further relevant analysis to our paper.\n\n**Q5: About reproducibility.**\n\nIn fact, the datasets involved in our work are common and widely used in existing research. For RST parsing, we will provide the corresponding training codes for convenient reproducing. We will also release our trained parsing model, which can be directly used. Regarding document-level translation, we will release the discourse parsing information of the relevant datasets, along with codes for our models and methods, to ensure the reproducibility of our work.\n\n**References:**\n\n[1] Guangsheng Bao, Yue Zhang, Zhiyang Teng, Boxing Chen, and Weihua Luo. 2021. G-transformer for document-level machine translation. In *Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)*, pages 3442–3455.\n\n[2] Elena Voita, Rico Sennrich, and Ivan Titov. 2019b. When a good translation is wrong in context: Context-aware machine translation improves on deixis, ellipsis, and lexical cohesion. In *Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics*, pages 1198–1212, Florence, Italy. Association for Computational Linguistics.\n\n[3] Pei Zhang, Boxing Chen, Niyu Ge, and Kai Fan. 2020. Long-short term masking transformer: A simple but effective baseline for document-level neural machine translation. In *Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)*, pages 1081–1087, Online. Association for Computational Linguistics.\n\n[4] Zewei Sun, Mingxuan Wang, Hao Zhou, Chengqi Zhao, Shujian Huang, Jiajun Chen, and Lei Li. 2022. Rethinking document-level neural machine translation. In *Findings of the Association for Computational Linguistics: ACL 2022*, pages 3537–3548.\n\n[5] Longyue Wang, Chenyang Lyu, Tianbo Ji, Zhirui Zhang, Dian Yu, Shuming Shi, Zhaopeng Tu. 2023. Document-level machine translation with large language models[J]. *arXiv preprint arXiv:2304.02210*.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Same as main contributions", "reasons_to_reject": "1. Relies entirely on automatic RST parsing, which remains error-prone. Better discourse understanding would likely improve benefits.\n2. It seems that such RST parsers are mainly available for English (as such, the submission also mainly conducted evaluation from English to German), making it hard to generalize to other languages.\n2. Using external TextTiling tool may limit end-to-end learning of document boundaries. \n3. Limited linguistic analysis of what discourse phenomena the model has learned to handle.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper explores utilizing discourse structure for document-level neural machine translation (DocMT). The authors introduce a paragraph-to-paragraph translation approach and generate paragraph segmentation using TextTiling. They train an end-to-end RST parser to predict discourse structure and propose several methods to incorporate it into DocMT, including integrating the linearized RST sequence (RST-Seq) and an RST tree-based multi-granularity attention mechanism (RST-Att). Experiments on TED, News, and Europarl datasets show their RST-Att model achieves the best results, outperforming prior work on DocMT.\n\n**Main contributions:**\n\n1. Introduces a more principled paragraph-level translation approach compared to prior segmentation methods in DocMT.\n2. Presents several concrete attempts to utilize predicted RST structure, with ablation studies showing the benefits of RST-specific guidance and multiple granularity levels.\n", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "coN9Meyn5l", "reviewer": "Reviewer_EkAb", "summary": "", "questions": "-Lines 188-191: The authors mentioned, \"TextTiling method has a more reasonable distribution, and the variance of the numbers of tokens and sentences is smaller.\" I am not confident that this claim is established since the variance of the number of tokens and sentences is an inherent feature of the language, which could be large. I hope the authors can explain more about this claim.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and efforts, as well as the insightful comments, and we address your concerns below:\n\n**Q1: The notation for the \"Para2Para Baseline (BART)\" lacks a formal definition.** \n\nThank you for your constructive suggestions. We apologize for the lack of a formal definition for this notation, and your understanding is correct. The corresponding description in our paper can be found in line 375. And there was no utilization of RST tree information or modification made to the model architecture in this baseline setting. Instead, it simply performed paragraph-to-paragraph translation based on the paragraphs we had previously segmented. If our paper is accepted, we will improve and clarify the definition of this notation.\n\n**Q2: I can not find enough support analysis about the ablation study about where the improvement of Para2Para Baseline (BART) comes from. It would be great if the authors could add more analysis.**\n\nThank you for your constructive suggestions. We believe that the improvement of Para2Para Baseline mainly comes from the paragraph translation mode we proposed. For better illustration, we have supplemented the results of the traditional sub-document translation baseline. As described in our introduction section, some studies have pointed out that the standard document-to-document translation mode is prone to training failures. Therefore, many studies adopted the method of splitting each document into sub-documents based on a specified maximum length for better training. In the table below, we provide the comparison between the sub-document (SDoc2SDoc) baseline, our Para2Para baseline, and the previous state-of-the-art model G-Transformer, all fine-tuned on mBART.\n\n| Model fine-tuned on mBART |  TED  | News  | Europarl |\n| ------------------------- | :---: | :---: | :------: |\n| SDoc2SDoc Baseline [1]    | 28.29 | 30.49 |  34.00   |\n| G-Transformer [1]         | 30.03 | 31.71 |  34.31   |\n| Para2Para Baseline        | 30.35 | 31.43 |  34.19   |\n\nOur Para2Para baseline outperforms the traditional SDoc2SDoc baseline, and the latter actually performs worse than many existing methods on both the TED and Europarl datasets. It can be attributed to the fact that the simple SDoc2SDoc mode, without any other improvements, may arbitrarily ignore the interactions between strongly related sentences. In contrast, our proposed paragraph-to-paragraph method is more reasonable and is expected to make better use of contextual information. This is because our paragraphs are segmented under consideration of the internal correlations of sentences in the paragraph. The results also prove the effectiveness of our new translation mode.\n\nMoreover, while the Para2Para baseline outperforms G-Transformer on the TED dataset, it is inferior on the other two datasets. For further improvements, we introduced discourse information and proposed our RST-Att model, which has achieved better translation performance. If our paper is accepted, we will add more relevant analysis as above.\n\n**Q3: Lines 188-191: The authors mentioned, \"TextTiling method has a more reasonable distribution, and the variance of the numbers of tokens and sentences is smaller.\" I am not confident that this claim is established since the variance of the number of tokens and sentences is an inherent feature of the language, which could be large. I hope the authors can explain more about this claim.**\n\nThank you for your concerns. We apologize that our expression here is not quite accurate. And we agree that different paragraphs should have certain differences in terms of word and sentence counts. What we intended to convey is that, when comparing the two segmentation methods, the TextTiling tool yields a more reasonable distribution of paragraph lengths, concentrating in a moderate number of sentences (4–7 sentences), as shown in Figures 1 and 2. In contrast, the other method results in a large number of short segments (1–3 sentences), which somewhat deviate from the concept of discourse and are not suitable for discourse parsing. If our paper is accepted, we will modify this statement as follows: \"The paragraphs from the TextTiling method have a more reasonable distribution, and most of them contain moderate numbers of tokens and sentences.\"\n\n**Q4: Supplementary evaluation experiments and analysis of discourse phenomena.**\n\nWe have supplemented an additional evaluation experiment based on the most frequently used contrastive test sets [2] in existing works. They were designed for targeted evaluation of several discourse phenomena: deixis, ellipsis (two test sets for VP ellipsis and morphological inflection), and lexical cohesion. Each test instance consists of a true example and several contrastive translations that differ from the true one only in the considered discourse aspect. The translation system needs to select the one that it considers to be true, which is evaluated by accuracy. We compared the results of existing research on document-level machine translation and popular large language models (LLMs) with our RST-Att model, as shown in the table below.\n\n| Model                  |  deixis  | lexical cohesion | ellipsis.infl | ellipsis.VP |\n| ---------------------- | :------: | :--------------: | :-----------: | :---------: |\n| Sent2Sent Baseline [2] |   50.0   |       45.9       |     53.0      |    28.9     |\n| Concat Baseline [2]    |   83.5   |       47.5       |     76.6      |    76.2     |\n| CADec [2]              |   81.6   |       58.1       |     72.2      |    80.0     |\n| LSTM-Transformer [3]   | **91.0** |       46.9       |     82.2      |    78.2     |\n| MR-Doc2Doc [4]         |   64.7   |       46.3       |     65.9      |    53.0     |\n| G-Transformer [1]      |   89.9   |        /         |     84.8      |    82.4     |\n| ChatGPT [5]            |   57.9   |       44.4       |     75.0      |    71.6     |\n| GPT4 [5]               |   85.9   |       72.4       |     69.8      |    81.4     |\n| RST-Att (ours)         |   87.2   |     **81.7**     |   **85.2**    |  **87.2**   |\n\nAlthough the time of rebuttal period is limited and our experiment is possibly not optimized to the best extent, our RST-Att model has achieved superior performance. It demonstrates the highest accuracy in three aspects: lexical cohesion, VP ellipsis, and morphological inflection, while having lower performance in deixis compared to SOTA. And the improvement in terms of lexical cohesion over previous methods is notably significant. Moreover, our approach outperforms LLMs comprehensively, including ChatGPT and GPT-4. It indicates the enhanced ability of our model to adapt to contextual information and handle discourse phenomena, which proves the promising way to introduce discourse information into document-level machine translation. If our paper is accepted, we will add the evaluation experiment mentioned above along with further relevant analysis to our paper.\n\n**References:**\n\n[1] Guangsheng Bao, Yue Zhang, Zhiyang Teng, Boxing Chen, and Weihua Luo. 2021. G-transformer for document-level machine translation. In *Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)*, pages 3442–3455.\n\n[2] Elena Voita, Rico Sennrich, and Ivan Titov. 2019b. When a good translation is wrong in context: Context-aware machine translation improves on deixis, ellipsis, and lexical cohesion. In *Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics*, pages 1198–1212, Florence, Italy. Association for Computational Linguistics.\n\n[3] Pei Zhang, Boxing Chen, Niyu Ge, and Kai Fan. 2020. Long-short term masking transformer: A simple but effective baseline for document-level neural machine translation. In *Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)*, pages 1081–1087, Online. Association for Computational Linguistics.\n\n[4] Zewei Sun, Mingxuan Wang, Hao Zhou, Chengqi Zhao, Shujian Huang, Jiajun Chen, and Lei Li. 2022. Rethinking document-level neural machine translation. In *Findings of the Association for Computational Linguistics: ACL 2022*, pages 3537–3548.\n\n[5] Longyue Wang, Chenyang Lyu, Tianbo Ji, Zhirui Zhang, Dian Yu, Shuming Shi, Zhaopeng Tu. 2023. Document-level machine translation with large language models[J]. *arXiv preprint arXiv:2304.02210*.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "-This idea of splitting documents into paragraphs and using an RST tree in machine translation is interesting and inspiring.\n\n-This paper proposes a new mode for document translation that splits documents into paragraphs first and then translates paragraphs. This paper also proposes new models to utilize discourse information through attention better to achieve better performance and reduce computation complexity. According to experiments, this method performs well by outperforming the Sota method when this work is done.", "reasons_to_reject": "-The notation for the \"Para2Para Baseline (BART)\" lacks a formal definition. If I understand correctly, this baseline means the method that performs paragraph-to-paragraph translation but without using RST tree information. It would be better if the authors could add a definition for this notation.\n-If my understanding of the first point about \"Para2Para Baseline (BART)\" is correct, it indicates the baseline could already achieve great performance (almost outperforming most previous works). I can not find enough support analysis about the oblation study about where this improvement comes from. It would be great if the authors could add more analysis.", "questions_to_authors": "-Lines 188-191: The authors mentioned, \"TextTiling method has a more reasonable distribution, and the variance of the numbers of tokens and sentences is smaller.\" I am not confident that this claim is established since the variance of the number of tokens and sentences is an inherent feature of the language, which could be large. I hope the authors can explain more about this claim.", "paper_topic_and_main_contributions": "This paper presents a new paragraph-to-paragraph model for document machine translation. This method firstly uses TextTiling to splits the document into paragraphs and parsers each paragraph into an RST tree. The RST tree is further linearized into a sequence to replace the original text input for the seq2seq model. The author further proposes a method using attention to better utilize the According to the experiment, this method achieves good results, especially the RST-Att method, which achieves sota performance when this work is done. \n\nIn general, the idea in this paper is well-motivated and achieves good performance. But this work still needs more analysis and experiments to figure out where the performance comes from.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Lrk1nBGWNz", "reviewer": "Reviewer_JdmR", "summary": "", "questions": "It is quite common to also look at performance on some dedicated contrastive test sets to further justify the improvements from utilizing document-level context. Have you checked those? How does your method compare to the baseline?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and efforts, as well as the insightful comments, and we address your concerns below:\n\n**Q1: In Table 2, have all models seen the same amount of data? If not, how different?**\n\nThank you for your constructive suggestions. In Table 2, we have indicated the pretrained language model (PLM) in parentheses if the corresponding work used it. And other works did not employ PLMs, or their methods could not be directly applied to PLMs. Moreover, all works in Table 2 were trained on the same document-level translation dataset without additional corpora. Given the prevalent use of PLMs in current NLP research and the good generalization of our method, we conduct our experiments directly based on mBART. The setting aligns with the state-of-the-art model G-Transformer in previous works, which serves as our primary comparison object.\n\nOn the other hand, we would like to argue that the improvements of our method do not mainly come from extra pre-training data, such as knowledge in mBART. We have supplemented the results of the traditional sub-document translation baseline. As described in our introduction section, some research has pointed out that the standard document-to-document translation mode is prone to training failures. Therefore, many studies adopted the method of splitting each document into sub-documents based on a specified maximum length for better training. In the table below, we provide the comparison between the sub-document (SDoc2SDoc) baseline, our Para2Para baseline, and G-Transformer, all fine-tuned on mBART.\n\n| Model fine-tuned on mBART |  TED  | News  | Europarl |\n| ------------------------- | :---: | :---: | :------: |\n| SDoc2SDoc Baseline [1]    | 28.29 | 30.49 |  34.00   |\n| G-Transformer [1]         | 30.03 | 31.71 |  34.31   |\n| Para2Para Baseline        | 30.35 | 31.43 |  34.19   |\n\nOur Para2Para baseline outperforms the SDoc2SDoc baseline with the same PLM, and the latter actually performs worse than many existing methods on both the TED and Europarl datasets. It can be attributed to the fact that the simple SDoc2SDoc mode, without any other improvements, may arbitrarily ignore the interactions between strongly related sentences. In contrast, our proposed paragraph-to-paragraph method is more reasonable and is expected to make better use of contextual information. The results also prove the considerable improvements from our new paragraph-to-paragraph translation mode itself. Furthermore, we introduced discourse information and proposed our RST-Att model, which has achieved better translation performance. Finally, we agree that the description in Table 2 is not sufficiently clear. If our paper is accepted, we will improve Table 2 with more information on data conditions as well as the relevant discussions above.\n\n**Q2: While the linearization and multi-gran.-atten steps are interesting, I feel simple baselines are missing to justify the method.**\n\nThank you for your constructive suggestions. In fact, we have experimented before with the baseline you mentioned, which involves adding <sep> tokens to denote EDU segmentation boundaries. However, the performance was basically the same as not adding these tokens, namely the Para2Para Baseline in Table 2. Furthermore, we have also attempted to initialize these segmentation tokens with the semantic representations of the corresponding EDUs to enhance relevant information, but this did not lead to improvements either. As a result, we did not include these baselines in our final paper. We suppose that these simple methods may not be sufficient to help the model leverage discourse information and improve translation performance.\n\nBased on the above attempts, our Para2Para baseline, together with the further RST-Seq setting presented in Table 2, can justify the effectiveness of the linearization step. Additionally, our ablation studies in Table 3 prove that the introduction of the RST parsing tree and the involved multi-granularity attention mechanism indeed further improve translation performance. Moreover, we agree that more baselines and analyses can enhance the clarity of our paper. If our paper is accepted, we will supplement the discussions regarding our attempts mentioned above and carefully improve our experiment section.\n\n**Q3: It is quite common to also look at performance on some dedicated contrastive test sets to further justify the improvements from utilizing document-level context. Have you checked those? How does your method compare to the baseline?**\n\nThank you for your constructive suggestions. We have supplemented an additional evaluation experiment based on the most frequently used contrastive test sets [2] in existing works. They were designed for targeted evaluation of several discourse phenomena: deixis, ellipsis (two test sets for VP ellipsis and morphological inflection), and lexical cohesion. Each test instance consists of a true example and several contrastive translations that differ from the true one only in the considered discourse aspect. The translation system needs to select the one that it considers to be true, which is evaluated by accuracy. We compared the results of existing research on document-level machine translation and popular large language models (LLMs) with our RST-Att model, as shown in the table below.\n\n| Model                  |  deixis  | lexical cohesion | ellipsis.infl | ellipsis.VP |\n| ---------------------- | :------: | :--------------: | :-----------: | :---------: |\n| Sent2Sent Baseline [2] |   50.0   |       45.9       |     53.0      |    28.9     |\n| Concat Baseline [2]    |   83.5   |       47.5       |     76.6      |    76.2     |\n| CADec [2]              |   81.6   |       58.1       |     72.2      |    80.0     |\n| LSTM-Transformer [3]   | **91.0** |       46.9       |     82.2      |    78.2     |\n| MR-Doc2Doc [4]         |   64.7   |       46.3       |     65.9      |    53.0     |\n| G-Transformer [1]      |   89.9   |        /         |     84.8      |    82.4     |\n| ChatGPT [5]            |   57.9   |       44.4       |     75.0      |    71.6     |\n| GPT4 [5]               |   85.9   |       72.4       |     69.8      |    81.4     |\n| RST-Att (ours)         |   87.2   |     **81.7**     |   **85.2**    |  **87.2**   |\n\nAlthough the time of rebuttal period is limited and our experiment is possibly not optimized to the best extent, our RST-Att model has achieved superior performance. It demonstrates the highest accuracy in three aspects: lexical cohesion, VP ellipsis, and morphological inflection, while having lower performance in deixis compared to SOTA. And the improvement in terms of lexical cohesion over previous methods is notably significant. Moreover, our approach outperforms LLMs comprehensively, including ChatGPT and GPT-4. It indicates the enhanced ability of our model to adapt to contextual information and handle discourse phenomena, which proves the promising way to introduce discourse information into document-level machine translation. If our paper is accepted, we will add the evaluation experiment mentioned above along with further relevant analysis to our paper.\n\n**Q4: Typos Grammar Style And Presentation Improvements:**\n\nThank you for your constructive suggestions. We apologize for our oversight, and we agree that adding more details about the previous works in our Table 2 can help provide readers with an overview and a more straightforward comparison. Due to the space limitation, it is challenging to incorporate additional columns containing so much relevant information into the table of experimental results. Therefore, we consider adding an extra table in the appendix to introduce these detailed setups. If our work is accepted, we will carefully address these improvements.\n\n**References:**\n\n[1] Guangsheng Bao, Yue Zhang, Zhiyang Teng, Boxing Chen, and Weihua Luo. 2021. G-transformer for document-level machine translation. In *Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)*, pages 3442–3455.\n\n[2] Elena Voita, Rico Sennrich, and Ivan Titov. 2019b. When a good translation is wrong in context: Context-aware machine translation improves on deixis, ellipsis, and lexical cohesion. In *Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics*, pages 1198–1212, Florence, Italy. Association for Computational Linguistics.\n\n[3] Pei Zhang, Boxing Chen, Niyu Ge, and Kai Fan. 2020. Long-short term masking transformer: A simple but effective baseline for document-level neural machine translation. In *Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)*, pages 1081–1087, Online. Association for Computational Linguistics.\n\n[4] Zewei Sun, Mingxuan Wang, Hao Zhou, Chengqi Zhao, Shujian Huang, Jiajun Chen, and Lei Li. 2022. Rethinking document-level neural machine translation. In *Findings of the Association for Computational Linguistics: ACL 2022*, pages 3537–3548.\n\n[5] Longyue Wang, Chenyang Lyu, Tianbo Ji, Zhirui Zhang, Dian Yu, Shuming Shi, Zhaopeng Tu. 2023. Document-level machine translation with large language models[J]. *arXiv preprint arXiv:2304.02210*.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well-written and easy to follow.\n\nThe proposed method to segment -> parse -> linearize -> attend is interesting and linguistically well-motivated.\n\nThe best setup (RST-Att) outperforms previous work on document Bleu scores.", "reasons_to_reject": "The experiment section needs to be improved.\n\nFor example, the proposed method is finetuned from pretrained BART - a natural question to ask is, in Table 2, have all models seen the same amount of data? If not, how different? Adding a column about this could help the reader to better understand the data conditions. Without it, one can argue that the improvements mainly come from the extra data seen during pre-training.\n\nWhile the linearization and multi-gran.-atten steps are interesting, I feel simple baselines are missing to justify the method. For instance, in Figure 3, after the EDU segmentation step, one can simply insert artificial <sep> tokens to denote the boundaries. Including such a baseline can help better justify the RST parsing tree step and the linearization step. Note that the multi-gran.-atten can still be applied in such a case.\n\nBasically, my impression after reading the paper was, \"hmm, the idea is interesting, but I just saw a bunch of d-Bleu scores that are better than some other work, where does the improvement really come from?\" I feel carefully redesigning some baselines/ablations could significantly improve the clarity of the paper.", "questions_to_authors": "It is quite common to also look at performance on some dedicated contrastive test sets to further justify the improvements from utilizing document-level context. Have you checked those? How does your method compare to the baseline?", "paper_topic_and_main_contributions": "This paper deals with the task of document-level NMT.\nThe authors proposed a pipeline, where the paragraphs are first segmented by the TextTiling algorithm, then RST parsing is used to predict the EDUs and construct the RST trees, a linearization step is then used to convert the tree into a 1-dimensional sequence with special syntax to preserve the tree information, and finally a multi-granularity attention is used where early encoder layers are constrained to process local information and later encoder layers are allowed to access global information.\n\nOverall, I think the method is interesting but the experiments are lacking.\nWhile the authors showed decent document Bleu score improvements compared to previous work, I feel several baselines are missing to justify the improvements.\n\nEDIT: After reading the author's rebuttal, many of my concerns are resolved. I feel if the authors can incorporate the points in the rebutal into the next version of the paper, e.g. including the clearly the pretraining data conditions, the contrastive test set performances, and detailed discussions about how a simple separator approach is worse than their method, the paper can be of better quality. Considering this and encouraged by the authors' ethusiasm in responding to questions and willingness to improve the paper, I decided to improve my soundness rating from 2 to 3.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "My biggest concern is Table 2, while it is not uncommon to see tables like this nowadays, I have to say that as a reader, little can be learned.\nEach row is behind a codename, which, unless one is super familiar with the work, tells not much about the actual data/model/training/search setup.\nAs a result, apart from \"this work is about +1 absolute d-Bleu better\", not much be can summarized from the table.\nI would prefer if relevant information like \"pretrained or not?\", \"training data size\", \"training criterion\", \"attention variant\", etc. can be listed as additional columns to aid the understanding of the reader."}}]

Author rebuttal:

Meta-review: This paper proposes an approach to document-level NMT that uses a rhetorical structure theory (RST) parser to capture discourse information. It explores different methods for adding this information to the encoder, notably a custom attention mechanism that uses the parse to control the context available to different layers. This is shown to outperform a wide range of baselines across different test settings.

Reviewers noted that the paper tackles a significant problem, and found the approach novel,  well-motivated, and interesting, with  positive results from extensive experiments, including ablation studies. The main weakness is the reliance on the RST parser, which is not guaranteed to be free from errors, and which limits source languages to those for which such a parser is available. Some reviewers also felt that analysis specifically linking quality improvements to linguistic aspects of the context would have strengthened the paper.

This is substantial work that represents a very strong baseline for document-level NMT. Although it’s hard to see it as the future of document-level MT - given the pipelined design, specially-modified architecture, and reliance on custom linguistic components - it is nonetheless an interesting point for comparison and analysis.
Decision: Accept-Main