TRIP: Accelerating Document-level Multilingual Pre-training via Triangular Document-level Pre-training on Parallel Data Triplets
Authors: Hongyuan Lu
Haoyang Huang
Shuming Ma
Dongdong Zhang
Wai Lam
Zhaochuan Gao
Anthony Aue
Arul Menezes
Furu Wei
Venue: EMNLP 2023 Findings (2023)
Keywords: multilingual pre-training; machine translation; cross-lingual summarization
TL;DR: using triplet parallel data pairs improves multilingual pre-training

Abstract: Despite the success of multilingual sequence-to-sequence pre-training, most existing approaches rely on document-level monolingual corpora in many different languages, sentence-level bilingual corpora,\footnote{In this paper, we use bilingual corpora to denote parallel corpora with bilingual translation pairs in many different language pairs, each consisting of two sentences/documents with the same meaning written in different languages. We use trilingual corpora to denote parallel corpora with trilingual translation pairs in many different language combinations, each consisting of three sentences/documents.} and sometimes synthetic document-level bilingual corpora. This hampers the performance with cross-lingual document-level tasks such as document-level translation. Hence, we propose to mine and leverage document-level trilingual parallel corpora to improve sequence-to-sequence multilingual pre-training. We present \textbf{Tri}angular Document-level \textbf{P}re-training (\textbf{TRIP}) as the first in the field to accelerate the conventional monolingual and bilingual objectives into a trilingual objective with a novel method called Grafting. Experiments show that TRIP achieves several strong state-of-the-art (SOTA) scores on three multilingual document-level machine translation benchmarks and one cross-lingual abstractive summarization benchmark, including consistent improvements by up to 3.11 d-BLEU points and 8.9 ROUGE-L points.

TRIP: Accelerating Document-level Multilingual Pre-training via Triangular Document-level Pretraining on Parallel Data Triplets*
Hongyuan Lu Haoyang Huang Shuming Ma Dongdong Zhang Wai Lam", Zhaochuan Gao Anthony A Arul Menezes Furu Wei
The Chinese University of Hong Kong
Microsoft Corporation
[hylu,wlam] @ee.culle dese.cukk.ede edu.
@microsoof.com
Abstract
Lewis et al., 2020b). Following the calls that the unsupervised scenario is not strictly realistic for cross-lingual learning (Artetxe et al., 2020), mul- tilingual pre-training advanced into a supervised setting through sentence-level bilingual translation pairs (Chi et al., 2021; Reid and Artetxe, 2022) to provide stronger signal for pre-training. Among these pioneering works, document-level multilin- gual pre-training with parallel data is currently an understudied topic. This direction is particu- larly significant for tasks that necessitate contextual comprehension, such as document-level machine translation and cross-lingual summarization. As workaround, DOCmT5 (Lee et al., 2022) resorts to using synthetic bilingual translation pairs to scale up document-level multilingual pre-training.
Despite the success of multilingual sequence- lo-sequence pre-training, most existing ap- proaches rely on document-level monolingual corpora in many different languages, sentence- level bilingual corpora, ! and sometimes syn- thetic document-leve billinuual corpora. This hampers the performance with cross-lingual document-level tasks such as document-leved translation. Hence, we propose to mine and leverage document-level trillngual parallel cor- multi-
pora to improve sequence-to-sequee lingual pre- treitrain We present Triangular Document-leved Pre-training (TRIP) the first in the field to accelerate the conventional mono- lingual and bilingual objectives into a trilingual objective with novel method called Grafting. Experiments show that TRIP achieves several strong state-of-thed -fett-rartt (SOTA) scores on three multilingual document- level machine transla- tion benchmarks and one cross-lingual abstrac- tive summarization benchmark, including con- sistent improvements by up to 3.11 d-BLEU points and 8.9 ROUGE-L points.
In addition to the lack of study for document- level multilingual pre-training with parallel data, prior works also overlooked the value of trilingual parallel data for multilingual pre-training. Com- pared to bilingual parallel data, trilingual paral- lel data is expected to better capture different lin- guistic clues and coherence among different lan- guages such as past tense and gendered expres- sions, which can enhance the model pre-training on aspects of document- cross- looss-lnngual under- standing and resolve cross-lingual ambiguities.
1 Introduction
Conventional multilingual pre-training achieved promising results on machine translation (Liu et al., 2020) and cross-lingual classification (Xue et al.. 2021). These pre-training paradigms usually rely monolingual corpora in many different lan- guages, with denoising objectives such as sentence permutation and span masking (Liu al., 2020;
To this end, present TRIP, lecument-level multilingual pre-training method using trilingual parallel corpora. Because there is no publicly avail- able document-leved trilingual corpus, we propose a novel method to construct trilingual document pairs from document-level bilingual corpora. Sub- sequently, we augment the conventional multilin- gual pre-training by (i) Grafting two documents pre- sented in two different languages into one mixed document, and (ii) predicting the remaining one language as the reference translation.
"The work described in this paper is substantially sup- ported by grant from the Research Grant Council of the Hong Kong Special Administrative Region, China (Project Code: 14200719).
"Contribution during internship Microsoft Research
Asia.
'In this paper, we use 'bilingual corpora' to denote parallel corpora with "bilingual translation pairs' in many different language pairs, each consisting of two sentences/documents with the same meaning written in different languages. We use trilingual corpora' to denote parallel corpora with "trilingual translation pairs' in many different language combinations, each consisting of three sentences/documants.
For example, Chinese does not have past tense for verbs, while Japanese and English do have relevant clues. See Figure I for further explanation.

---

Models
Demoising
Tranelation
Tralingnal
Triingua
Ducument
Making the use of sentence-level translation pairs bilingual pre-training strategy for multilingual models (Kale et al., 2021; Tang et al., 2021).
Translation Pre-training:
Pre-trsining Pre-training
Decument Pairs Objective
Land
X
X
X
IIAAT
X
X
X
X
X
DTT6
PARADISE
V
X
X
X
Constructing Trilingual Objective In com- parison, TRIP is the first in the field to introduce trilingual objective for multilingual pre-training. The core making better of trilingual data is to Grafting' the documents by splitting the docu- ments written in two different languages but with the same meaning half by half and concatenating each half to form a new document that retains the same meaning written in two different languages. TRIP then applies sentence permutation and span corruption the Grafted documents.
DOOmT5
V
X
X
V
TRIP
V
V
Table 1: Comparisons of various multilingual pre- training methods. We denote the intermediate value O. For example, mT5 uses span corrptiion solely without sentence permutation, so we put value of O for the column of Denoising fre-training for mT5. The columns of Denoising Pre-training and Translation Pre-training refer to the pre-training objectives we in- troduce at the start Section
Conventional monolingual and bilingual pre- training objectives overlooked the value to take such an advantage (Liu et al., 2020; Reid and Artetxe, 2022) of linguistic clues from different languages. In contrast, TRIP fuses authentic trilin- gual data, in which linguistic clues such as past tense and gendered nouns are usually preserved.
We conduct experiments on document-level ma- chine translation TED Talks (Cettolo et al., 2015), News benchmark (News-commentary) and Europarl (Koehn, 2005), and cross-limg ponssll abstrac- tive summarization on Wikilingua (Ladhak et al., 2020; Gchrmann et al., 2021). We found that TRIP clearly improves previous multilingual pre-training paradigms that use monolingual and bilingual ob- jectives (Lee et al., 2022), and achieves strong SOTA results on both tasks.
We present in Figure to illustrate how TRIP op- erates make use of linguistic clues through trilin- gual data. Given three documents with the same meaning written in Chinese, Japanese, and English, two of the documents are split and concatenated. The concatenation is randomly permutated at the sentence level, and the remaining unchanged docu- ment is used as the translation reference. Here, Chi- nese is tenseless, and TRIP effectively fuses useful linguistic clues for past tense written in Japanese and English into the Chinese text to resolve cross- lingual ambiguities.
In summary, we make three key contributions:
TRIP proposes a novel trilingual pre-training objective through Grafting for multilingual pre-training, along with a novel method to construct trilingual data from parallel corpora.
TRIP yields SOTA scores on both multilingual document-level MT and cross- moossllnngual abstrac- tive summarization.
Table presents the characteristics that TRIP exhibits compared to existing methods. We report whether the models use trilingual document pairs for pre-training, and we report whether document- level tasks such document-level machine trans- lation or abstractive summarization are reported in their original papers. To our best knowledge, this is the first paper in our field to mine and use trilingual document pairs for multilingual pre-training. This is also the first work that features Grafting.
We conduct in-depth analyses on document- level cross-lingual understanding and com- pare TRIP to commercial systems.
2 Triangular Document-level Pre-training
We start by introducing the conventional method- ologies previously used by the monolingual and bilingual objectives for multilingual pre-training:
More formally, we first denote N as the num- ber of training document pairs in trilingual transla- tion triplets of (x1, 22, 2z) in pre-training corpus D. Given a Seq2Seq generation model (Sutskever
Denoising Pre-training: Sentence permuta- tion (Liu al., 2020) and span corruption (Xue et al., 2021) are effective denoising pre- training objectives for document-level multi- lingual pre-training.
'Grffting refers to joining two plants together by cutting and using scion (the upper part of the grafting) as the top and the understock (the lower part of the grafting) as the root.

---

Itr anteenen
J00a VOL Good mode See See UAMARRS
Thiingual Document
MASKED
EANEL. HEL.
Garding
231 & Min XL
The rerroener wer enlly
Tlank IMA MASSEO
M MASS 1
Lankowa
Cinfting
Thank (MASK MASKI: 10.3
park railly fon.
Figure 1: Overview of Triangular Document-level Pre-training (TRIP). We select three languages for demonstration. Here, Chinese is tenseless, and Japanese and English contain past tense the linguistic clues that can resolve cross- losss-lingual ambiguities. We present three Grafting cases, representing three different language combinations. For each trilingual pair, two languages serve the input with the remaining one the reference translation, We define a novel symbol ( that denotes noise function that combines operations in sequence: splitting by half and concatenation (Grafting), and sentence permutation and span corruption. Zn Jn, and E, for n = {1, 3, 4} denotes four sentences written in three different languages. Z., Jn. and E,, denotes corrupted sentences.
et al., 2014), TRIP optimizes the likelihood:
Bilingral Decument Pairs
Tiilingul Ducument Pair
Document: (:5T-
Σ axgpagagcc log Po(33) x 1 x2)]. (1)
aww.coma ana.con/b77langzz
Docomnell: S:05 URL: ana belknn
Documeet:
URI aea combein
Docvment A URL: com bllangiip
where we define ① as a novel operation that takes two documents in different languages the input and takes three operations in sequence: splitting by half, concatenating, and sentence permutation."
URL: ana.com blangzzh
Dacurmant Hi. URL: amn.com brez
Docnmert: J.f2 lit UKL: um com belaag
Creating Trilingual Document Pairs As there is no public corpus with trilingual document pairs, TRIP creates MTDD (Microsoft Trilingual Document Dataset), a high-quality trilingual paral- lel corpus with document translation pairs across 67 languages, 4,422 bilingual directions, and 99,628 trilingual combinations. The corpus is sourced from high-quality news documents scoped from an in-house website timestamped from April 2021 to July 2022. The whole procedure composed of two steps: (i) creating bilingual document pairs and (ii) creating trilingual document pairs based on the bilingual document pairs.
Figure 2: Illustration for the URL matching mechanism to create trilingual document pairs from bilingual data. In this case, we construct trilingual data by successfully matching the URL address for the Chinese document.
The resulting translation is used for pairing only and the documents are paired and thresholded with similarity scores such as tf-idf computed on their English translation (Bañón et al., 2020). To im- prove efficiency, we attempt to pair documents only if they are timestamped within a small win- dow such semantic news with the same meaning in different languages are often reported within small times- tamp window in high probabilities. The resulting document pairs are further thresholded and filtered with LASER (Artetxe and Schwenk, 2018), which is a multilingual sentence representation.
week. The motivation is that the
To obtain bilingual document pairs, we follow ParaCrawl (Bañón al., 2020) translate all the documents we have into English using a light- weighted word-based machine translation model.
*As a pre-training method, TRIP robust does re- quire the sentences in trilingual document pairs to be perfectly aligned in their orderings. Filtering the non- non-perfect pairs can throw away the data deteriorate the performance gains. www..bing.com
Given the bilingual data constructed as above,

---

we follow previous works (Bañón al., 2020; EI- Kishky et al., 2020). These previous works lever- age URL addresses for constructing bilingual data, contrast, we use URL addresses to construct trilingual data pairs by matching and linking. Fig- ure 2 depicts detailed illustration.
Medel
Fr-FEn
Dn
Vi-+Fn
Co-
Th-
Aug
devel Models
M2M-100
50.18
42.24
20.62
34,92
57.84
27.25
36,51
MRART
48,69
44,00
28.30
57.18
50.47
Bascline Model
50.69
47.07
30.35
39,59
43.05
52,30
40.51
Documeet lesel MP Modelss
For space reasons, we present statistics to illus- trate the scale of MTDD in Table 10 in Appendix D. We also note that existing MTmCC4 (Lee et al.,
M2M-100
49.45
43.83
20.63
35,91
29.04
25,93
36,79
miAAKT
89.16
64,86
29.00
37.08
39.61
MARGET
28.40
DOCwTTS
2022) used by DOCmT5 can be less favourable for our experiments as (i) MTmC4 composed of synthetic data that could be of lower quality, (ii) MTmC4 is not publicly available at the time of writing, and (iii) MTmC4 can lead to potential data leakage for the test sets on TED Talks.
Baseline Mishel
49.53
45.90 96
30.17
19 78
42.33
30.62
59.65
50.74
39.67
40.31
TRIP (Ours)
$1.94
48.34
31.63
40.52
64.22
32,87
41.57
Table 2: Results for document-level MT TED Talks in the direction of (X - En). We report the d-BLEU scores for all results. t: scores are taken from the official papers for these models. the scores not reported or the language is not supported. The Baseline Model refers to model described Section 3.1. The Baseline Model represents document-level model continually pre-trained with the bilingual data in MTDD. For fair comparison, the trilingual data used by TRIP are constructed from these bilingual data. We perturbed them on sentence permutation and span corruption as the noise functions, with use of trilingual data.
3 Experiments
3.1 TRIP Pre-training
Model Configuration We use Transformer ar- chitecture that is composed of 24 Transformer en- coder layers and 12 interleaved decoder layers. In addition, it has an embedding size of 1024, and a dropout rate of 0.1. The feed-forward network is configured to have size of 4096 with 16 atten- tion heads. For parameter initialization, we follow Ma et al. (2021) and Yang et al. (2021) train sentence-level MT system. The motivation is that previous studies have shown that the hybrid train- ing of sentence-level and decument-level MT can improve the performance of document-level trans- lation (Sun et al., 2022). We call it the Baseline Model in the remaining of this paper.
of 0.1 for model training. All of our pre-trainings are conducted on 16 NVIDIA V100 GPUs. We set the batch size as 512 tokens per GPU. To simulate a larger batch size, we update the model every 128 steps. For the Grafting operation ① defined for TRIP, we split the documents 50% by 50%.
3.2 Multilingual Document-level MT
Data and Pre-processing As described in Sec- tion 2, we create trilingual document-level corpus, MTDD, for TRIP pre-training with the use trilin- gual document pairs. We create a list of keywords to automatically clean and remove noisy text such claims and advertisements. We follow Ma et al. (2021) to use SentencePiece (Kudo and Richard- son, 2018) for tokenization, and we the same SentencePiece model as Yang al. (2021). Follow- ing the previous works, we prefix the inputs with a language tag that indicate the target language of the generation for both pre-training and fine-tuning,
3.2.1 TED Talks
Experimental Settings Following DOCmT5, we use the IWSLT15 Campaign for the evaluation of TED Talks. Prior systems have reported scores only 1 or 2 translation directions (Lee et al., 2022; Sun et al., 2022), and DOCmT5 supports only the translation direction into English (X -+ En), We re- port more language directions while DOCmTS only evaluates on (Zh + En). Following DOCmTS, we split all documents into a maximum of 512 tokens for all train/dev/test sets during training and infer- ence. We use the official parallel training data from IWSLT15 without any additional monolingual data, with the official 2010 dev set and 2010-2013 test set for evaluation (Lee et al., 2022). We compute d- d-BLEU (Papineni et al., 2002: Liu et al., 2020; Bao et al., 2021), a BLEU score for documents.
Training Details We use the Adam optimizer (Kingma and Ba, 2014) with 31 0.9 and /22 0.98 for our multilingual pre-training. The learning rate is set as le-5 with a warmup step of 4000. We use the label smoothing cross-entropy for our trans- lation loss and we set label smoothing with a ratio

---

Model
Fr-
De
Zn
We also found that (i) the Baseline Model clearly surpasses the Baseline Model and (ii) TRIP Cs En
Arg.
beats the prior SOTA system DOCm'T5, For space reasons, we present in Appendix A the evaluations in the (X+x) directions, which also show that TRIP effectively improves language pairs that are unseen during pre-training.
Sentence-tevel MT Modelf
M2M-100
31,58
25.65
18.47
28.17
25.97
mBART
29,93
29.31
18:33
30.15
26,93
Raseline Model
34,71
27.23
37.39
33.73
Documeat-lesel level MT Models
M2M-100
25.78
17.85
26.34
mBART
30.14
26.35
15.01
29.79
25.32
Baseline Model
36,38
34,24
25.58
36.97
33.29
clearly surpasses the Baseline Model+. This obser- vation indicates two points: (i) the bilingual data in
Biscline Model
38.47
35,20
26.74
37.26
34.42
TRIP (Ours)
39.49
35.48
27.58
38,06
35.15
MTDD used to construct the trilingual data are of high quality and (ii) the trilingual objective with the Grafting mechanism is superior to the conventional bilingual objectives for multilingual pre- trainini.
Table 3: Results for document-level MT on the News benchmark the direction of (X ++ En).
We use SacreBLEU for evaluation.? ?
3.2.2 News
Baseline Systems We report strong baselines evaluated at both sentence and document levels, in- cluding SOTA models DOCmT5t (Lee et al., 2022), M2M-100 (Fan et al., 2022), mBART (Liu et al.. 2020), HANf (Miculicich et al., 201 8), MARGEI (Lewis et al., 2020a), and the Baseline Model that we to initialize the weights for TRIP. the scores are taken from existing papers. We also compare the Baseline Model+, document- level model pre-trained continually on the Baseline Model with the bilingual data used to construct the trilingual data in MTDD. We do not compare to PARADISE (Reid and Artetxe, 2022), pre-trained model that uses dictionary denoising on monolin- gual data, as its weights are not publicly available far. During our trials, we found that monolin- gual dictionary denoising can degrade document- level systems. We think that could better serve sentence- levell tasks such as sentence-level MT and cross- cross-lingual classification as conducted in its origi- nal paper. See Appendix C for the number of model parameters.
Experimental Settings For evaluation on the News benchmark, we follow Sun et al. (2022) to use News Commentary vll as the training set. For Cs and De, newstest2015 as the dev set, and wwwssest22166newsssstt
the test set respec- tively. For Fr, we use newstest2013 as the dev set and newstest2015 as the test set. For Zh, we use newstest2019 as the dev set and newstest2020 as the test set. We use the same dataset preprocessing and evaluation metric as for the TED Talks.
Baseline Systems As the weights for DOCmT5 are not available at the time of writing, we com- pare our system to various strong baselines such as M2M-100, mBART, the Baseline Model, and the Baseline Mode The scores are obtained by fine-tuning the official checkpoints.
Results Table 3 shows obvious and consistent improvements by up 3.11 d-BLEU points (from 36.38 to 39.49) with TRIP for (Fr - En) compared to the Baseline Model.
3.2.3 Europarl
Results Table 2 presents the evaluation results for TED Talks in the directions of (X En). TRIP clearly surpasses the baselines. TRIP surpasses the Baseline Model when both are fine-tuned at the document level by an average of 1.87 points in d-BLEU. TRIP surpasses the Baseline Model fine- tuned at the sentence level by an average of 1.01 points in d-BLEU. We postulate that the Baseline Model fine-tuned the document level is no better than that of the sentence level due to the reason of the long input problem (Kochn and Knowles, 2017), and also due to the reason that the Baseline Model itself is pre-trained at the sentence level. TRIP
Experimental Settings For the Europarl dataset (Koehn, 2005), we follow Sun et al. (2022) to use Europarl-v7, and we experiment with the setting of (X + En) where we test nine languages: Da De. EI, Es, Fr, It, NI, Pt, and Sv. Like previous works 2022), the dataset is (Bao et al., 2021; Sun train/dev/test divisions. randomly partitioned into
Additionally, we split by English document IDs avoid information leakage.
Baseline Systems As the weights for DOCmT5 are not available the time of writing, we com- pare our system to various strong baselines such as M2M-100, mBART, the Baseline Model, and

---

Model
Da-+En
De-FEn
Es-
Fr-=En
It-FEn
NI-EE
Pt-FEn
Sv-+En
Sentence-level MT Models
M2M-100
50.40
47.38
52.28
52.03
48.26
49.70
46.78
49.84
52.34
mBART
48.28
49.16
50.83
47.48
Baseline Model
48.94
47.25
53.46
50.57
47.68
49.49
45.95
50.65
52.77
Document-levwe MI Models
M2M-100
50.33
47.00
52.24
52.14
48.13
49.71
46.65
40.68
52.28
mBART
47.70
48.98
50.62
46.96
Baseline Model
49.85
47.64
53.34
51.32
48.46
50.26
47.12
50.13
52.42
Baseline Model
49.90
47.75
53.75
$1.78
48.70
50.37
47.18
50,49
52,49
TRIP (Ours)
51.13
48.30
54.38
52.29
49.36
51.23
48.07
51.03
53,43
Table Results for document-level machine translation on Europarl in the direction of (X + En).
Case Study Table 5 presents three case studies that demonstrate and compare the outputs between TRIP and the baseline systems. We highlight the correct translation in aqua and the wrong transla- tion in hot pink. In addition to the comparison to the Baseline Models, we also present the out- puts from popular commercial translation systems Google Translate, Microsoft Translator, and DeepL Translate. Each case demonstrates that TRIP is the best in terms of three characteristies respectively: (i) tense consistency (Jiang al., 2022; Sun al., 2022) across the sentences, (ii) noun-related issues (Jiang al., 2022) such as singular and plural con- sistency as well as attaching definite article 'the' to a previously mentioned object 'light', and (iii) conjunction presence that indicates the relation- ship between sentences and makes the translation natural and fluent (Xiong et al., 2019; Sun et al., 2022). While some translations in the third case acceptable, missing coordinating conjunction does not precisely capture the relationship between sentences and can make the translation less fluent.
BlonDe Scores the TED dataset for X>>En Directions
TAIP
Baseine ModNl
Language Directars
Th PE.
2h-5hh
BlonDe Scores
Figure 3: BlonDe scores on the TED Talks evaluated with TRIP and the Baseline Model (Document-level),
the Baseline Model+, The scores are obtained by fine-tuning the official checkpoints.
Results By comparing TRIP strong baselines, we see that the improvements with TRIP are con- sistent in all directions, and surpass the strong baselines. This validates TRIP's effectiveness.
3.2.4 Coherence and Consistency Evaluation
TRIP the best one among the systems at re- solving cross-lingual ambiguities. This observa- tion highlights the necessity of translating with document-level contexts for resolving cross-lingual ambiguities. The observations also align with the BlonDe measurements reported above.
BlonDe Evaluation Figure depicts the evalu- ations on TED Talks with BlonDe scores (Jiang et al., 2022), an evaluation metric designed for document-level MT which considers coherence and consistency issues that require the model to resolve loossllnngual ambiguities. Consistent im- provements can be observed in all the directions on TED Talks with TRIP, meaning that TRIP gen- erates more coherent and consistent translations than the baseline does. As discussed in Section 2, we postulate that these improvements attribute to the Grafting mechanism that resolves cross-lingual ambiguities by exploiting useful linguistic clues in trilingual data. This improves translation in coher- ence and consistency as reflected in the BlonDe scores. We demonstrate case studies for more anal- ysis coherence and consistency issues.
3.2.5 Large Language Models
Table 7 compares TRIP to popular ChatGPT (GPT- 3.5- TSRBOO on TED Talks. We use a prompt: "Translate the following text into English:". The re- sults indicate ChatGPT still lags behind supervised system TRIP document-level MT. This conclu- sion aligns with the previous study on sentence- level MT (Zhu et al., 2023), and we postulate that the reason ChatGPT fails handling contexts perfectly for document-level MT.
"https//chaa.pppaaiccc/chhhe

---

Come F. Reuse Consunency Ulang di, a022, Sun et al. 2023)
N:222 RE
Referency
shetnact
point
puina
could the TROTSIL"
the TEDTalk out very simple senms.....
Google Translaee
Rui noughly shoract disorssion when some Dittovio said. "aall, nuyhe we waich the TED " TTh TrisTbik joni ii
semple
Microooft Translator
But MS roughly aboract discrasion when, at octtaun ineevvals. Ocsato sai "Puul, maybe we can watch Tabiaai TEEITIS explains
simple way.
Deepl. Transdale
heceally absrntt thacsssion, and
were brosks.
mayhe can watch TROTIIL" TEDTTak
simple way illustrne....
Baseline Model (Sentence-evel)
Rui Rt's Lind of abstraci ilsensciin, and wome point, Disowin "Panl, muyhe we con woich the TRDTTak"" And the TRDTT
semply
that.....
Baseline Model (Document-levell
But R3 soet of at aboract discusson, And at some point Octavoo said "Pud, maghe we can watch the TiDiaals "The Hallii explained
a wrr smmlle way...
TRIP
But WiR soit abstiset discusinn, and it some point in the intemission Detioso saidl "Paul, maybi we can watch the TEDDIIL" And
Nowt-rolaidd Toune Ching 2002
Source
PA AENE SEEERRAA
Reference
the light washos
cuntinass timalo, vo
doos'' dark. Why that? Boznss
setually
bouncing around
Gongle Translate
the light passed over the tomitoes It shiiniing It didn't darker. Why? Because the somistnes are ris and light seflected insid:
the tomaliass....
Microooft Tranelator
the light walls over the lemalies korps shining, Ii didn't chekem, Why? Beccass the tomoscees and rinE and is reflected inside the Somstines..
Despl. Translate
the light inseled
shining diess'' dim. Why?
light reflecting incide
tomatoes....
Basellie Model (Sentence- level)
as the light goes over the lomain, It's always glowing. It's not darkening. Why? Because the tomato rips, and sahi reflcced inside the comato.....
Baveling Model (Decument-levdl)
the light possrs ovr The lnmonns loops fashinn dueen' gei carker Why" Becaae the tomaiiess bne mpt .and light rellected
asside the somatin...
TRIP
the light passes over the 5oumino It's fashhng all the time. It's not getting dafker. Why? Because the tomate sip, and the light reflected inside the
Cow Coyenriinn Presenee (Kinng 20W6 Sun 3023 H:-bill UI ut &
Source
I EERRR I 1E i AM 25RHEE 'E th &. 22 it & ARdtee
Reference
I wand suggest you that I've boen talking lot thout the fathers And I wrant think about the possitiities what democracy might
look like. migh phe have looked like, if we had mume insulved the mothen....
Gougle Trandalee
I want remind morryons
I'me tallad
sbost my prodenssurs, Falaa wore about what demuenac would look
already What the possiiiliiis look like if we could get more mothers involcad,
Mierosoft Translater
you thut have puscibilines
shout my prollnsssoess Ewould like
democracy wnuld lnok
already
can
Deepl. Transsale
I want remind you that I've talked about oo things that have come before, Ealeo want to consider the possiilitt of what democracy
would look like, what already looks like if we coukd get muthen insolvedin,
Bavelng Model (Sentence-level)
...I want remind you the I'm ialled sbust lut of my penteemamms Fallo vani in thini abust what demoumacy might look
deuoocracy might lock like if we could get more mothers involved.... ...
Baseline Model (Decument-lecc)
Fil like remind for I've talked could pet more mochers pantoogaee... ...
things
Edl like
shout possibilities what demourscy might
might
TRIP
...I wan remind you that I've takedd a lot about the past. And want to think about the possiblitees what demoerccy might. Joos like,
already looks Like if we can get more mothers involved....
Table 5: Cases from TED Talks demonstrate that TRIP captures better tense consistency, noun-related issues, and conjunction presence. We highlight the correct translation in aqua (the darker one when printed in B&W), and the mistakes in hot pink (the lighter one when printed in B&W). Google Translate: https://tranllate. httss//transaategoool...oomm Microsoft Translator: https://www.bigg woagg///s////atttooo DeepL. Translate: https:/wwww.deell..ommrrassss Time- stamped on 15th June 2023, can be subject to change.
3.3 Cross-lingual A Abstractive Summarization
et al., 2022). We also employ mBART and the Baseline Models as the baselines. See Appendix C for the number of model parameters.
Experimental Settings We follow the same set- ting used by DOCmT5 (Lee et al., 2022) to evalu- ate cross-lingual abstractive summarization on the benchmark of Wikilingua (Ladhak et al., 2020). The only difference is that they put special prefix "Summarize X to Y" where X and Y are the source and target language tags for summarization like mT5. We put target language tag as the prefix. We use the Fl measure for ROUGE- --UGUE-/ 2/ROUGE-L scores (Lin, 2004) for evaluation.
Table 6 demonstrates that TRIP clearly exceeds previous SOTA systems in several dirce- tions, including up 8.9 ROUGE-L points in (Hi + En) compared to DOCmT5. Hence, we con- clude that TRIP is an effective multilingual pre- training framework for cross-lingual abstractive summarization. We postulate that the improvement is attributed to the trilingual pre-training objective overlooked by previous works such as DOCmT5.
Results
Baseline Systems We report the scores for DOCmT5 taken from Lee et al. (2022), and we use prior SOTA scores from the official GEM bench- mark (Gehrmann et al., 2021) for mT5, ByT5 (Xue
Also, we found that using bilingual data for Base- line Model+ seems less beneficial on Wikilingua for cross-lingual abstractive summarization. TRIP

---

Model
Ir-=En
Vi++Ea
Ru-En
Es-=En
Hi-+En
Fr--En
Id Id-Ea
mT3 XLF
10.0/18.3/33.3
37.61149/22
37.2/14.6/0..9
41.2/17.2/34.6
J.
ByT5t
35.9/15,822.88
22.7/12.222..2
11.4/11022.2
35, 55..9.1
MBARTT
34/4/13.028.11
1//6/
33.1/11
38.3/15:4/2.4
-/4--
-hi-
DOC-mTSf
37,7/16.7/11.4
32.4/11.9/2..0
33.6/128/28.5
16,8/15.0/3155
34.2/13./22.9
36.3/14.//0.8
35.2/13.7/29.5
Baseline Model
34/9/13.4/29.7
40.9/18.0035.0
37.3/14.9/3199
37.8/15.3152.2
Baseline Model
12.6/19.6/36.6
38.8/16,1333.
34.9/13.329.6
37.1/14.831.5
10.7/18.1/34.9
37.2/14.9/1.8
37.6/15.2/31.9
TRIP (Oues)
45.3/22.5/39.0
30.8/17.334.4
36.6/14.6/30.8
38.3/15.9/32.7
42.8/19.9/36.8
38.5/16.0/32.9
39.4/16.4/33.3
Table Results for cross-lingual abstractive summarization on Wikilingua in (X + En). We report the scores of the score is not reported. t: the scores taken from Lee al.
F-measure for ROUGE- E////OOEE///OUUSS
(2022) and the official GEM benchmark (Gehrmann et al., 2021): http://gem-bennhmarkccom com.//esrr...o/ssulsu......
Fr-
De-FEn
Vi-Eo
Cs-EEn
Th.
Ang.
tion for document-level multilingual pre-training.
CharGPT
40.47
36.76
23.31
28.26
30.29
20.94
30.01
TRIP
54.13
49.94
42.73
34.22
42,00
4.2 Document-level Cross-lingual Tasks
Table Comparison TRIP to ChatGPT on the task of document-leve machine translation TED Talks in the direction of (X + En). The results are snapshotted in May 2023 and can be subject change.
Document-level MT and cross-lingual abstractive summarization are the two document-leve cross- lingual tasks that we investigate in this paper. Document-level MT (Miculicich al., 2018; Maruf et al., 2019, 2021; Lu et al., 2022) is chal- lenging translation task, possibly due the long input problem (Pouget-Abadie et al., 2014; Koehn and Knowles, 2017) when directly modelling the long document and the necessity in understand- ing contexts (Voita et al., 2018, 2019). Therefore, many works focus on using sentence-level mod- els with a smaller contextual window to simulate document-level MT (Zheng et al., 2020; Chen al., 2020). This paper follows the challenging setting (Bao al., 2021; Lee al., 2022) that directly optimizes document-level model with longer context window that provides a richer source of context, which also a double-edged sword that could be harder due to the long input problem. Abstractive summarization is a generation task that requires understanding of texts (Chopra et al., 2016; Fan et al., 2018). We focus on a cross- lingual setting where source and target are written in different languages (Ladhak et al., 2020).
clearly surpasses the Baseline Model This obser- vation indicates that the trilingual objective with the Grafting mechanism is superior to the conventional bilingual objectives for multilingual pre-training.
Case Study Table in Appendix shows three case studies that TRIP outputs better abstractive cross-lingual summarization. For space reasons, we leave more details in Appendix B.
4 Related Work
4.1 Multilingual Pre-training
Multilingual pre-training has achieved great suc- cess. Previous works can categorized into two streams: monolingual pre-training (Conneau et al., 2020; Liu et al., 2020; Xue et al., 2021) and bilin- gual pre-training (Huang et al., 2019; Chi et al., 2021; Ouyang et al., 2021; Tang et al., 2021; Chi al., 2021; Reid and Artetxe, 2022: Lee al., 2022). Monolingual pre-training uses monolingual corpora many different languages and perturbs the inputs with sentence permutation (Liu et al., 2020) and span corruption (Xue et al., 2021) and requires the model to reconstruct the original input. Reid and Artetxe (2022) also proposes dictionary denoising monolingual data. For bilingual pre- training, Tang et al. (2021) uses clean sentence- level bilingual translation pairs on pre-trained mod- els to improve MT. Chi al. (2021) extends m'T5 with objectives such as translation span corruption. DOCmT5 (Lee et al., 2022) creates synthetic bilin- gual translation pairs and uses sentence permuta-
Conclusions
We present a novel sequence-to-sequenee multi- lingual document-level pre-training methodology called TRIP, which the first in our field to pro- pose a trilingual objective for multilingual pre- training through Grafting. We also propose novel method to construct high-quality trilingual docu- ment pairs. Experimental results indicate that TRIP achieves competitive SOTA scores on both multilin- gual document-level machine translation and cross- lingual abstractive summarization. Future works could improve TRIP to include polygonal parallel

---

translation pairs in multilingual pre-training. We plan to release the model checkpoints and a manu- ally annotated benchmark created using our created document-leve corpus MTDD to facilitate future research on multilingual document-level MT.
Mikel Artetxe and Holger Schwenk, 2018. Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond. arXiv e-prints, page 1rXiv:1812.1000
Marta Bañón, Pinzhen Chen, Barry Haddow, Kenneth Heafield, Hieu Hoang, Miquel Esplà-Gomis, Mikel L.. Foreada, Amir Kamran, Faheem Kirefu, Philipp Koehn, Sergio Ortiz Rojas, Leopoldo Pla Sempere, Gema Comfrez-Sáncheer Elsa Sarrias, Marek Strelec, Brian Thompson, William Waites, Dion Wiggins, and Jaume Zaragoza. 2020. ParaCrawl: Web-scale acqui- sition of parallel corpora. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4555-4567, Online, Association for Computational Linguisties.
Limitations
TRIP TRIP leverages high-quality document- level trilingual translation pairs for pre-training on multilingual models. It is usually harder to collect high-quality trilingual data than to collect mono- lingual data written in different languages used by conventional methods. While we can possibly relax the quality bar for the data, additional experiments should be done to verify this view.
Guangsheng Bao, Yue Zhang. Zhiyang Teng, Boxing Chen, and Weihua Luo. 2021. G-transformer for document-level machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 3442-3455, Online. Association for Computational Linguisties.
MTDD
We create MTDD, a corpus that is com- posed of trilingual document pairs. It could be further extended to include polygonal parallel doc- ument pairs to provide a stronger signal for multi- lingual pre-training. We leave this to future works.
Mauro Cettolo, Jan Niehues, Sebastian Stüker, Luisa Bentivogli, Roldano Cattoni, and Marcello Federico. 2015. The IWSLT 2015 evaluation campaign. In Pro- ceedings of the 12th International Workshop on Spo- ken Language Translation: Evaluation Campaign. pages 2-14, Da Nang, Vietnam.
Large Language Models Large language mod- els (LLMs) such as ChatGPT have shown good translation abilities et al., 2023), while they still lag behind supervised systems (Jiao et al., 2023; Zhu et al., 2023). We conduct a limited comparison to them, as they are much larger in their number of parameters than the systems described in this work.
Junxuan Chen, Xiang Li, Jiarui Zhang, Chulun Zhou, Jianwei Cui, Bin Wang, and Jinsong Su. 2020. Mod- eling discourse structure for document-level neural machine translation. In Proceedings of the First Workshop on Automatic Simultaneous Translation, pages 30-36, Seattle, Washington. Association for Computational Linguistics.
Ethics Statement
We honour and support the EMNLP Code of Ethics. The datasets used in this work are well-known and widely used, and the dataset pre- prepprocessing does not use any external textual resource. We also cu- rate corpus for pre-training language models. Al- though we have made our best efforts in reducing potentially offensive and toxic data, the models are subject to generating offensive context. But the is- sues mentioned above are widely known to exist for these models commonly. Any content generated does not reflect the view of the authors.
Zewen Chi, Li Dong, Shuming Ma. Shaohan Huang, Saksham Singhal, Xian-Ling Mao, Heyan Huang. Xia Song, and Furu Wei. 2021. mT6: Multilingual pretrained text-to-text transformer with translation pairs. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. pages 1671-1683, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Zewen Chi, Li Dong, Shuming Ma. Shaohan Huang Xian-Ling Mao, Heyan Huang, and Furn Wei. 2021. MT6: Multilingual Pretrained Text-to- Text Trans- former with Translation Pairs. arXiv e-prints, page arXiv:2104.08692.
References
Zewen Chi, Li Dong. Furu Wei, Nan Yang. Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, and Ming Zhou. 2021. InfoXLM: An information theoretic eerormation-thhoretiic
Mikel Artetxe, Sebastian Ruder, Dani Yogatama, Gorka Labaka, and Eneko Agirre. 2020. A call for more Pro-
framework for cross-lingual language model pre- greitrain In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 3576-3588, On- line, Association for Computational Linguistics.
rigor in unsupervised cross- lingull learning. ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 7375- 7388, Online. Association for Computational Lin- guistics.

---

Sumit Chopra, Michael Auli, and Alexander M. Rush. 2016. Abstractive sentence summarization with at- tentive recurrent neural networks. In Proceedings of the 2016 Conference of the North American Chap- ter of the Association for Computational Linguisties: Human Language Technologies, pages 93-98, San Diego, California. Association for Computational Linguistics.
its evaluation and metries. In Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021), pages 96-120, Online. Association for Computational Linguistics.
Haoyang Huang, Yaoho Liang, Nan Duan, Ming Gong, Linjun Shou, Daxin Jiang, and Ming Zhou. 2019. Unicoder: universal language encoder by pre- training with multiple cross-lingual tasks. In Pro- ceedings of the 2019 Conference Empirical Meth- ods in Natural Language Processing and the 9th In- ternational Joint Conference on Natural Language Processing (EMNLP-IJCNLP) pages 1485-2494 Hong Kong, China, Association for Computational Linguistics.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Out, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics. pages 8440- 8451, Online. Association for Computational Lin- guistics.
Yuchen Jiang, Tianyu Liu, Shuming Ma. Dongdong Zhang, Jian Yang. Haoyang Huang, Rico Sennrich, Ryan Cotterell, Mrinmaya Sachan, and Ming Zhou. 2022. BlonDe: An automatic evaluation metric for document-level machine translation. In Proceedings of the 2022 Conference of the North American Chap- of the Association for Computational Linguistics: Human Language Technologies, pages 1550-1565, Seattle, United States. Association for Computational Linguistics.
Ahmed El-Kishky, Vishrav Chaudhary, Francisco Guzmán, and Philipp Koehn. 2020. CCAligned: massive collection of cross-lingual web-document pairs. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pages 5960-5969, Online. Association for Computational Linguistics.
Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, and Zhaopeng Tu. 2023. Is ChatGPT A Good Translator? Yes With GPT-4 As The Engine, arXiv e-prints, page arXiv:2301. 100745.
Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma. Ahmed El-Kishky, Siddharth Goyal, Man- deep Baines, Onur Celebi. Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vi- taliy Liptchinsky, Sergey Edunov, Edouard Grave, Michael Auli, and Armand Joulin. 2022. Beyond english-centric multilingual machine translation. J. Mach. Learn. Res., 22(1).
Mihir Kale, Aditya Siddhant, Noah Constant, Melvin Johnson, Rami Al-Rfou, and Linting Xue. 2021. nmT5 - Is parallel data still relevant for pre-training massively multilingual language models? arXiv e- prints, page arXiv:2106.02171.
Angela Fan, David Grangier, and Michael Auli. 2018. Controllable abstractive summarization. In Proceed- ings of the 2nd Workshop on Newral Machine Transla- tion and Generation, pages 45-54, Melbourne, Aus- tralia. Association for Computational Linguistics.
Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. International Conference on Learning Representations.
Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Proceedings of Machine Translation Summit X: Papers, pages 79-86. Phuket, Thailand.
Schastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Anuoluwapo Aremu, Antoine Bosselut, Khy- athi Raghavi Chandu, Miruna-Adriana Clinciu, Dipanjan Das, Kaustubh Dhole, Wanyu Du, Esin Durmus, Ondrej Dušek, Chris Chinenye Emezue, Varun Gangal, Cristina Garbacea, Tatsunori Hashimoto, Yufang Hou, Yacine Jernite, Harsh Jham- tani, Yangfeng Ji. Shailza Jolly, Mihir Kale, Dhruv Kumar, Faisal Ladhak, Aman Madaan, Mounica Maddela, Khyati Mahajan. Saad Mahamood, Bod- hisattwa Prasad Majumder, Pedro Henrique Martins, Angelina McMillan- Majoor, Simon Mille, Emiel van Miltenburg, Moin Nadeem, Shashi Narayan, Vitaly Nikolaev, Andre Niyongabo Rubungo. Salomey Osei, Ankur Parikh, Laura Herez-Beltraching Niranjan Ramesh Rao, Vikas Raunak, Juan Diego Rodriguez, Sashank Santhanam, João Sedoc. Thibault Sellam, Samira Shaikh, Anastasia Shimo- rina, Marco Antonio Sobrevilla Cabezudo, Hendrik Strobelt, Nishant Subramani, Wei Xu, Diyi Yang, Akhila Yerukola, and Jiawei Zhou. 2021. The GEM benchmark: Natural language generation,
Philipp Koehn and Rebecca Knowles. 2017. Six chal- lenges for neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation, pages 28-39, Vancouver. Association for Computa- tional Linguistics.
Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tok- enizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66-71, Brussels, Belgium. Association for Computational Linguistics.
Faisal Ladhak, Esin Durmus, Claire Cardie, and Kath- leen McKeown. 2020. WikiLingua: A new bench- mark dataset for cross-lingual ahstractive summariza- tion. In Findings of the Association for Computa- tional Linguisties: EMNLP 2020, pages 4034-4043 Online. Association for Computational Linguistics.

---

Chia-Hsuan Lee, Aditya Siddhant, Viresh Ratnakar, and Melvin Johnson. 2022. DOCm'T5: Document- pretraining of multilingual language models. In Find- ings the Association for Computational Linguistics: NAACL 2022. pages 425-437, Seattle, United States. Association for Computational Linguistics.
Sameen Maruf, Fahimeh Saleh, and Gholamreza Haffari. 2021. A survey on document-leve neural machine translation: Methods and evaluation. ACM Comput. Surv. 54(2).
Lesly Miculicich, Dhananjay Ram, Nikolaos Pappas, and James Henderson. 2018. Document-level neural machine translation with hierarchical attention net- works. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. pages 2947-2954, Brussels, Belgium. Association for Computational Linguisties.
Mike Lewis, Marjan Ghazvininejad, Gargi Ghosh, Ar- men Aghajanyan, Sida Wang, and Luke 7eulemoyer. 2020a. Pre-training via paraphrasing. In Proceedings of the 34th International Conference Neural In- formation Processing Systems, NIPS'20, Red Hook, NY, USA. Curran Associates Inc.
Xuan Ouyang, Shuohuan Wang, Chao Pang. Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. 2021. ERNIE-M: Enhanced multilingual representation by aligning cross-lingual semantics with monolingual corpora. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pages 27-38, Online and Punta Cana, Dominican Re- public. Association for Computational Linguistics.
Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, AbdeIrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020b. BART: Denoising sequence-to-sequeeee pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7880, Online, Association for Computa- tional Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: A method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computa- tional Linguistics, ACL '02, page 311-318, USA. Association for Computational Linguistics.
Chin- Yew Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Yinhan Liu, Jiatao Gu. Naman Goyal, Xian Li. Sergey Edunov, Marjan Ghazvininejad, Mike Lewis. and Luke Zettlemoyer. 2020. Multilingual denoising pre- training for neural machine translation. Transac- tions of the Association for Computational Linguis- fics, 8:726-742.
Jean Pouget-Abadic, Dzmitry Bahdanau, Bart van Mer- riënboer, Kyunghyun Cho, and Yoshua Bengio. 2014. Overcoming the curse of sentence length for neu- ral machine translation using automatic segmenta- tion. In Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Trans- lation, pages 78-85, Doha, Qatar. Association for Computational Linguistics.
Hongyuan Lu, Haoyang Huang, Shuming Ma, Dong- dong Zhang, Furu Wei, and Wai Lam. 2022. Multilingual Transitivity and Bidirectional Mul- tilingual Agreement for Multilingual Document- level Machine Translation. arXiv e-prints, page arXiv:2209.1 13940.
Machel Reid and Mikel Artetxe. 2022. PARADISE:
Exploiting parallel data for multilingual sequence- to-sequence pretraining. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 800-810, Seatlle, United States. Association for Computational Lin- guistics.
Hongyuan Lu, Haoyang Huang, Dongdong Zhang, Haoran Yang, Wai Lam. and Furu Wei. 2023. Chain-of Siain--f-Dictionryy Prompting Elicits Translation in Large Language Models. arXiv e-prints, page arXiv:2305.06575.
Zewei Sun, Mingxuan Wang, Hao Zhou, Chengqi Zhao, Shujian Huang. Jiajun Chen, and Lei Li. 2022. Re- thinking document-level neural machine translation. In Findings of the Association for Computational Linguistics: ACL 2022. pages 3537-3548, Dublin. Ireland. Association for Computational Linguistics.
Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, and Furu Wei. 2021. DeltaLM: Encoder-Decoder Pre-training for Lan- guage Generation and Translation by Augmenting Pretrained Multilingual Encoders. arXiv e-prints, page arXiv:2106. 13736.
Ilya Sutskever, Oriol Vinyals. and Quoe V. Le. 2014. Sequence sequence learning with neural networks. In Proceedings of the 27th International Conference on Neural Information Processing Systems- Volume 2. NIPS' 14, page 3104-31 12, Cambridge, MA, USA. MIT Press
Sameen Maruf, André T. Martins, and Gholamreza Haffari. 2019, Selective attention for context-aware neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume I (Long and Short Pupers), pages 3092-3102 Minneapolis, Min- nesota. Association for Computational Linguistics.
Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Na- man Goyal, Vishrav Chaudhary, Jiatao Gu, and An- gela Fan. 2021. Multilingual translation from de- noising pre-training. In Findings of the Association

---

for Computational Linguistics: ACL-IJCNLP 2021. pages 1450-3466, Online. Association for Computa- tional Linguistics.
Large Language Models: Empirical Results and Anal- ysis. arXiv e-prints, page arXiv:2304.06675
Elena Voita, Rico Sennrich, and Ivan Titov, 2019. When a good translation is wrong in context: Context-aware machine translation improves on deixis, ellipsis, and lexical cohesion. In Proceedings nual Meeting of the Association for Computational Linguistics, pages 1198-121 1212, Florence, Italy. Asso- ciation for Computational Linguistics.
the 57th An-
Elena Voita, Pavel Serdyukov, Rico Semnrich, and Ivan Titov. 2018. Context-aware neural machine trans- lation learns anaphora resolution. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 1264-1274, Melbourne, Australia. Association for Computational Linguistics.
Hao Xiong, Zhongjun Hc, Hua Wu, and Haifeng Wang. 2019. Modeling coherence for discourse neural machine translation. In Proceedings the Thirty- Third AAAI Conference on Artificial Intelligence and Thirty-First Innovative Applications of Artificial In- telligence Conference and Ninth AAAI Symposium on Educational Advances Artificial Intelligence, AAAI Press.
19/1AAI
Linting Xue, Aditya Barua, Noah Constant, Rami AI- Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2022. ByT5: Towards a token-free future with pre-trained byte-to-byte models. Transac- tions the Association for Computational Linguis- tics, 10:291-306.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami A1-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: massively multilingual pre- pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 483-498, On- line. Association for Computational Linguistics.
Jian Yang. Shuming Ma. Haoyang Huang, Dongdong Zhang, Li Dong. Shaohan Huang, Alexandre Muzio, Saksham Singhal, Hany Hassan, Xia Song, and Furu Wei. 2021. Multilingual machine translation systems from Microsoft for WMT21 shared task. In Proceed- ings of the Sixth Conference on Machine Translation. pages 446-45 455, Online. Association for Computa- tional Linguistics.
Zaixiang Zheng, Xiang Yue, Shujian Huang, Jiajun Chen, and Alexandra Birch. 2020. Towards making the most of context in neural machine translation. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pages 3983-3989, International Joint Conferences on Artificial Intelligence Organization. Main track.
Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Shujian Huang, Lingpeng Kong. Jiajun Chen, and Lei Li, 2023. Multilingual Machine Translation with

---

Case
Source
Ayakizaillanıi (ve bagesklarmna tahanloklarım) kuramasi için orta derecede 151k alan bir koy. Sicak bir yere (örnegin, tadyalirin yamna) ya degrudan günns uuguu koymu bu, syskkabulur zarur verebilir. Ayakkabilamm kurulucuya kuymak lavsiye edilmes çinkü kurutucu, ayakkab labunlanma yumultabilir.
Source (Grnsle-tramssate))
Pui your shoes (and your laces and imsoles) in mederatell light place to dry. Do not place it in a bot place Uor example, ridintor) direct sunlight this may damage the shoes. Purting your shoes the dryer is recommended because the dryer can warp the soles of your shoes.
Reference
dry your shoes.
Buseline Model (Document-level)
Allow your shoes (and laces) dry.
TRIP
Let your shoes (and laces) air dry.
Case 2
Sonrce
Bunan için yeşil arka Uzerindeki beyar konayma halonuna dokun. Ana Firanlarndaan hirinde olmasi genakir, 'ye dokun. Mesajlar ekraninn sol list kngesindadir. Açık hir sobbatin varsa Mesajlar mentiilie dännmek için ekramın sol üist kögesindeki butonuna dokun. "e dokun. Ekrammnn sag alt kögesindedir. Seçilen mesajiano silinir.
Source (ranslerr
To do this, lpp the while speech bubble on green buckground. It should be on one of their Home Screens. Tap It's in open tap
upper-lett corner Messages screen to return the Messages menu, Tap It's in lower right comer of your screen. The selected messages are deleted.
you
upper
of
Reference
Open your iPhone's 's messages. Tap Edit. Select each conversation you wish to delete. Tap Delete.
Baseline Model (Document-level)
Open your iPhone's Settings. Tap Messages. Top Delete Messages.
TRIP
Open Messages, Tap Messages
Tup Delete, Tap Delete confirm.
Case
Source
Bazılan için goçmiç yaçamlar gidlleeek bir yer degil, soni sen yapan soyin bir kismidır. insonligan tarihi boyunca birçok kaltür reenkarnccyom inanglanmin merkezine koymuştur. Islam Hiristiyanlik remkaamasyona inanmasa da, Hinduistler, bus Museviler ve buz Budistler buns immnr En jyisi kendini bir dine timüüyle adamnkkims (çünkü dinler çok lisitlayier olahilir) kendi yolunn keşfetmen. Kendi manevi dogrulamm kendin bul.
Source (Gongle-ranssated)
For some, past lives are not place go but part of what makes you who you are. Throughout the history of humanity. many cultures reincamaatio,, Hinduists, some Jews, and some Baddhists do. It's best to explore your own path rather than devote restrietive), Find your own spiritual truth.
reincamation
center of their heliefs. Although Islam and Christianiry do not helieve
yourself' entirely religion (because religions
Reference
Become spiritual.
Baseline Model (Decument-leeed)
Undersland that some people believe reincarmation. Find your own way.
TRIP
Explore your spiritual journey.
Table 8: Three case studies from Wikilingua (Tr - En) demonstrate that TRIP outputs better summarization.

---

7
Model
Number of Parameters
Zb
17.07/
26.78/
26.66/
16.9%
21.85/
M2M-100
418M
16.96
25.58
27.26
16.63
21.46
28. 60/
34.69/
31.33/
18.72/
30.84/
mBART
611M
27.32
34.06
31.72
18.40
30.17
MARGE
963M
Fm
31.07/
30.94/
49.58/
38.23/
22.19/
38.63/
31.33
30.74
49.22
37.91
21.92
38.50
mT5
1.23B
Fr
35.05/
DOCmT5
1.23B*
27.55
24.83
35,09
19.93
33.47
ByTS-Small
300M
Vi
26.93/
21.08/
34.37/
18.94/
26.13/
27.01
20:77
34.37
25.44
ByT5-Base Base
582M
Th
25.56/
18.24/
27.74/
27.69/
22.78/
ByTS-Large
1.23B*
24.81
17.74
26.54
27.38
21.87
De
27.91/ 27.72
24.72/
37.03/
30.70/
19.61/
mT5-XL
3.74B
Baseline Model
862M
Baseline Model
862M
Figure 4: Results TED Talks in (X-X) with our TRIP checkpoint pre- pre-trained in (X+Enn) directions only.
TRIP (Ours)
862M
The scores are written in TRIP as the former and the Baseline Model as the latter. Rows represent the source languages and columns represent
Table 9: the pre- ee-irained models used in our experiments. *: these models all use the model architecture of m and report the number of model parameters taken from the original paper of mT5 reported by Xue et al. (2021).
Comparison in the number of parameters for
target languages. We highlight in aqua when TRIP wins (darker one when printed in B&W) and in hot pink (lighter one when printed B&W W) when the Baseline Model wins.
Source
Target
Size/GB
Source
Target
Size/GB
A Unseen (X-xX) Language Pairs on MT
Es
En
3.22
Pt
Es
2.71
Ca
2.07
Uk
Ru
1.60
Es
1.47
Figure 4 reports the performance on TED Talks in the direction of (X-X) with our TRIP check- point pre-trained in (X-+En) directions with our corpus. The represents the translation source language and the column represents the translation target language. TRIP clearly improves most of these translation directions which are unseen dur- ing pre-training. This indicates that fact the TRIP can generalize the lingsal understanding abil- ity to unseen language pairs. This aligns with the fact reported in Lee et al. (2022).
Ea
De
1.25
Pt
En
1.14
Ca
Rs
1.12
Fr
Em
1,03
Ru
Lk
0.87
Pt
Fr
0.73
Table 10: A language list in ISO code for the top 12 language directions for the bilingual high-quality pre- training data illustrate the scale of size.
Table 9 presents the number of model parameters for the pre-trained models used in experiments. For the scores of ByT5 presented in Table 6, we report the maximum scores for each direc- tion among ByT5-Base, ByT5-Small, and ByT5- Large. This is due to space reasons. See https:// www.hmark.com/eereeeeee for the tailored scores.
B Case Study on Summarization
Table 8 shows that TRIP outputs better summariza- tion in (i) precisely capturing the context in Case 1, (ii) outputting consistent nouns, i.e., 'messages' in- stead of 'settings' in Case 2 and (iii) producing con- cise and accurate summarization in Case 3. This highlights that TRIP captures better cross- lingsalingual understanding than the baseline system, which ef- fectively mitigates cross-lingual ambiguities.
D MTDD Corpus Scale
Table 10 presents the top-12 English-centric bilin- gual data statistics to illustrate the scale of MTDD. The total size of the data is about 40/80 GB spectively for the bilingual and the trilingual data applied with Grafting.
Number of Model Parameters


Reviews:
[{"review_id": "Er0TBbZn7C", "reviewer": "Reviewer_WyK4", "summary": "", "questions": "There are also pro-drops for Zh, esp. in social media style languages, as shown in Figure-1.  \n\nBesides Gender and Tense, can the authors do more analysis on pro-drops for pronouns AND SVO(zh/en) vs VSO(ja) in translations error analysis?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n\nThank you for spending time assessing the paper and writing the reviews.  \n\nWe would like to provide our response to your concerns.\n\n> **Q1:** One motivation is to fix the tense and gender issues in hallucinated translations or summarizations. The languages chosen, zh, ja and en, are suitable for such study. However, the experiments were not detailed on simple qualitative error analysis on these two streams.\n\n**A1:** A showcase in Table 5 (Case 1) demonstrates that TRIP mitigates the tense issue. TRIP also clearly enhances BlonDe scores (refer to Figure 3), and BlonDe primarily focuses on assessing the coherence of tense and noun translations at the document level.\n\n> **Q2:** There are also pro-drops for Zh, esp. in social media-style languages, as shown in Figure-1.\nBesides Gender and Tense, can the authors do more analysis on pro-drops for pronouns AND SVO (zh/en) vs VSO (ja) in translation error analysis?\n\n**A2:** Thank you for the excellent suggestion. We will include an analysis of pro-drop in our future endeavours.\n\nBest Regards,\n\nAuthors", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Idea is simple and convincing with all experiments;  paper is well written, and the authors claimed they will release the checkpoint w/ code for public research baseline on MTDD; the research community will benefit from this.", "reasons_to_reject": "One motivation is to fix the tense and gender issues in hallucinated translations or summarizations. The languages chosen, zh, ja and en, are suitable for such study.   However, the experiments were not detailed on simple qualitatively error analysis on these two streams. ", "questions_to_authors": "There are also pro-drops for Zh, esp. in social media style languages, as shown in Figure-1.  \n\nBesides Gender and Tense, can the authors do more analysis on pro-drops for pronouns AND SVO(zh/en) vs VSO(ja) in translations error analysis?", "paper_topic_and_main_contributions": "The paper is well motivated, by mix-and-mask bilingual comparable docs at sentence level.  One motivation is to fix the tense and gender issues in hallucinated translations or summarizations. The languages chosen, zh, ja and en, are suitable for such study.   However, the experiments were not detailed on simple qualitatively error analysis on these two streams.  This maybe the biggest key stats lacking from experiments.\n\nPaper is well written, and it'd be great to release MTDD corpus for the community; the authors also claimed they will release the checkpoint w/ code for public research baseline.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Justification_for_Ethical_Concerns": "NO ethical concerns"}, "extra_text": {}}, {"review_id": "tWW5LtLmfR", "reviewer": "Reviewer_4LSK", "summary": "", "questions": "* As one main contribution, the description of MTDD is vague. Section D is not enough to illustrate the statistics for MTDD. Readers would expect a table showing the number of sentences/documents/tokens for each triplet language rather than only giving some vauge numbers for top-12 language pairs. For example, some language pairs might only have a handful of examples, which are not significant at all.\n* The experimental settings are pretty unclear with respect to initialization, pretraining and finetuning. The lack of these details hinders the readers from fully understanding the effectiveness of the proposed trilingual pertaining.\n  - What's the Baseline Model? How did you train it? What hyperparameters and training data did you use for it? \n  - How did you train Baseline Model +? Did you apply translation objective or denoising training on MTDD?\n  - Did you perform finetuning on each test task? How did you perform the finetuning? Did you also apply finetuning to similar finetuning to the baselines, such as Baseline Model+, mBart, and m2m-100?\n  - Is your TRIP model a multilingual model? How did you mix different languages? Why would you prefer Yang et al. (2021)'s sentencepiece model for your setup as you have 67 languages, many of which shouldn't be covered?\n  - Many translation tasks are essentially bilingual tasks. In the results, TRIP almost always outperforms Baseline Model+. Could you explain why the trilingual objective would help a bilingual task?\n* The title shows \"accelerating ... pretraining\". I didn't find any evidence from the paper supporting the acceleration statement.\n* M2M-100 adopts many bilingual and monolingual data for the pretraining, but the authors compared it with a significantly smaller one in the paper. I believe the comparison is unfair. What about the 1B M2M-100 model?\n* Scaling model size has been shown effective for the pretraining. Would scaling still prefer the trilingual objective over the bilingual one?\n\nI'd love to increase my score if the authors could address my concerns.\n\n== Update\nThe authors' response addressed some of my concerns and I increased the score accordingly.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n\nThank you for spending time assessing the paper and writing the reviews.\n\nThere might be some miscommunication and we would like to provide our response to your concerns.\n\n>**Q1:** The experimental settings are pretty unclear with respect to initialization, pretraining and finetuning. The lack of these details hinders the readers from fully understanding the effectiveness of the proposed trilingual pertaining.\n\n**A1:** **We assure you that we have conducted the experiments fairly. More experimental settings are given below. We will add them to our camera-ready version.**\n\n> **Q2:** What's the Baseline Model? How did you train it? What hyperparameters and training data did you use for it?\n\n**A2:** The Baseline Model is trained exactly following Yang et al. (2021). (Line 208), using the exactly same hyperparameters and training data. One may refer to their paper for detailed settings.\n\n> **Q3:** How did you train Baseline Model +? Did you apply translation objectives or denoising training on MTDD?\n\n**A3:** Yes, we apply shuffling and masking on MTDD (bilingual version) to Baseline Model+. TRIP uses shuffling and masking on the trilingual version of MTDD.\n\n> **Q4:** Did you perform finetuning on each test task? How did you perform the finetuning? Did you also apply finetuning to similar finetuning to the baselines, such as Baseline Model+, mBart, and m2m-100?\n\n**A4:** We apply fine-tuning fairly on all baselines. We apply the fine-tuning following hyperparameters in lines 227-238. We do not have Grafting here, and we apply validation patience of 5 epochs.\n\n> **Q5:** Is your TRIP model a multilingual model? How did you mix different languages? Why would you prefer Yang et al. (2021)'s sentencepiece model for your setup as you have 67 languages, many of which shouldn't be covered?\n\n**A5:** TRIP is a multilingual model. One reason to use the model from Yang et al. (2021) is for the scalability of future development based on our current checkpoint. And there is a plan to cover hundreds of languages in a future version of TRIP.\n\n> **Q6:** Many translation tasks are essentially bilingual tasks. In the results, TRIP almost always outperforms Baseline Model+. Could you explain why the trilingual objective would help a bilingual task?\n\n**A6:** The trilingual objective incorporates more information into the pre-training stage. This helps in resolving coherence issues. We present in Figure 1 for our motivation and Table 5 for the showcases. \n\n> **Q7:** The title shows \"accelerating ... pretraining\". I didn't find any evidence from the paper supporting the acceleration statement.\n\n**A7:** No, we are not speeding up the model in any format (efficiency, decoding). We originally intended to say that we are \"accelerating\" the progress of the multilingual pre-training in its literature. We chose the word accelerating to make it verbally catchy. We are open to replacing it with other words like \"improving\" or \"advancing\".\n\n> **Q8:** M2M-100 adopts many bilingual and monolingual data for the pretraining, but the authors compared it with a significantly smaller one in the paper. I believe the comparison is unfair. What about the 1B M2M-100 model?\n\n**A8:** We note that M2M-100 uses different data for pre-training. To conduct a fair comparison, our original intention is to focus more on the (Baseline Model+ VS TRIP). Doing trilingual pre-training with TRIP clearly surpasses Baseline Model+, which supports our claim about the effectiveness of TRIP. Nevertheless, according to the original paper of M2M-100 (Figure 7 in their paper), the performance gap is not large between 418M and 1B (2 points of BLEU).  However, TRIP obviously surpasses M2M-100 (418M) by 5-10 points of BLEU (Table 2, 3). Here are some additional results:\n\nDocument-level M2M-100 on TEDTalks benchmark (Table 2)    \nModel | Fr→En | De→En | Zh→En | Vi→En | Cs→En | Th→En | Avg.  \n :----: |  :----: |  :----: |  :----: |  :----: |  :----: |  :----: |  :----:\nM2M (418M) | 49.43 | 43.82 | 26.63 | 35.91 | 39.04 | 25.93 | 36.79  \nM2M (1B__)  | 51.13 | 46.01 | 27.66 | 37.82 | 40.88 | 27.33 | 38.47  \nTRIP (Ours)  |  **51.94** | **48.24** | **31.63** | **40.52** | **44.22** | **32.87** | **41.57**  \n\nDocument-level M2M-100 on News benchmark (Table 3)  \nModel | Fr→En | De→En | Zh→En | Cs →En | Avg.  \n :----: |  :----: |  :----: |  :----: |  :----: |  :----:\nM2M (418M) | 32.67 | 25.78 | 17.85 | 29.06 | 26.34  \nM2M (1B__)  | 34.81 | 27.25 | 20.01 | 30.92 | 28.25  \nTRIP (Ours)  | **39.49** | **35.48** | **27.58** | **38.06** | **35.15**  \n\n> **Q9:** Scaling model size has been shown effective for the pretraining. Would scaling still prefer the trilingual objective over the bilingual one?\n\n**A9:** We plan to investigate it in our future work and we will open-resource TRIP model checkpoints with various sizes. Pre-training can be computationally expensive, and we are trying our best to analyse other model sizes as well as scaling to even larger model sizes.\n\n> **Q10:** As one main contribution, the description of MTDD is vague. Section D is not enough to illustrate the statistics for MTDD. Readers would expect a table showing the number of sentences/documents/tokens for each triplet language rather than only giving some vague numbers for top-12 language pairs. For example, some language pairs might only have a handful of examples, which are not significant at all.\n\n**A10:** To get a better insight into the dataset statistics, for example, we report in our submission that we have Es-En, about 3.22GB 250 million English tokens on MTDD. This converts to tens of millions of English sentences and millions of documents. We will release all information/statistics with a camera-ready version of this work as soon as possible. We will also release more details, e.g., more information on data sources upon publication. We did not write much in this submission to avoid a potential violation of anonymity. \n\n> I'd love to increase my score if the authors could address my concerns.\n\nWe hope our response can address your concerns and we appreciate it if you could kindly raise your scores. Thank you very much.\n\nBest Regards,\n\nAuthors", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. MTDD, a large-scale multilingual and trilingual document-level corpus.\n2. Trilingual pretraining that adopts triplet data to improve pretraining", "reasons_to_reject": "1. Details of MTDD are unclear, particularly the statistic and distribution for different language pairs.\n2. Experimental settings are unclear: not sure how the authors performed the pretraining.\n3. Comparisons might be misleading. The conclusions -- the value of MTDD and trilingual training -- will need further evidence.", "questions_to_authors": "* As one main contribution, the description of MTDD is vague. Section D is not enough to illustrate the statistics for MTDD. Readers would expect a table showing the number of sentences/documents/tokens for each triplet language rather than only giving some vauge numbers for top-12 language pairs. For example, some language pairs might only have a handful of examples, which are not significant at all.\n* The experimental settings are pretty unclear with respect to initialization, pretraining and finetuning. The lack of these details hinders the readers from fully understanding the effectiveness of the proposed trilingual pertaining.\n  - What's the Baseline Model? How did you train it? What hyperparameters and training data did you use for it? \n  - How did you train Baseline Model +? Did you apply translation objective or denoising training on MTDD?\n  - Did you perform finetuning on each test task? How did you perform the finetuning? Did you also apply finetuning to similar finetuning to the baselines, such as Baseline Model+, mBart, and m2m-100?\n  - Is your TRIP model a multilingual model? How did you mix different languages? Why would you prefer Yang et al. (2021)'s sentencepiece model for your setup as you have 67 languages, many of which shouldn't be covered?\n  - Many translation tasks are essentially bilingual tasks. In the results, TRIP almost always outperforms Baseline Model+. Could you explain why the trilingual objective would help a bilingual task?\n* The title shows \"accelerating ... pretraining\". I didn't find any evidence from the paper supporting the acceleration statement.\n* M2M-100 adopts many bilingual and monolingual data for the pretraining, but the authors compared it with a significantly smaller one in the paper. I believe the comparison is unfair. What about the 1B M2M-100 model?\n* Scaling model size has been shown effective for the pretraining. Would scaling still prefer the trilingual objective over the bilingual one?\n\nI'd love to increase my score if the authors could address my concerns.\n\n== Update\nThe authors' response addressed some of my concerns and I increased the score accordingly.", "paper_topic_and_main_contributions": "This paper proposes to adopt trilingual document-level data to improve multilingual document-level pretraining. A trilingual example is a triplet, including three inputs from three different languages which are translations of each other. To get such data, the authors collected MTDD, a large-scale multilingual document-level data covering 67 languages. Experiments on several translation tasks and cross-lingual tasks show improved performance.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dvi0ih4NJ2", "reviewer": "Reviewer_fb6A", "summary": "", "questions": "- any thoughts on why performance improvements seem much better on TEDTalks vs. other corpora? ", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n\nThank you for spending time assessing the paper and writing the reviews.\n\nThere might be some miscommunication and we would like to provide our response to your concerns.\n\n> **Q1:** there is a claim and a footnote on robustness to documents that aren't cleanly aligned at the sentence-level. I think this is an important claim, and could be backed up with more evidence (maybe I missed it, but simply presenting the result in the appendix would suffice)\n\n**A1:** Here are two points we would like to address. (i) We originally supported such a claim via our empirical observation described in the paper that TRIP can obviously improve performance on MT and Cross-lingual Summarisation. (ii) Here is another shortcut to further simplify and solve your concern. Incorporating noise has been previously shown as useful for pre-training (e.g., sentence order and masked words in BART). Following this solid statement, the alignment (sentence order & presence of the words, or, simply masked words) is also not malicious to TRIP. This is then verified by our experiments.\n\n> **Q2:** lack of analysis on how performance is heterogeneous across different language pairs. Also, the motivation of the paper was interesting in that grafting can help bridge or transfer linguistic phenomena, but there wasn't much analysis done on this.\n\n**A2:** We will add more analysis on how performance is heterogeneous across different language pairs in our succeeding work.\n\nRegarding the linguistic phenomena, instead of human evaluation, we have used an automatic evaluation (BlonDe) which is a good evaluation metric for document-level MT that considers various consistency issues. There is also a case study in Table 5 that shows TRIP effectively improves various linguistic considerations.\n\n> **Q3:** any thoughts on why performance improvements seem much better on TEDTalks vs. other corpora?\n\n**A3:** One clue could be our experimental setup. We have the Baseline Model and Baseline Model+ as our baseline, where they can have different performance standards on different datasets. This can affect the performance gain from TRIP. Another clue is that we are also motivated by the linguistic phenomena as discussed in your last comments. We postulate that such motivation can bring better benefits to the talk text style on TEDTalks than the other two datasets on MT. \n\nBest Regards,\n\nAuthors", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- dataset with trilingual document pairs\n- pretraining technique with relatively minimal requirements (i.e. can be applied on other datasets that fit the criteria relatively easily)\n- strong baselines and good to evaluate on two both xlingual summarization and MT. ", "reasons_to_reject": "- there is a claim and a footnote on robustness to documents that aren't cleanly aligned at the sentence-level. I think this is an important claim, and could be backed up with more evidence (maybe I missed it, but simply presenting the result in the appendix would suffice)\n- lack of analysis on how performance is heterogeneous across different language pairs. Also, the motivation of the paper was interesting in that grafting can help bridge or transfer linguistic phenomena, but there wasn't much analysis done on this. ", "questions_to_authors": "- any thoughts on why performance improvements seem much better on TEDTalks vs. other corpora? ", "paper_topic_and_main_contributions": "This paper proposes a novel pretraining technique that only requires document-level alignment for three (or more) languages (a dataset of which they also release). Given a three-way correspondence between documents across 3 languages, for all three language pairs their method a) combines half of a document from one language with another (\"grafting\"), b) permutes the result, c) adds span corruption. The remaining unchanged document is used as the reference. Their method is robust to noisy sentence alignments, and they show non-trivial improvements over a strong document-level baseline (which is trained on the bilingual document pair data from the dataset) in translation and cross-lingual abstractive summarization. The authors also leverage prior work (ParaCrawl) to extract bilingual document pairs from which they extract trilingual matches, and release this dataset under the name MTDD. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper received mixed reviews with respect to soundness and excitement. 

While the MTDD corpus has been praised as an interesting resource for the community, the reviewers have raised concerns about the evaluation protocol and the validity of some claims.
Decision: Accept-Findings