Understanding Translationese in Cross-Lingual Summarization
Authors: Jiaan Wang
Fandong Meng
Yunlong Liang
Tingyi Zhang
Jiarong Xu
Zhixu Li
Jie Zhou
Venue: EMNLP 2023 Findings (2023)
Keywords: summarization
multi-lingual summarization
cross-lingual summarization

Abstract: Given a document in a source language, cross-lingual summarization (CLS) aims at generating a concise summary in a different target language. Unlike monolingual summarization (MS), naturally occurring source-language documents paired with target-language summaries are rare. To collect large-scale CLS data, existing datasets typically involve translation in their creation. However, the translated text is distinguished from the text originally written in that language, i.e., translationese. In this paper, we first confirm that different approaches of constructing CLS datasets will lead to different degrees of translationese. Then we systematically investigate how translationese affects CLS model evaluation and performance when it appears in source documents or target summaries. In detail, we find that (1) the translationese in documents or summaries of test sets might lead to the discrepancy between human judgment and automatic evaluation; (2) the translationese in training sets would harm model performance in real-world applications; (3) though machine-translated documents involve translationese, they are very useful for building CLS systems on low-resource languages under specific training strategies. Lastly, we give suggestions for future CLS research including dataset and model developments. We hope that our work could let researchers notice the phenomenon of translationese in CLS and take it into account in the future.

Understanding Translationese in Cross-Lingual Summarization
Jiaan Wang Fandong Meng?, Yunlong Liang", Tingyi Zhang* Jiarong Xu*, Zhixu Li and Jie Zhou
Linl
Shanghai Key Laboratory of Data Science, School of Computer Science, Fudan University, Shanghai, China
Pattern Recognition Center, WeChat AI, Tencent Inc, China
Beijing Key Lab of Traffic Data Analysis and Mining, Beijing Jiaotong University, Beijing, China
* School of Management, Fudan University, Shanghai, China
(fandongmeng, wi http:///wwooooetttttneeommc.. (jiarongxu, wwwulllffuaaneeeeee edu.cn
Abstract
Wan al., 2010; Yao al., 2015; Zhu et al., 2019; Ouyang et al., 2019; Ladhak al., 2020; Perez- Beltrachini and Lapata, 2021; Liang et al., 2022). As pointed by previous literature (Ladhak et al., 2020;
Given a document in a source language, cross- lingual summarization (CLS) aims gener- ating a concise summary in different target language. Unlike monolingual summarization (MS), naturally occurring source-language doc- uments paired with farget-language summaries are rare. To collect large-scale CLS data, ex- isting datasets typically involve translation in their creation. However, the translated text is distinguished from the text originally written in that language. i.e., translationese. In this pa- per. we first confirm that different approaches of constructing CLS datasets will lead to dif- ferent degrees of translationese. Then we sys- tematically investigate how translationese af- fects CLS model evaluation and performance when it appears in source documents or target summaries. In detail, we find that (1) the trans- lationese in documents summaries of test sets might lead to the discrepancy between hu- man judgment and automatic evaluation; (2) the translationese in training sets would harm model performance in real-world applications; (3) though -eraiinne-rraltteddd documents in- volve translationese, they are very useful for building CLS systems on low-resource lan- guages under specific training strategies. Lastly, we give suggestions for future CLS research in- cluding dataset and model developments. We hope that our work could let researchers notice the phenomenon of translationese in CLS and take it into account in the future.
and Lapata, 2021; Wang et al., 2022b), one of the key challenges lies in CLS is data scarcity. In detail, naturally occurring documents in a source language paired with the corresponding summaries in a target language are rare Srerez-Beltraching and Lapata, 2021), making difficult to collect large-scale and sigh-quality CLS datasets. For example, it is costly and labor- intensive to employ bilingual annotators to create target-language summaries for the given source- language documents (Chen et al., 2022). Generally, to alleviate data searcity while controlling costs, the source documents or the target summaries in existing large-scale CLS datasets (Zhu et al., 2019; Ladhak et al., 2020; Perez-Beltrache ata, 2021; Bai et al., 2021; Wang et al., 2022a; Feng et al., 2022) are (automatically or manually) translated from other languages rather than the text originally written in that language (Section 2.1). Distinguished from the text originally written in one language, translated text! in the same lan- guage might involve artifacts which refer to "trans- lationese" (Gellerstam, 1986). These artifacts in- clude the usage of simpler, more standardized and more explicit words and grammars (Baker et al., 1993; Scarpa, 2006) well the lexical and word order choices that are influenced by the source lan- guage (Gellerstam, 1996; Toury, 2012). It has been observed that the translationese in data can mislead model training as its special stylistic is away from native usage (Selinker, 1972; Volansky et al., 2013; Bizzoni et al., 2020; Yu et al., 2022). Nevertheless, translationese is neglected by previous CLS work, leading to unknown impacts and potential risks.
and Lap-
1 Introduction
Cross-lingual summarization (CLS) aims to gener- ate a summary in target language from a given document in different source language. Under the globalization background, this task helps people grasp the gist of foreign documents efficiently, and attracts wide research attention from the computa- tional linguistics community (Leuski et al., 2003;
Grounding the truth that current large-scale CLS
Work was done when Jiaan Wang was interning at Pat- tem Recognition Center, WeChat AI, Tencent Inc, China. Corresponding author.
1 "Translated text" is equal to "translations". and we alter- natively use these two terms in this paper.

---

datasets are typically collected via human or ma- chine translation, in this paper, we investigate the effects of translationese when the translations ap- pear in target summaries (Section 3) or source doc- uments (Section 4), respectively. We first confirm that the different translation methods (i.e., human translation or machine translation) will lead to dif- ferent degrees of translationese. In detail, for CLS datasets whose source documents (or target sum- maries) are human-translated texts, we collect their corresponding transsaadd documents (or summaries). The collected documents (or sum- maries) contain the same semantics as the original ones, but suffer from different translation meth- ods. Then, we utilize automatic metrics from var- ious aspects to quantify translationese, and show the different degrees of translationese between the original and collected documents (or summaries).
translationese in source documents and target sum- maries, respectively. (3) Based on our findings, we discuss and give suggestions for future research.
2 Background
2.1 Translations in CLS Datasets
To provide a deeper understanding of the transla- tions in CLS datasets, we comprehensively review previous datasets, and introduce the fountain of their documents and summaries, respectively.
Zhu et al. (2019) utilize machine translation (MT) service to translate the summaries of two En- glish monolingual summarization (MS) datasets (i.e., CNN//ailymail (Nallapati et al., 2016) and MSMO (Zhu al., 2018)) to Chinese. The trans- lated Chinese summarics together with the original English documents form En2ZhSum dataset. Later, Zn2EnSum (Zhu et al., 2019) and En2DeSum (Bai et al., 2021) are also constructed in this manner. The source documents of these CLS datasets are originally written in those languages (named natu- ral text), while the target summaries are automati- cally translated from other languages (named MT text). Feng et al. (2022) utilize Google MT service? to translate both the documents and summaries of an English MS dataset (SAMSum (Gliwa et al., 2019)) into other five languages. The translated data together with the original data forms MSAM- Sum dataset. Thus, MSAMSum contains six souree languages as well as six target languages, and only the English documents and summaries are natural text, while others are MT text. Since the transla- tions provided by MT services might contain flaws, the above datasets further use round-trip translation strategy ($ 2.2) to filter out low-quality samples. In addition to MT text, human-translate text (HT lext) is also adopted in current CLS datasets. Wik- iLingua (Ladhak al., 2020) collects document- summary pairs in 18 languages (including English) from 3 In this dataset, only the English documents//uummrries are natural text, while all those in other languages are translated from the corresponding English versions by WikiHow's hu- man writers (Ladhak et al., 2020). XSAMSum and XMediaSum (Wang et al., 2022a) are constructed through manually translating the summaries of SAMSum (Gliwa et al., 2019) and MediaSum (Zhu et al., 2021), respectively. Thus, the source docu- ments and target summaries are natural text and
Second, investigate how translationese af- feets CLS model evaluation and performance. To this end, we train and evaluate CLS models with the original and the collected data, respectively, and analyze the model performances via both automatic and human evaluation. We find that (1) the trans- lationese in documents or summaries of test sets might lead to the discrepancy between human judg- ment and automatic evaluation (i.e. ROUGE and BERTScore). Thus, the test sets of CLS datasets should carefully control their translationese, and avoid directly adopting machine-translated docu- ments or summaries. (2) The translationese in train- ing sets would harm model performance in real- world applications where the translationese should be avoided. For example, a CLS model trained with machine-translated documents summaries shows limited ability generate informative and fluent summaries. (3) Though it is sub-optimal to train a CLS model only using machine-trans/l translated documents as source documents, they are very use- ful for building CLS systems on low-resource lan- guages under specific training strategies. Lastly, since the translationese affects model evaluation and performance, we give suggestions for future CLS data and model developments especially on low-resource languages.
Contributions. (1) To our knowledge, first to investigate the influence of translationese on CLS. We confirm that the different translation methods in creating CLSS datasets will lead to dif- ferent degrees of translationese.
the
(2) We conduct systematic experiments to show the effects of the
hhttps:/cclou www.ccoodd.ooogoo.ooo google. con/translate phttps://www. wikihow. hwm....coo/./........o.........e

---

Sre
Text
Tig
Test
2.2 Round-Trip Translation
Datasct
Lang-
Typo
Lang-
Type
En2ZhSum
En
Natural
2h
MT
The round-trip translation (RTT) strategy used to filter out low-quality CLS samples built by MT services. In detail, for given text that needs to be translated, this strategy first translates t into the tar- get language E, and then translates the result back to the original language t' based on MT services. Next, I is considered high-quality translation if the ROUGE scores (Lin, 2004) between and t' exceed a pre-defined threshold. Accordingly, the CLS samples will be discarded the translations in them are not high-quality.
Zh2EnSum
Natural
En
MT
Ea2DeSum
En
Natural
De
MT
En
Natural
MSAMSum
MT
MT
WikiLingua
En
Natural
En
Natural
ApC
HT
AoC
HT
XSAMSum
En
Natural
De/Za
HT
XMedisSum
En
Natural
DoZh
HT
XWikis
BNHrDDeCss
Natural HT
RaRoDDeCss
The summary of existing CLS datasets. "Sre Lang." and "Trg Lang." denote the source and target languages in each dataset, respectively. "Text Type" rep- resents the fountain of the source documents target summaries ("Natural": natural text, "M7": Machine- translated text, "H7"": human-translated text). Language nomenclature is based on ISO 639-1 codes.
Table 1:
2.3 Translationese Metrics
To quantify translationese, we follow Toral (2019) and adopt automatic metrics from three aspects, i.e. simplification, normalization and interference.
Simplification. Compared with natural text, trans- lations tend to be simpler like using lower number of unique words (Farrell, 2018) content words (i.e., nouns, verbs, adjectives and adverbs) (Scarpa, 2006). The following metries are adopted:
HT text, respectively.
XWikis (Perez-Beltrachini and Lapata, 2021)
collects document-summarry pairs in 4 languages (i.e., English, French, German and Czech) from Wikipedia. Each document-summar pair is ex- tracted from Wikipedia page. To align the parallel pages (which are relevant the same topic but in different languages), Wikipedia provides Interlan-
Type-Token Ratio (TTR) is used to evaluate lexi- cal diversity (Templin, 1957) calculated by divid- ing the number of types (i.e., unique tokens) by the total number of tokens in the text.
Vocabulary Size (VS) calculates the total number of different words in the text.
guage links. When creating a new Wikipedia page, it is more convenient for Wikipedians to create by translating from its parallel pages (if has) than editing from scratch, leading to large number of HT text in XWikis.* Thus, XWikis is formed with both natural text and HT text. Note that though the translations commonly appear in XWikis, can- not distinguish which documents/summaries are natural text or HT text. This because we are not provided with the translation relations among the parallel contents. For example, some documents in XWikis might be translated from their parallel doc- uments, while others might be natural text serving origins to create their parallel documents.
Lexical Density (LD) measures the information lies in the text by calculating the ratio between the number of its content words and its total num- ber of words (Toral, 2019).
Normalization. The lexical choices in translated text tend to be normalized (Baker et al., 1993). We use entropy to measure this characteristic: Entropy of distinct n-grams (Ent-n) in the text. Entropy of content words (Ent-ew) in the text. Interference. The structure of translated text tends to be similar to its source text (Gellerstam, 1996). Syntactic Variation (SV) is calculated by the nor- malized tree edit distance (Zhang and Shasha, 1989) between the constituency parse trees of the translated text and the source t
Table summarizes the fountain of the source documents and target summaries in current datasets. We can conclude that when performing CLS from a source language to different target language, translated text (MT and HT text) is extremely com- mon in these datasets and appears more commonly in target summaries than source documents.
Part-of-Speech Variation (PSV) is computed by the normalized edit distance between the part-of- speech sequences of the translated text and the source text.
It is worth noting that, ideally, the fewer / lower- level translationese in the translations, the higher
'https://en http::///enwwiiiiii.ia... htt.e/ia...rg/iiii/iiippppeee Translation
*We remove tokens from the constituency parse trees to let the metric focus on syntax rather than lexical.

---

all the above metries will be.
Statities
XSAMSam
WhaLingua
Dinectio
Fur
Fart
Euro
3 Translationese in Target Summaries
Sum Type
MTT
HT
Scale
9929
31273
20751
5686
In this section, we investigate how translationese affects CLS evaluation and training when it appears in the target summaries. For CLS datasets whose source documents are natural text while target sum- maries are HT text, we collect another summaries (in MT text) for them via Google MT. In this man- ner, one document will pair with two summaries (containing the same semantics, but one is HT text and the other is MT text). The translationese in these two types of summaries could be quantified. Subsequently, we can use the summaries in HT text and MT text as references, respectively, to train CLS models and analyze the influence of transla- tionese on model performance.
TIR
8888
76,92
76.81
77.45
77.51
76.43
75.97
12357
11207
44912
41782
20264
18538
16707
10063
ED
47.30
46.54
57.70
57.53
54.20
54.03
57.06
57.62
4.75
4.67
501
4,95
Emi-2
4,062
1.048
5.37
5.31
5,20
5.09
5.50
5.41
Em--ww
3,99
3,96
3,95
SV
0.365
0.296
PSV
0.358
0.356
0.2877
0.152
0.404
0.283
0.274
0,172
Table 2: Translationese statistics of target summaries. "Direction" indicates the source and target languages. "Sum Type" denotes the text type of target summaries (HT or MT text). "Scale" means the number of samples.
we directly utilize their released summaries? as the translated summaries of XSAMSum.
Data Splitting, After preprocessing, WikiLingua (En->Ru, An and En=>Cs) contain 34,273, 20,751 and 5,686 samples, respectively. We split them into 30,273/2,000/2,000, 16,751/2,0000,,000 and 4,686/500/500 w.r.t training/validationnnes sets. For XSAMSum, since the summaries in MT text are provided by Feng et al. (2022), we also follow their splitting, i.e., 5307/302/320.
3.1 Experimental Setup
Datasets Selection.
First, we should choose CLS datasets with source documents in natural text and target sum- maries in HT text. Under the consideration of the diversity of languages, scales and domains, we de- cide to choose XSAMSum (En=>Zh) and WikiLin- gua (En=Ru/Ar/C.s). Summaries Collection. The original Chinese (Zh) summaries in XSAMSum, as well as the Russian (Ru), Arabic (Ar) and Czech (Cs) summaries in WikiLingua are HT text. Besides, XSAMSum and WikiLingua also provide the corresponding English summaries in natural text. Therefore, in addition to these original target summaries (in HT text), we can automatically translate the English summaries to the target languages to collect another summaries (in MT text) based on Google MT service.
Implementation Details. Following recent CLS work (Feng et al., 2022; Wang et al., 2022a), we use mBART-50 (Tang et al., 2021) as the CLS model. The implementation details of model training and testing are given in Appendix B.
3.2 Translationese Analysis
We analyze the translationese in the target sum- maries of the preprocessed datasets. As shown Table 2, the scores (measured by the metrics de- scribed in Section 2.3) in HT summaries are gener- ally higher than those in MT summaries, indicating the HT summaries contain more diverse words and meaningful semantics, and their sentence structures are less influenced by the source text (i.e., English summaries). Thus, the degree of translationese in HT summaries is less than that in MT summaries, which also verifies that different methods of col- lecting target- languag summaries might lead to different degrees of translationese.
RTT strategy (c.f. Section 2.2) is further adopted to remove the low-quality translated summaries. As a result, the number the translated summaries is less than that of original summaries. To ensure the comparability in subsequent experiments, we also discard the original summaries if the corresponding translated ones removed. Lastly, the remaining original and translated summaries together with source documents form the final data we used.
Thanks to MSAMSum (Feng et al., 2022) which has already translated the English summaries of SAMSum to Chinese via Google MT service, thus,
3.3 Translationese's Impact on Evaluation
For each dataset, we train two models with the same input documents but different target sum- marics. Specifically, one uses HT summarics as ref- erences (denoted as mBART-HHE while the other
"Since a CLS dataset might contain multiple source and target languages, we use "X= Y" to indicate the source lan- guage and target language are X and Y, respectively. Language nomenclature is based on ISO 639-1 codes.

---

uses MT summaries (denoted mBART-MT).
Ref.
HT Summaries
MT Summaries
Model
Table 3 gives the experimental results in terms of ROUGE-1/2/L (R1/R2/R-L) (Lin, 2004) and BERTScore (B-S) (Zhang et al., 2020). Note that there are two ground-truth summaries (HT and MT) in the test sets. Thus, for model performance on each dataset, we report two results using HT and MT summarics as references to evaluate CLS mod- els, respectively. It is apparent to find that when using MT summaries as references, mBART-MT performs better than mBART-HT, but when using HT summaries references, mBART-MT works worse. This is because the model would perform better when the distribution of the training data and the test data are more consistent. Though straight- forward, this finding indicates that a CLS model achieves higher automatic scores on the test set whose summaries are MT text, it does not mean that the model could perform better in real applica- tions where the translationese should be avoided.
WikiLingua (En=+Ru)
mBART--T mBART- HTT
23.9/6.6/ 20.5/68.0 24.6 8.0 21.5/11//// 68.2
28.0/98/ 69.6
23.9 6.2/6////..0 20.7 /67.8
Wikil ingua (Fo->An)
mBBART MT mBART-HH HTT
22.5/6.2/ 19.5//99.6777.
27.8 10.5 24.1 69.4
23./ 20.9//20///6.
23.0/6.//2..//77
WikiLingua (EnesC))
MBART-MT MBAKI-HT
149/3.1/12.9/555
/6.2/42//44//// /4.266666666
16.5 /4.///44//6///
15.4/38/ 13.6/649
XSAMSum (En-47h)
mBART
14.7/146/319/755
42.7 35.7/1///5.//4.
MBARR-RTT
/4./144//2/////
40.21 15.67 15.6/33.//74 34.1/74.2
Table 3: Experimental results (R1 / R2 R-L B-S) on WikiLingua and XSAMSum. "Ref." indicates the references of each test set are HT summaries or MT summaries. mBART-IrT and mBART-MT are trained with HT summaries and MT summaries, respectively.
Inform.
Fluency
Overall
mBART-M
1.53
1.77
mBART-HT
2.24
1.95
2.16
Ground Truth
2.42
2.81
2.73
To confirm the above point, we further con- duct human evaluation on the output summaries of mBART-HT and mBART-MT. Specifically, we randomly select 100 samples from the test set of XSAMSum, and employ five graduate students as evaluators score the generated summaries of mBART-HT and mBART-M', MT, and the ground-truth HT summaries in terms of informativeness, fluency and overall quality with a 3-point seale. During scoring, the evaluators are not provided with the source of each summary. More details about hu- man evaluation are given in Appendix C. Table 4 shows the result of human evaluation. The Fleiss' Kappa scores (Fleiss, 1971) of informativeness, fluency and overall are 0.46, 0.37 and 0.52, respec- tively, indicating a good inter-agreement among our evaluators. mBART-HT outperforms mBART- MT in all metrics, and thus the human judgment is in line with the automatic metrics when adopt- ing HT summaries (rather than MT summaries) as references. Based on this finding, we argue that when building CLS datasets, the translationese in target summaries of test sets should be carefully controlled.
Table 4: Human evaluation on the generated and ground- truth summaries (Inform.: Informativeness).
maries tend to have good lexical overlap with the MT references since both the translationese texts contain normalized lexical usages. However, such summarics may not satisfy people in the real scene (c.f., our human evaluation in Table 4). Thus, the translationese in target summaries during training has negative impact on CLS model performance. Furthermore, we find that mBART-HT has the following inconsistent phenomenon: the generated summaries of mBART-HT achieve a higher simi- larity with HT references than MT references on the WikiLingua (En= Ar and Cs) datasets (e.g., 24.6 vs. 23.9, 23.6 vs. 23.0 and 16.5 vs. 15.4 R1, respectively), but more similar MT ref- erences on XSAMSum (e.g.. 40.2 vs. 39.1 RI). We conjecture this inconsistent performance is caused by the trade-off between the following fac- tors: (i) mBART-HT is trained with the HT refer- ences rather than the MT references, and (ii) both the generated summaries and MT references are translationese texts containing normalized lexical usages. Factor (i) tends to steer the generated sum- maries closer to the HT references, while factor (ii) makes them closer the MT references. When the CLS model has fully learned the mapping from the source documents to the HT summaries during training, factor (i) will dominate the generated sum-
3.4 Translationese's Impact on Training
Compared with HT summaries, when using MT summaries as references to train CLS model, is easier for the model learn the mapping from the source documents to the simpler and more standard- ized summaries. In this manner, the generated sum-

---

Stansties
WikiLingua
Source
HT Documents
MT Documents
Dircction
Ar=En
Ru
Fr==En
Model
Doc Type
HT
MT
HT
MT
HT
MT
Wikil ingua (Ar-eFn)
Seale
25195
36503
600S8
mBARRTIITT
32.67 10.9 /2.88/26.// 77.0
247/1128/2877718
MAAKT-IIT
32.91 11. /7
355/121/22./711.
TTR
60.45
59.45
60.01
58.48
46.15
46.98
MRART--C.
/ 26.77/11
VS
27360
25883
79306
77230
31285
30755
50.75
49.73
51.95
50.73
44.00
43.74
miBART-TT
(Rums-En)
Ent-1
7.33
7.29
7.11
7.09
7.00
6.99
8.23
mBART IMT
32.7/112//2.0/713 153/123/29.//17
Ent-ow
6.84
6.82
6.77
6.72
6.74
6.68
mBART-iHT
335/112/278/111
MBART-CL. mBBART IT
351/1122/269//11
WikiLingua (Fr=)Ea)
Table 5: Translationese statistics of source documents. "Direction" indicates the source and target languages. "Doe Type" denotes the text type of source documents.
mBARI-MMT mBART-iHT
3/6/119/28.2/71
366/14.5/308/26
34.8 13.0//9.//...
35.0/ 13.2/29.//71
mBART-CL
35.1 12.6/288 /
mBART-FT TT
35.8/ 13.2 /3./299./725
Table 6: Experimental results (R1 /2 R-LL / B-S) WikiLingua (Ar/Ru/Fr=EEn) "Source" denotes using HT MT documents inputs evaluate CLS models. mB mBART-iHT and mBART-iMT are trained with HT and MT documents, respectively. mBART-CL and mBART- TT are trained with both HT and MT documents via the curriculum learning and tagged training strategies, respectively, mBART-CL and mBART-TT are signifi- cantly better than mBART-iHT with t-test p 0.05.
maries closer to the HT references, otherwise, the translationese in the generated summaries will lead them closer to the MT references. Therefore, the difficulty of CLS training data would lead to the inconsistent performance of mBART-HT. The veri- fication of our conjecture is given in Appendix A.
4 Translationese in Source Documents
4.2 Translationese Analysis
In this section, we explore how translationese af- fects CLS evaluation and training when appears in the source documents. For CLS datasets whose source documents are HT text while target sum- maries are natural text, we collect another docu- ments (in MT text) for them via Google MT ser- vice. In this way, one summary will correspond to two documents (containing the same semantics, but is HT text and the other is MT text). Next, we can use the documents in HT text and MT text as source documents, respectively, to train CLS models and analyze the influence of translationese.
We analyze the translationese in the preprocessed documents. Table 5 shows that most scores of HT documents higher than those of MT documents, indicating a lower degree of translationese in HT documents. Thus, the different methods to collect the source documents might also result in different degrees of translationese.
4.3 Translationese's Impact on Evaluation
For each direction the WikiLingua dataset, we train two mBART models with the same output summaries but different input documents. In de- tail, one uses HT documents as inputs (denoted as mBART-iHT), while the other uses MT documents (denoted as mBARRTiMMT).
4.1 Experimental Setup
Datasets Selection. The CLS datasets used in this section should contain source documents in HT text and target summaries in natural text. Here, choose WikiLingua (An/Ru/Fr=<En)
Table 6 lists the experimental results in terms of ROUGE-I/2/L (R-1/2/L) and BERTScore (B- S). Note that there are two types of input docu- ments (HT and MT) in the test sets. For each model, we report two results using HT and MT documents as inputs to generate summaries, re- spectively. Compared with using HT documents inputs, both mBART-iHT and mBART-iMT achieve higher automatic scores when using MT documents as inputs. For example, mBART-iHT achieves 32.7 and 33.7 R1, using HT documents and MT docu- ments as inputs WikiLingua (Ru=Enn, respec- tively. The counterparts of mBART-iMT are 32.4
Documents Collection. To collect MT documents, translate the original English documents of Wik- iLingua to Arabic (Ar), Russian (Ru) and French (Fr) via Google MT service. Similar to Section 3. RTT strategy is also adopted to control the quality. Data Splitting. After preprocessing, WikiLin- gua (Ar/Ru/Fir+EEn) contain 25.195/36.503/66.00 samples. We split them into 21,11552,,00/2,000, 30.503/3,000/33000 and 14.088/3,000/3.00 w.r.t training/validatiionness sets, respectively.

---

and 34.8 RI. In addition the above automatic evaluation, we conduct human evaluation on these four types (mBART-iHT/iMM with HT/MT docu- ments as inputs) of the generated summaries. In detail, we randomly select 100 samples from the test set of WikiLingua (Ar=Enn. Five graduate students are asked as evaluators to assess the gener- ated summaries in a similar way to Section 3.3. For evaluators, the parallel documents in their mother tongue are also displayed to facilitate evaluation. As shown in Table though using MT documents leads to better results terms of automatic metrics, human evaluators prefer the summaries generated using HT documents as inputs. Thus, automatic metrics like ROUGE and BERTScore cannot cap- ture human preferences if input documents are ma- chine translated. Besides, translationese in source documents should also be controlled in the test sets.
Inform.
Fluency
Overall
mBART-IMT (input MT doc.)
1.62
1.84
1.74
mBBRTTiMT (inpui doc.)
2.05
2.07
mBART-iHT (input MT doc.)
2.17
2.29
2.11
mBART-iHT (input HT doc.)
2.39
2.43
2.23
Ground Truth
2.78
2.84
2.88
Table 7: Human evaluation summaries (Inform.: In- formativeness, doc.: document).
rections (according to the conclusion of our human evaluation, we only use HT documents as inputs to evaluate mBART-CL and mBART-TT). Besides, mBART-TT outperforms mBART-CI, confirming the superiority of tagged training in CLSS To give deeper analysis of the usefulness of MT doe- uments, we use a part of (10%, 30%, 50% and 70%, respectively) HT documents (paired with summaries) and all MT documents (paired with summaries) to jointly train mBART-TT model. Be- sides, we use the same part of HT documents to train mBART-iHT model for comparisons. Table 8 gives the experimental results. With the help of MT documents, mBART-TT only uses 50% of HT doc- uments to achieve competitive results with mBART- iHT. Note that compared with HT documents, MT documents are much easier to obtain, thus the strat- egy is friendly to low-resource source languages.
4.4 Translationese's Impact on Training
When using HT documents as inputs, mBART-iHT outperforms mBART-iMT both automatic and human evaluation (Table 6 and Table 7). Thus, the translationese in source documents during train- ing has negative impact on CLS model perfor- mance. However, different from the translationese in summaries, the translationese in documents do not affect the training objectives. Consequently, we wonder if possible to train CLS model with both MT and HT documents and further improve the model performance. In this manner, MT docu- ments are also utilized to build CLS models, bene- fiting the research on low-resource languages. We attempt the following strategies: (1) mBART-CU heuristically adopts curriculum learning (Bengio al., 2009) strategy train a mBART model from (MT document, summary) samples to (HT docu- ment, summary) samples in each training epoch. (2) mBART-TT adopts the tagged training strat- egy (Caswell et al., 2019; Marie et al., 2020) to train mBART model. The strategy has been stud- ied in machine translation to improve the MT per- formance low-resource source languages. In de- tail, the source inputs with high-level translationese (i.e., MT documents in our scenario) are prepended with a special token [TT]. For other inputs with low-level translationese (i.e,, HT documents), they remain unchanged. Therefore, the special token explicitly tells the model these two types of inputs.
Discussion and Suggestions
Based on the above investigation and findings, we conclude this work by presenting concrete sugges- tions to both the dataset and model developments. Controlling translationese in test sets. As we discussed in Section 3.3 and Section 4.3, the trans- lationese in source documents or target summaries would lead to the inconsistency between auto- matic evaluation and human judgment. In addi- tion, one should avoid directly adopting machine- translated documents or summaries in the test sets of CLS datasets. To make the machine-translater documents or summaries suitable for evaluating model performance, some post-processing strate- gies should be conducted to reduce translationese. Prior work (Zhu et al., 2019) adopts post-editing strategy to manually correct the machine-transiae translated summaries in their test sets. Though post-editing in- creases productivity and decreases errors compared to translation from scratch (Green al., 2013). Toral (2019) finds that post-editing machine trans- lation also has special stylistic which is different from native usage, i.e., edst-ed Thus, the post-
As shown in Table 6, both mBART-CL and mBART-TT outperform mBART-iHT in all three di-

---

HT
MT
Wikil ingua FAr=Fn)
Wikil.ingua (Ru=Fnn
Wikilinngua (Fr=
(10%)
X
26.9/7.4/21.8 /68.3 68.3
27.97.6 /2.766//2..///./
29.7/9.11 /
(30%)
X
29.6/9.2/ 24.2//2442/// 69.7
29.9/9.2/ 24.5//24.///// 69.7
31.7 10.6 26.1 70.4
mBART-HIIT
(50%)
31.4/ 10.3 25.8
31.3/ 10.3/25.7/70
33.6/ (0.6/11.//77./111
(70%)
X
32.2/ 26.01 70.9
31./ 10.9 /26.5/ 70.7
33.9/ 12.2222772.277
(100%)
X
2L.9/1.6/77.//77
32.9/ 27.3 70,9
34.8/ 13.0/ 29.2/71.7
(10%)
(100%)
31.8 26.6/
32.9/ 12.911../27../711
34.4/12.3/8..7711
(30%)
(100%)
32.7/ 12.11111.26./// 71.
32.9/ 2..911..//7.2//1
14.7/12.6/28.7771.
TT
(50%)
(100%)
33.4/ 11.5/27.3/713
33.0/ 11.5 2
34.6/
(70%)
(100%)
33.8 /1.9/2..//15
33.7 / 12.0 71.
12.9/ 29.2 72.2 35.2
(100%)
(100%)
33,9/ 11.8/ 27.8 /71.6
33.8/ 12.1 28.0 71.5
35.8 13.2/ 29.7 72.5
Table 8: Experimental results (RI / R2 R-L./ B-S). "ITT" and "MT" indicate the percentages of (HT document, summary) and (MT document, summary) pairs used to train each CLS model, respectively. The bold and underline denote the best and the second scores, respectively.
editing strategy cannot reduce translationese. Fu- ture studies can explore other strategies to control translationese in CLS test sets.
ferent degrees of translationese (such as prepending special tokens), or (2) implicitly let the CLS model be aware of different degrees of translationese (e.g., designing auxiliary tasks in multi-task learning).
semi-supervised CLS datasets. Since high-quality CLS pairs difficult collect (especially for low-resource languages), it is costly ensure the quality of all samples in a large-scale CLS dataset. Future work could col- leet mix-quality CLS datasets that involve both high-quality and low-quality samples. In this man- ner, the collected datasets could encourage model development in more directions (e.g., the curricu- lum learning and tagged training strategies dis- cussed in Section 4.4) while controlling the cost. In addition, grounding the truth that monolingual summarization samples are much easier than CLS samples to collect under the same resources (Hasan et al., 2021b,a), b,a), supr-supervised datasets, which involve CLS samples and in-domain monolingual summarization samples, are also a good choice.
Building srixed-quality
Related Work
Cross-Lingual Summarization. Cross-lingual summarization (CLS) aims summarize source- language documents into a different target lan- guage. Due to data scarcity, early work typically focuses on pipeline methods (Leuski et al., 2003; Wan et al., 2010; Wan, 2011; Yao et al., 2015), i.e. translation and then summarization or sum- marization and then translation. Recently, many large-scale CLS datasets are proposed one after another. According an extensive survey on CLS (Wang al., 2022b), they can be divided into synthetic datasets and multi-lingual website datasets. Synthetic datasets (Zhu et al., 2019; Bai et al., 2021; Feng et al., 2022; Wang et al., 2022a) are constructed by translating monolingual sum- marization (MS) datasets. Multi-lingual website datasets (Ladhak et al., 2020; Perez-Beltrachini and Lapata, 2021) are collected from online sources. Based on these large-scale datasets, many researchers explore various ways to build CLS sys- tems, including multi-task learning strategies (Cao et al., 2020; Liang et al., 2022), knowledge distilla- tion methods (Nguyen and Luu, 2022; Liang et al., 2023a), resource-enhanced frameworks (Zhu et al., 2020) and pre-training techniques (Xu et al., 2020; Wang et al., 2022a, 2023b; Liang et al., 2023b). More recently, Wang al. (2023a) explore zero- shot CLS by prompting large language models. Dif- ferent from them, we are the first to investigate the influence of translationese on CLS.
Designing translationese-awarr CLS models. It is inevitable to face translationese when training CLS models. The degree of translationese might be different in the document summary of each training sample when faced with one of the fol- lowing scenarios: (I) training multi domainn CLS models based on multiple CLS datasets; (2) train- ing CLS models in low-resource languages based on mixed-quality datasets. In this situation, it is necessary to build conslationese-aware CLS mod- els. The tagged training strategy (Caswell et al., 2019) is simple solution which only considers two-granularity translationese in source documents. It is more general to model the three-granularity translationese (i.e,, natural text, HT text and MT text) documents as well as summaries. Future work could attempt to (1) explicitly model the dif-
Translationese. Translated texts are known to

---

have special features which refer to "transla- tionese" (Gellerstam, 1986). The phenomenon of translationese has been widely studied in machine translation (MT). Some researchers explore the in- fluence of translationese on MT evaluation (Lem- bersky et al., 2012; Zhang and Toral, 2019; Graham et al., 2020; Edunov et al., 2020). To control the effect of translationese on MT models, tagged train- ing (Caswell et al., 2019; Marie al., 2020) is pro- posed to explicitly tell MT models if the given data translated texts. Besides, Artetxe et al. (2020) and Yu et al. (2022) mitigate the effect of transla- tionese in cross -ongull transfer learning.
U21A20488, 62206056), Shanghai Science and Technology Innovation Action Plan (No. 22511104700), and Key Projects of Industrial Foresight and Key Core Technology Research and Development in Suzhou (SYC2022009).
References
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2020. Translation artifacts in cross-lingual transfer learning. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 7674-7684, Online. Association for Computa- tional Linguistics.
7 Conclusion
Yu Bai, Yang Gao, and Heyan Huang. 2021. Cross- lingual abstractive summarization with limited par- allel resources. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6910-6924, Online. Association for Computational Linguistics.
In this paper, we investigate the influence of transla- tionese CLS. We design systematic experiments to investigate how translationese affects CLS model evaluation and performance when translationese appears in source documents or larget summaries, Based on our findings, we also give suggestions for future dataset and model developments.
Mona Baker, Gill Francis, and Elena Tognini-Boune Bonelli. 1993. Corpus Linguistics and Translation Studies: Implications and Applications, chapter 2. John Ben- jamins Publishing Company, Netherlands.
Ethical Considerations
In this paper, we use mBART-50 (Tang et al., 2021) the CLS model in experiments. During fine- tuming, the adopted CLS samples mainly come from WikiLingua (Ladhak al., 2020), XSAM- Sum (Wang et al., 2022a) and MSAMSum (Feng et al., 2022). Some CLS samples might contain translationese and flawed translations (provided by Google Translation). Therefore, the trained models might involve the same biases and toxic behaviors exhibited by these datasets and Google Translation.
Yoshua Bengio. Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009. Curriculum learning. In Proceedings of the 26th Annual International Con- ference on Machine Learning, ICML 2009, Montreal, Quebec, Canada, June 14-18, 2009, volume 382 of ACM International Conference Proceeding Series, pages 41-48. ACM.
Yuri Bizzoni, Tom S Juzek, Cristina España-Bonee Koel Dutta Chowdhury. Josef van Genabith, and Elke Te- ich. 2020. How human is machine translationese? comparing human and machine translations of text and speech. In Proceedings of the 17th International Conference on Spoken Language Translation, pages 280-290 Online. Association for Computational Lin- guistics.
Limitations
While we show the influence of translationese on CLS, there are some limitations worth considering future work: (1) We do not analyze the effects of translationese when the translations appear in both source documents and target summaries; (2) Our experiments cover English, Chinese, Russian, Arabic, Czech and French, and future work could extend our method to more languages and give more comprehensive analyses w.r.t different lan- guage families.
Chris Callison- Callison-Burch, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2007. (meta-) evaluation of machine translation. In Proceedings of the Second Workshop on Statistical Machine Transla- tion, pages 136-158, Prague, Czech Republic. Asso- ciation for Computational Linguistics.
Yue Cao, Hui Liu, and Xiaojun Wan. 2020. Jointly learning to align and summarize for neural cross- lingual summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6220-6233 Online. Association for Computational Linguistics.
Acknowledgements
Isaac Caswell, Ciprian Chelba, and David Grangier. 2019. Tagged back translaannl...on In Proceedings of the Fourth Conference on Machine Translation (Volume
This work is supported by the National Natural Science Foundation of China (No.62072323,

---

1: Research Papers). pages 53-63, Florence, Italy, Association for Computational Linguistics.
Spence Green, Jeffrre Heer, and Christopher D Man- ning. 2013. The efficacy of human post-editing for language translation. In Proceedings of the SIGCHI conference on human fumanfactors in computing systems, pages 439-448.
Yulong Chen, Ming Zhong, Xuefeng Bai, Naihao Deng. Jing Li, Xianchao Zhu, and Yue Zhang. 2022. The cross-lingual conversation summarization challenge. ArXiv, abs/2205.000 00379.
Tahmid Hasan, Abhik Bhattacharjee, Wasi Uddin Ah- mad, Yuan-Fang Li, Yong-Bin Kang, and Rifat Shahriyar. 2021a. Crosssum: Beyond english- centric cross linsull abstractive text summariza- tion for 1500+ language pairs. ArXiv preprint, abs/21 18.80411..
Michael Denkowski and Alon Lavie. 2010. Choosing the right evaluation for machine translation: an ex- amination of annotator and automatic metric perfor- mance human judgment tasks. In Proceedings of the 9th Conference of the Association for Machine Translation in the Americas: Research Papers. Den- ver, Colorado, USA. Association for Machine Trans- lation in the Americas.
Tahmid Hasan, Abhik Bhattacharjee. Md. Saiful Is- lam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M. Sohel Rahman, and Rifat Shahriyar. 202 lb. XI.- sum: Large-scale multilingual abstractive summariza- tion for 44 languages. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4693-4703, Online, Association for Computa- tional Linguistics.
Sergey Edumov, Myle Ott, Marc" Aurelio Ranzato, and Michael Auli. 2020. On the evaluation of machine translation systems trained with back-translation. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 2836- 2846, Online. Association for Computational Lin- guistics.
Faisal Ladhak, Esin Durmus, Claire Cardie, and Kath- leen McKeown. 2020. WikiLingua: A new bench- mark dataset for cross- gross-lingual abstractive summariza- tion. In Findings of the Association for Computa- tional Linguistics: EMNLP 2020, pages 4034-4048, Online, Association for Computational Linguistics.
Michael Farrell. 2018. Machine translation markers post-edited machine translation output. In Pro- ceedings of the 40th Conference Translating and the Computer, pages 50-59.
Xiachong Feng, Xiaocheng Feng, and Bing Qin. 2022. MSAMSum: Towards benchmarking multi-lingual dialogue summarization. In Proceedings of the Sec- ond DialDoc Workshop on Document-groun grounded Dia- logue and Conversational Question Answering, pages 1-12, Dublin, Ireland. Association for Computational Linguistics.
Gennadi Lembersky, Noam Ordan, and Shuly Wintner. 2012. Adapting translation models to translationese improves SMT. In Proceedings of the 13th Confer- ence of the European Chapter of the Association for Computational Linguistics, pages 255-265, Avignon, France. Association for Computational Linguistics.
Anton Leuski, Chin-Yeu Lin, Liang Zhou, Ulrich Ger- mann, Franz Josef Och, and Eduard H. Hovy. 2003, Cross-lingual st English access to hindi in- formation. ACM Trans. Asian Lang, Inf. Process., 2:245-269.
Joseph L Fleiss. 1971. Measuring nominal scale agree- ment among many raters. Psychological bulletin, 76(5):378.
Martin Gellerstam. 1986. Translationese in swedish novels translated from english. In Lars Wollin and Hans Lindquist, editors, Translation Studies in Scan- dinavia, page 88-95. CWK Gleerup.
Qian Li, Shu Guo, Cheng Ji, Xutan Peng, Shiyao Cui, Jianxin Li, and Lihong Wang. 2023a. Dual-gated fusion with prefix-tuning for multi-modal relation extraction. Findings of the Association for Com- putational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023, pages 8982 8994, Association for Computational Linguistics.
Martin Gellerstam. 1996. Translations as source for cross-linguistic studies. Lund Studies in English, 88:53-62.
Bogdan Gliwa, Iona Mochol, Macicj Biesek, and Alek- sander Wawer. 2019. SAMSum corpus: A human- annotated dialogue datasstt for abstractive summa- rization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 70-79, Hong Kong, China. Association for Computational Linguis- tics.
Qian Li, Shu Guo, Yangyifei Luo, Cheng Ji. Li- hong Wang, Jiawei Sheng, and Jianxin Li. 2023b. Attribute-consistent knowledge graph representation learning for multi-modal entity aligmment. ceedings of the ACM Web Conference 2023, WWW 2023, Austin, TX, USA, 30 April 2023 4 May 2023, pages 2499 2508.508808 ACM.
Pro-
Yvette Graham, Barry Haddow, and Philipp Kochn. 2020. Statistical power and translationsse machine translation evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 72-81, Online. Association for Computational Linguistics.
Yunlong Liang, Fandong Meng, Jiaan Wang. Jinan Xu, Yufeng Chen, and Jie Zhou. 2023a. D2tv: Dual knowledge distillation and target-oriented vi- sion modeling for many-to-many multimodal sum- marization. arXiv preprint 235.

---

Yunlong Liang, Fandong Meng, Jinan Xu, Jiaan Wang. Yufeng Chen, and Jie Zhou. 2023b. Summary- oriented vision modeling for multimodal abstractive summarization, In Proceedings of the 61st Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 2934-2951, Toronto, Canada. Association for Computational Lin- guistics.
Larry Selinker. 1972. Interlanguage. International Re- view of Applied Linguistics in Language Teaching (IRAL., (0(1-4):209-2))
Yuqing Tang, Chau Tran. Xian Li, Peng-Jen Chen, Na- man Goyal, Vishrav Chaudhary, Jiatao Gu, and An- gela Fan. 2021. Multilingual translation from de- noising pre-training. In Findings of the Association for Computational Linguisties: ACI-IJCNLPP 2021, pages 3450-3466, Online, Association for Computa- tional Linguisties.
Yunlong Liang. Fandong Meng. Chulun Zhou, Jinan Xu, Yufeng Chen, Jinsong Su, and Jie Zhou. 2022. A vari- ational hierarchical model for neural cross- cross-lingual summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics ( (Volume I: Long Papers), pages 2088-2099, Dublin, Ireland. Association for Computational Lin- guistics.
Mildred C Templin. 1957. Certain language skills in children: Their development and interrelationships. volume 10. JSTOR.
Antonio Toral. 2019. Post-editese: an exacerbated translationese. In Proceedings of Machine Trans- lation Summit XVII: Research Track, pages 273-281, Dublin, Ireland. European Association for Machine Translation.
Yew Lin. 2004, ROUGE: package for auto- matic evaluation of summaries. In Text Summariza- fion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Benjamin Marie, Raphael Rubino, and Atsushi Fujita. 2020. Tagged tracslltanl revisited: Why does it really work? In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 5990-5999 Online. Association for Computational Linguistics.
Gideon Toury. 2012. Descriptive translation studies: And beyond. Descriptive Translation Studies, pages 1-366.
Vered Volansky, Noam Ordan, and Shuly Wintner. 2013. On the features of translationese. Digital Scholarship in the Humanities, 30(1):98-118
Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Çağlar Gulgehre, and Bing Xiang. 2016. Abstrac- tive text summarization using pouueneeeto-squenee RNNs and beyond. In Proceedings of The 20th SIGNLL Conference on Computational Natural Lan- guage Learning. pages 280-290, Berlin, Germany, Association for Computational Linguistics.
Xiaojun Wan. 2011. Using bilingual information for cross- langu-an document summarization. In Pro- ceedings of the 49th Annual Meeting of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies. pages 1555,,55,, Portland, Ore- gon, USA. Association for Computational Linguis- tics.
Thong Thanh Nguyen and Anh Tuan Luu, 2022. Im-
proving neural cross-lingual abstractive summariza- tion via employing optimal transport distance for knowledge distillation. Proceedings of the AAAI Conference on Artificial Intelligence, 36( (o(10):11 1103- 11111.
Xiaojun Wan, Huiying Li, and Jianguo Xiao. 2010. Cross-language document summarization based machine translation quality prediction. In Proceed- ings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 917-926, Uppsala, Sweden. Association for Computational Linguistics.
Jessica Ouyang, Boya Song, and Kathy McKeown. 2019. A robust abstractive system for cross-lingual summarization. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Associ- ation for Computational Linguistics: Human Lan- guage Technologies, Volume I (Long and Short Pa- pers), pages 1025-2031, Minneapolis, Minnesota. Association for Computational Linguistics.
Jiaan Wang, Yunlong Liang. Fandong Meng, Beiqi Zou, Zhixu Li, Jianfeng Qu, and Jie Zhou. 2023a. Zero- shot cross-lingual summarization via large language models.
Jiaan Wang. Fandong Meng, Ziyao Lu, Duo Zheng, Zhixu Li, Jianfeng Qu, and Jie Zhou. 2022a. Clid- Sum: A benchmark dataset for cross- cons-lingual dialogue summarization. In Proceedings of the 2022 Confer- ence on Empirical Methods in Natural Language Pro- cessing, pages 7716-7729, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Laura Perez-Beltrachini and Mirella Lapata. 2021. Mod- els and datasets for cross-ling longull summarisation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9408-9423, Online and Punta Cana, Dominican Re- public. Association for Computational Linguistics.
Federica Scarpa. 2006. Corpus-basee quality assess- ment of specialist translation: A study using parallel and comparable corpora in english and italian. In Insights into specialized translation lingussisss in- sights.
Jiaan Wang, Fandong Meng, Duo Zheng, Yunlong Liang, Zhixu Li, Jianfeng Qu, and Jie Zhou. 2022b. A Survey on Cross-Lingur Summarization. Transac- tions of the Association for Computational Linguis- tics, 10: 1304-1333

---

Jiaan Wang, Fandong Meng, Duo Zheng, Yunlong Liang, Zhixu Li. Jianfeng Qu, and Jie Zhou. 2023b. Towards unifying multi-lingual and cross-lingual summarization. In Proceedings of the 61st Annual Meeting of the Association for Computational Lin- guisticy (Volume 1: Long Papers), pages 15127- 15143, Toronto, Canada. Association for Computa- tional Linguistics.
Datasct
Scale
Coverage
Wikil Linuua (En==Ru)
34,273
57.58
WikiLingua (En=>Ar
20,751
58.68
Wikil.ingu (sn=+Cs)
5,686
57.54
XSAMSum
5,929
37,99
Table 9: The scales and coverage of CLS data.
Ruochen Xu, Chenguang Zhu, Yu Shi, Michael Zeng. and Xuedong Huang. 2020. Mixed-lingual pre- training for cross-lingual summarization. In Pro-
Ref.
HT summaries
MT summaries
Test
Hard Suhset
/6.9/4.9/ 15.6
17.6/6.6/ 16.6
ceedings of the 1st Conference of the Pac-fai Chapter of the Association for Computational Lin- guistics and the 10th International Joint Conference on Natural Language Processing, pages 536-541, Suzhou, China. Association for Computational Lin- guistics.
Simple Subset
16.5 14.5
15.2/3.5/33. 13.4
Table 10: Results of mBART-HT (R1 / R2 / R-L) on the hard and simple test subsets of WikiLLingur (Ein Cs).
Jin-ge Yao, Xiaojun Wan, and Jianguo Xiao. 2015. Phrase-hased compressive cross-language summa- rization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 118-127, Lisbon, Portugal. Association for Computational Linguistics.
4154-4164, Brussels, Belgium. Association for Com- putational Linguistics.
Junnan Zhu, Qian Wang. Yining Wang, Yu Zhou, Ji- ajun Zhang, Shaonan Wang. and Chengqing Zong. 2019. NCLS: Neural cross-lingual summarization. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing DEMNLP-IJCNLA pages 3054- 3064, Hong Kong, China. Association for Computa- tional Linguisties.
Sicheng Yu, Qianru Sun, Hao Zhang, and Jing Jiang. 2022. consslaee-trinn embracing translationese arti- facts. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Vol- ume 2: Short Papers), pages 362-370, Dublin, Ire- land. Association for Computational Linguistics.
Junnan Zhu, Yu Zhou, Jiajun Zhang, and Chengqing Zong. 2020. Attend, translate and summarize: An efficient method for neural trogu-lingual summariza- tion. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1309 1321, Online. Association for Computational Linguistics.
Kaizhong Zhang and Dennis Shasha. 1989. Simple fast algorithms for the editing distance between trees and related problems. SIAM Journal on Computing. 18(6):1255 1262.
Mike Zhang and Antonio Toral. 2019. The effect of translationese in machine translation test sets. In
Proceedings of the Fourth Conference on Machine Translation (Volume I: Research Papers), pages 73- 81, Florence, Italy, Association for Computational Linguistics.
A The Inconsistent Performance of mBART-HT
According to our conjecture, XSAMSum (En+>Zh) should be more difficult than WikiLingua RU/-////ArC(s)) for CLS models to perform. Con- sequently, factor (i) dominates in WikiLingua, while factor (ii) dominates in XSAMSum, lead- ing to the inconsistent performance. To convince that, we illustrate the difficulty of each CLS dataset from the following aspects: (1) Scale calculates the number of CLS samples in each dataset. Generally, the more samples used to train CLS model, the easier it is for the model to learn CLS. (2) Cover- age measures the overlap rate between documents and summaries. which is defined the average pro- portion of the copied bigram in summaries for each dataset. 8 The higher coverage of a dataset, the less
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinherger, and Yoav Artzi. 2020. Bertscore: Evalu- ating text generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020. Addis Ababa, Ethiopia, April 26-30, 2020. OpenRe- view.net.
Chenguang Zhu, Yang Liu, Jie Mei, and Michael Zeng. 2021. MediaSum: A large-scale media interview dataset for dialogue summarization. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 5927-5934, Online. Association for Computational Linguistics.
Junnan Zhu, Haoran Li, Tianshang Liu, Yu Zhou, Ji- ajun Zhang, and Chengqing Zong. 2018. MSMO: Multimodal summarization with multimodal output. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages
"Since the documents and summaries in CL.S datasets are in different languages, the coverage is calculated based on

---

search space for models learn the mapping from source documents target summaries. As shown in Table 9, XSAMSum more difficult than other datasets based on the overall consideration from both aspects. To provide a deeper explanation of our conjecture, we further evenly split the test set of WikiLingua (En- Cs) into hard and simple sub- sets according to the coverage of each sample. The coverage of samples in the hard subset is less that in the simple subset. As shown in Table 10, the gen- erated summaries of mBART are closer to HT references on the simple subset, but more similar to MT references on the hard subset, demonstrating that the difficulty of CLS training data would lead to the inconsistent performance of mBART-HT
judgments, all summaries of given document are provided for our evaluators simultancously let them make comparisons among different models (note that all evaluators do not know every sum- mary is generated by which model, and the appear- ance order of summaries is shuffled). We do not provide detailed breakdown for each score in each metric due to the following reasons: (1) We encour- age each evaluator follow their actual feelings make judgments since everyone in the real applica- tions might have different criteria (for each metric). For example, somcone might sensitive to flu- ency while others might for informativeness. Thus, we want to make our human evaluation more in line with this real-world scenario. (2) It is hard and even unrealistic to construct perfect quantitative human evaluation principle. (3) This evaluation method is commonly used in machine translation evaluation (Callison-Burl
B Implementation Details
The implementation of mBART-50 (Tang et al., 2021) (610M parameters) used in experiments is provided by the Huggingface Transformers." 9 We fine-tune the model on NVIDIA Tesla V100 GPUs (32G) and set the learning rate to 5e-6, the warmup steps to 500, the epochs to 10, and the batch size is 4. The maximum number of tokens for input sequences is 1024. In the test process, beam size is set to 5. All experimental results listed in this paper are the average of 3 runs
al., 2007; Denkowski and Lavie, 2010), cross- loosslinggull summarization eval- uation (Zhu et al., 2020; Cao et al., 2020; Liang et al., 2022) and other tasks (Li et al., 2023b,a).
To calculate ROUGE scores, employ the multilingual ROUGE toolkit that considers seg- mentation and stemming algorithms for various languages. To calculate BERTScore, we use the bertscore !!
C Human Evaluation
We tell our evaluators a brief guideline about three metrics: (1) Informativeness measures how infor- mative the summary is. (2) Fluency measures how fluent, and grammatical the summary is. Is sum- mary well-written and grammatically correct? (3) Overall measures the overall quality of each gen- erated summary. It can be judged under the con- sideration of informativeness, fluency, relevance, consistency and so on. Then, all evaluators are re- quired to give each summary score selected from "I", "2" and "3" for each metric. When making the
the source-language summaries (i.e, English summaries in Wikil.ingua and XSAMSum) and source documents.
ctttcc/boo.ccc//////// momme------mmy---tt--------m /ttttpp://githubb... wttp://ssbbuttpplll/uumm//////// /ree/
5e-llarge-5e-mnyy ton-tto--mann many-mmt
"hhttps://gittee http:///..cmm/iiigge/eeeetteeeee


Reviews:
[{"review_id": "f6KqcDTVzC", "reviewer": "Reviewer_pBBk", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for your comments concerning our manuscript entitled “Understanding Translationese in Cross-Lingual Summarization”. We thank you for the very helpful suggestions and we provide below with our answers to your comments. We have followed closely the suggestions, and made clarifications and revisions accordingly. We hope the newly provided content could help to further strengthen our work.\n\n**Comment 1**: In section 3.3, the human evaluation part isn't clear.  \n**Comment 1.1**: Were the evaluators native speakers? What was their expertise level?  \n**Response to comment 1.1**: We employ five **Chinese** graduate students who are fluent in **English** as our evaluators to conduct human evaluation on both Section 3 (Table 4) and Section 4 (Table 7). (1) The human evaluation in Section 3 is conducted with English documents and Chinese summaries (i.e., XSAMSum En->Zh), thus, our evaluators can deal with both involved languages. (2) The human evaluation in Section 4 is conducted with Arabic documents and English summaries (i.e., Wikilingua Ar->En). To facilitate the evaluation on Arabic documents, the Chinese parallel documents are also shown to our evaluators (as we mentioned in **Lines 480-481**). Note that, the Chinese parallel documents are provided by the original WikiLingua dataset and contain the same semantics as the Arabic documents. In this way, our evaluators can focus on the generated English summaries and give their judgments about the summary quality.  \n\n**Comment 1.2**: What guidelines were given for human evaluation? The specifics of the 3-point scale breakdown for metrics like Informativeness, Fluency, and Overall are missing.   \n**Response to comment 1.2**: Actually, we do not make the 3-point scale breakdown, and we only tell our evaluators a brief guideline about three metrics: (1) Informativeness measures how informative the summary is. (2) Fluency measures how fluent, and grammatical the summary is. Is a summary well-written and grammatically correct? (3) Overall measures the overall quality of each generated summary. It can be judged under the consideration of informativeness, fluency, relevance, consistency and so on.  \nThen, all evaluators are required to give each summary a score selected from “1”, “2” and “3” for each metric. When making the judgments, all summaries of a given document are provided for our evaluators simultaneously to let them to make comparsions among different models (note that all evaluators do not know every summary is generated by which model, and the appearance order of summaries is shuffled).  \nWe do not provide detailed guidelines/breakdown for each score in each metric due to the following reasons: (1) We encourage each evaluator to follow their actual feelings to make judgments since everyone in the real applications might have different criteria (for each metric). For example, someone might be sensitive to fluency while others might for informativeness. Thus, we want to make our human evaluation more in line with this real-world scenario.  (2) It is hard and even unrealistic to construct a perfect quantitative human evaluation principle.  (3) This evaluation method is commonly used in machine translation evaluation [1,2], and cross-lingual summarization evaluation [3-5].   \n\n[1] (Meta-) Evaluation of Machine Translation (WMT 2007)  \n[2] Choosing the Right Evaluation for Machine Translation: an Examination of Annotator and Automatic Metric Performance on Human Judgment Tasks  (AMTA 2010)  \n[3] Attend, Translate and Summarize: An Efficient Method for Neural Cross-Lingual Summarization (ACL 2020)  \n[4] Jointly Learning to Align and Summarize for Neural Cross-Lingual Summarization (ACL 2020)  \n[5] A Variational Hierarchical Model for Neural Cross-Lingual Summarization (ACL 2022)\n\n**Comment 1.3**: On line 363, what does \"overall quality\" mean?  \n**Response to comment 1.3**: Please refer to response to comment 1.2   \n\n**Comment 1.4**: The authors mentioned selected 100 samples from the XSAMSUM test set randomly. How many of these belonged to mBART-HT and mBART-MT? However, Table 4 indicates separate human evaluation for these subsets. If the human evaluation was done separately, I would expect additional Fleiss' kappa scores for Informativeness, Fluency, and Overall.  \n**Response to comment 1.4**: (1) Each sample from the test set means an input document and the corresponding ground truth summary. We randomly select 100 samples from the test set, and use mBART-HT and mBART-MT to generate their 100 summaries, respectively. Thus, we have 100 summaries generated by mBART-HT and other 100 summaries generated by mBART-MT. Besides, there are also 100 ground truth summaries. (2) The Fleiss' kappa scores of informativeness, fluency, and overall on all 300 summaries are 0.46, 0.37 and 0.52 (as we mentioned in **Lines 367-368**).  The additional Fleiss' kappa scores of each subset are listed in the following table:   \n\nFleiss' Kappa |Informativeness|Fluency|Overall\n-|-|-|-\n100 summaries generated by mBART-MT|0.41|0.32|0.48\n100 summaries generated by mBART-HT|0.41|0.33|0.45\n100 ground truth summaries|0.48|0.36|0.62\nall 300 summaries|0.46|0.37|0.52  \n\n**Comment 1.5**: On line 369, mentioned that \"good agreement between evaluators.\" But according to Fleiss' kappa score interpretation, the scores show fair to moderate agreement. It's important to explain the reasons for the lower agreement scores.  \n**Response to comment 1.5**: We think the Fleiss' kappa scores we achieved (0.46, 0.37 and 0.52 in Section 3) are decent compared with previous cross-lingual summarization work. For example, the Fleiss' kappa scores in [6] are 0.20, 0.22 and 0.37 (c.f., Table 4 in [6]), and the scores in [7] are 0.26, 0.37 and 0.43 (c.f., Section 6.3 in [7]). Therefore, please do not worry about the degree of evaluators' agreement.  We also additionally provide the percentage agreements of human evaluation to ensure the soundness of our human evaluation, please refer to **the response to reviewer WQsK's comment 6.1** if you are interested.   \n\n[6] Cross-Lingual Abstractive Summarization with Limited Parallel Resources (ACL 2021)  \n[7] ClidSum: A Benchmark Dataset for Cross-Lingual Dialogue Summarization (EMNLP 2022)\n\n**Comment 1.6**: Opting for human evaluation on two different datasets would be preferable.  \n**Response to comment 1.6**: Thanks for this constructive suggestion! Due to the space limitation, we did human evaluation on XSAMSum (En->Zh) in Section 3, and on WikiLingua (Ar->En) in Section 4. We think current results have already verified our claims and showed the influence of translationese in cross-lingual summarization. We sincerely hope the current settings can be accepted, and we also would like to take this suggestion into account in the future.  \n\n**Comment 2**: The above raised issue apply to section 4.3 also, where Fleiss' kappa scores and metric ratings are missing.  \n**Response to comment 2**: Thank you for pointing out this! The human evaluation metrics used in Section 4 are the same as those in Section 3. The Fleiss' kappa scores of human study in Section 4 are listed in the following table:  \n\nFleiss' Kappa |Informativeness|Fluency|Overall\n-|-|-|-\n100 summaries generated by mBART-iMT (input MT docs)|0.48|0.40|0.44\n100 summaries generated by mBART-iMT (input HT docs)|0.51|0.52|0.60\n100 summaries generated by mBART-iHT (input MT docs)|0.50|0.48|0.54\n100 summaries generated by mBART-iHT (input HT docs)|0.45|0.41|0.44\n100 ground truth summaries|0.55|0.50|0.60\nall 500 summaries|0.56|0.52|0.58  \n\n**Comment 3**: As stated in Appendix B, all reported experimental scores are averages of 3 runs. However, the corresponding standard deviation scores are absent.  \n**Response to comment 3**: To ensure the soundness of this study, we conduct each experiment three times and report the average results. When we conduct the experiments multiple times, we find the results in terms of ROUGE scores and BERTScore are almost the same, and the standard deviation scores are typically less than 0.1. Therefore, we do not report the standard deviation scores. Besides, reporting standard deviation scores is not a common practice in the cross-lingual summarization research field.\n\n**Comment 4**: Regarding lines 022-023, there's no evidence or empirical study provided on how translationese might negatively affect CLS model performance in real-world applications.  \n**Response to comment 4**: We are sorry for this misunderstanding. The “real-world applications” here means “how people really feel about the model-generated cross-lingual summaries when using CLS systems”. We want to use “real-world applications” to emphasize our study goes beyond automatic evaluations which may not match what humans really feel. To this end, we use human evaluation in both Section 3 and Section 4 to show the influence of translationese in CLS systems.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper is the first attempt to analyze the impact of translationese in cross-lingual summarization tasks.\n2. The authors provided a detailed analysis of the impact of translationese concerning source documents and Target summaries for various languages and datasets.\n3. The authors highlight findings, such as the potential promise in constructing mixed-quality or semi-supervised cross-lingual summarization datasets, which could be a valuable direction for further research.\n4. This paper emphasizes how important it is to use human-translated documents and summaries to create strong cross-lingual summarization models.\n5. The analysis process used to identify the impact of translationese allows other groups to reproduce for different language families.", "reasons_to_reject": "1. In section 3.3, the human evaluation part isn't clear.\n  \n      a. Were the evaluators native speakers? What was their expertise level?\n\n      b. What guidelines were given for human evaluation? The specifics of the 3-point scale breakdown for metrics like Informativeness, \n          Fluency, and Overall are missing.                                                     \n\n     c.  On line 363, what does \"overall quality\" mean?\n \n     d. The authors mentioned selected 100 samples from the XSAMSUM test set randomly. How many of these belonged to mBART-HT \n           and mBART-MT? However, Table 4 indicates separate human evaluation for these subsets.\n\n     e. If human evaluation was done separately, I would expect additional Fleiss' kappa scores for Informativeness, Fluency, and Overall.\n\n     f. On line 369, mentioned that  \"good agreement between evaluators.\" But according to Fleiss' kappa score interpretation \n           (https://en.wikipedia.org/wiki/Fleiss%27_kappa), the scores show fair to moderate agreement. It's important to explain the reasons \n          for the lower agreement scores.                                                                                                                                    \n\n\n      g. Opting for human evaluation on two different datasets would be preferable.\n\n2. The above raised issue apply to section 4.3 also, where Fleiss' kappa scores and metric ratings are missing.\n\n3. As stated in Appendix B, all reported experimental scores are averages of 3 runs. However, the corresponding standard deviation scores are absent.\n\n4. Regarding lines 022-023, there's no evidence or empirical study provided on how translationese might negatively affect CLS model performance in real-world applications.", "questions_to_authors": "", "paper_topic_and_main_contributions": "1. This paper explores the impact of translationese on the evaluation and performance of cross-lingual summarization models.\n2. The paper outlines the process of creating different cross-lingual summarization datasets and highlights potential disparities between automated and human evaluations.\n3. A comprehensive overview of the development of current cross-lingual summarization datasets is provided.\n4. The paper also delves into the factors within translated text that contribute to the phenomenon known as 'translationese'.\n5. Experiments were conducted to measure the influence of translationese on both documents and summaries, and the resulting analysis is presented in a clear and easily understandable manner.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Minor issue:\n1. line183 --> check the spacing"}, "extra_text": {}}, {"review_id": "sp4old3SET", "reviewer": "Reviewer_pmRA", "summary": "", "questions": "1. where are details of the wide range of linguistic analysis to compute translationese metrics (parse, pos, entropy, etc).  how do factor varying levels of quality for such analysis in different languages?\n\n2. section 3.3 is quite confusing:  here the direction is en2x and one would expect the summaries to be in the x languages and annotation is conducted in those 4-5 languages?  also any explanation for the fair/weak agreement for fluency and also the fact that overal agreement is higher than the individual factors?\n\n3. Re: Table 8, the MT data is always kept at 100% in the ablation study.  do you think using a cleaner subset of the MT data (using your translationese metrics) can make the impact of that data stronger?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for your comments concerning our manuscript entitled “Understanding Translationese in Cross-Lingual Summarization”. We thank you for the very helpful suggestions and we provide below with our answers to your comments. We have followed closely the suggestions, and made clarifications and revisions accordingly. We hope the newly provided content could help to further strengthen our work.\n\n**Comment 1**: Paper has a poor writing and it is difficult to follow. Moreover, it is not organized well (long sentences, latex mistakes, many acronyms, long repetitions of table results in the text body). Even some basic concepts like translationese are not defined well. There's not a single example in the entire paper.\n\n**Comment 1.1**: Paper has a poor writing and it is difficult to follow. Moreover it is not organized well (long sentences, latex mistakes, many acronyms, long repetitions of table results in the text body).  \n**Comment 1.2**: Typos Grammar Style And Presentation Improvements: (1) latex errors: line 057 (long referencing at the middle of sentence); (2) Table 9 & 10 in page 8.  \n**Response to comments 1.1 & 1.2**: Please kindly let us know if something in our manuscript is difficult to follow, and we would appreciate it. (1) For latex mistakes in line 57, we promise to revise the long referencing in the middle. (2) For acronyms, we explain the full meaning of each acronym the first time it occurs, and we hope this could be accepted. (3) As for long repetitions of table results in the text body. Since our work is an analytical study, we have tried our best to analyze the phenomenon in experiments, and it is inevitable to repeat some results. (4) About Tables 9 & 10 in page 8: both Table 9 and table 10 belong to Appendix, and they are in page 11 currently.\n\n**Comment 1.3**: Even some basic concepts like translationese are not defined well.  \n**Response to comment 1.3**: We explain translationese in Lines 64-78, and the definition follows the main-stream definition like the first paragraph in [1]. We hope this definition can be accepted.  \n\n[1] Translationese as a Language in “Multilingual” (ACL 2020)\n\n**Comment 1.4**: There's not a single example in the entire paper  \n**Response to comment 1.4**: Currently, we do not provide any cases of translationese since (1) it is recognized that the translated or model-generated text might involve translationese. (2) space limitation. If this work could be accepted, we promise to add a case in additional page of content.\n\n[2] Post-editese: an Exacerbated Translationese (MTSummit 2019)\n\n**Comment 2**: poor replicability of the work. so much details are missing in computing various translationese metrics for different languages and cls models.  \n**Response to comment 2**: Actually, our translationese metrics typically follow previous translationese work [2] (as mentioned in **line 221**). We also would like to share the full version of the calculating scripts once publication. Here, we provide the key implementation details of these metrics as follows:  \n(1)Type-Token Ratio (TTR) and vocabulary size (VS): To calculate TTR and VS, we need to tokenize the input sentence, here we use mBART50's tokenizer with the following Python code:  \n```python\nfrom transformers import MBart50TokenizerFast\ntokenizer = MBart50TokenizerFast.from_pretrained(\"facebook/mbart-large-50\")\n\ninput_sentence = \"...\"\ntokenized_sentence = tokenizer.tokenize(input_sentence)\n```\n(2) Lexical Density (LD):  To calculate LD, we need the POS tags of input sentences in different languages. We use [stanza](https://stanfordnlp.github.io/stanza/) to obtain them ([stanza supports many languages](https://stanfordnlp.github.io/stanza/available_models.html)) via the following Python code:  \n```python\nimport stanza\nlang2stanzacode = {\n    'arabic': 'ar',\n    'chinese': 'zh',\n    'russian': 'ru',\n    'czech': 'cs',\n    'french': 'fr',\n} # these languages are used in our experiments\n\nnlp = stanza.Pipeline(lang2stanzacode[lang], processors='tokenize,pos', use_gpu=True)\n\ncontent_words = 0 # number of content words\ntotal_words = 0 # number of all words\n\ninput_sentence = \"...\"\nstanza_doc = nlp(input_sentence)\nfor sent in stanza_doc.sentences:\n    for word in sent.words:\n        if word.pos in ['ADJ','ADV','VERB','NOUN']:\n            content_words += 1\n        total_words += 1\n        \nlexical_density = content_words / float(total_words)\n```\n(3) Entropy of distinct n-grams (Ent-n) and Entropy of content words (Ent-cw): These two metrics are also calculated based on tokens (using mBART50's tokenizer)  \n(4) Syntactic Variation (SV): The constituency parse trees are calculated via Stanza:  \n```python\nimport stanza\n\nnlp = stanza.Pipeline(\"zh\", processors='tokenize,pos,constituency', use_gpu=True) # calculating constituency parse trees for Chinese\n\nChinese_sentence = \"...\"\nstanza_doc = nlp(Chinese_sentence)\n\nconstituency_parse_tree = zh_sum_xsamsum.sentences[0].constituency\n```\nPlease also note that the stanza's constituency algorithm [does not support Arabic, Russian, Czech and French](https://stanfordnlp.github.io/stanza/constituency.html), thus we only calculate SV for Chinese text (see **Table 2** in our paper)  \n(5) Part-of-Speech Variation (PSV): The POS tags used to calculate PSV are obtained via the same method as LD.\n\n**Comment 3**: Where are details of the wide range of linguistic analysis to compute translationese metrics (parse, pos, entropy, etc). how do factor varying levels of quality for such analysis in different languages?    \n**Comment 3.1**: Where are details of the wide range of linguistic analysis to compute translationese metrics (parse, pos, entropy, etc).   \n**Response to comment 3.1**: Our translationese metrics typically follow previous translationese work [2] (as mentioned in **line 221**). The effectiveness of these metrics to measure translationese has already been verified in previous work. Therefore, we do not give details of linguistic analysis for them, and please do not worry about these metrics.\n\n[2] Post-editese: an Exacerbated Translationese (MTSummit 2019)\n\n**Comment 3.2**: how do factor varying levels of quality for such analysis in different languages?  \n**Response to comment 3.2**: We think the translationese evaluation is another interesting research topic, but it goes beyond this work. In this work, we use existing metrics to analyze the influence of translationese in cross-lingual summarization systems, rather than propose new translationese metrics. We sincerely hope the current analysis could be accepted.\n\n**Comment 4**: Section 3.3 is quite confusing: here the direction is en2x and one would expect the summaries to be in the x languages and annotation is conducted in those 4-5 languages? also any explanation for the fair/weak agreement for fluency and also the fact that overal agreement is higher than the individual factors?  \n**Comment 4.1**: here the direction is en2x and one would expect the summaries to be in the x languages and annotation is conducted in those 4-5 languages?  \n**Response to comment 4.1**: Yes, for a given English document, there are annotated summaries in multiple languages.\n\n**Comment 4.2**: Any explanation for the fair/weak agreement for fluency  \n**Response to comment 4.2**: We think the Fleiss' kappa scores we achieved (0.46, 0.37 and 0.52 in Section 3) are decent compared with previous cross-lingual summarization work. For example, the Fleiss' kappa scores in [3] are 0.20, 0.22 and 0.37 (c.f., Table 4 in [3]), and the scores in [4] are 0.26, 0.37 and 0.43 (c.f., Section 6.3 in [4]). Therefore, please do not worry about the degree of evlautors' agreement. We also additionally provide the percentage agreements of human evaluation to ensure the soundness of our human evaluation, please refer to **the response to reviewer WQsK's comment 6.1** if you are interested.\n\n[3] Cross-Lingual Abstractive Summarization with Limited Parallel Resources (ACL 2021)  \n[4] ClidSum: A Benchmark Dataset for Cross-Lingual Dialogue Summarization (EMNLP 2022)\n\n**Comment 4.3**: Any explanation for the fact that overal agreement is higher than the individual factors.  \n**Response to comment 4.3**: We do not provide detailed guidelines for each score in each metric (i.e., informativess, fluency or overall), and the scores are annotated via comparing outputs among different models. Therefore, the results of different metrics are not comparable. In addition, if you are interested in the process of our human evaluation, please refer to **the response to reviewer pBBk's comment 1.2**.\n\n**Comment 5**: Table 8, the MT data is always kept at 100% in the ablation study. do you think using a cleaner subset of the MT data (using your translationese metrics) can make the impact of that data stronger?  \n**Response to comment 5**: These translationese metrics are used to measure text distributions rather than a single sentence or document. Therefore, using them to select a cleaner MT subset might have potential risks and need to be further verified, which is beyond the scope of this work. If we can select a cleaner MT subset, we think there might be a trade-off between quality and quantity, thus it needs experiments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Interesting topic (cross-language summarization).\n\n2. Fairly comprehensive look at the impact of translationese in source, target, eval and models.  The comprehensive work just requires some substantial polishing and restructuring.", "reasons_to_reject": "1. Paper has a poor writing and it is difficult to follow.  Moreover it is not organized well (long sentences, latex mistakes, many accronyms, long repetitions of table results in the text body). Even some basic concepts like translationese are not defined well.  There's not a single example in the entire paper.\n\n2. poor replicability of the work.  so much details are missing in computing various translationese metrics for different languages and cls models.\n\n3. The conclusion and macro level impact of the work does not seem significant.", "questions_to_authors": "1. where are details of the wide range of linguistic analysis to compute translationese metrics (parse, pos, entropy, etc).  how do factor varying levels of quality for such analysis in different languages?\n\n2. section 3.3 is quite confusing:  here the direction is en2x and one would expect the summaries to be in the x languages and annotation is conducted in those 4-5 languages?  also any explanation for the fair/weak agreement for fluency and also the fact that overal agreement is higher than the individual factors?\n\n3. Re: Table 8, the MT data is always kept at 100% in the ablation study.  do you think using a cleaner subset of the MT data (using your translationese metrics) can make the impact of that data stronger?", "paper_topic_and_main_contributions": "Paper is a comprehensive look at the impact of translationese in cross-lingual summarization.  authors motivate the work based on the shortage of cross-lingual summarization resources.  After introducing some metrics to estimate translationese, they look at their impact when present in the source text, target summaries and also their impact on summarization models.  Results indicate their presence and motivates some suggestions about containing their presence with automatic and manual efforts.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1.  latex errors: line 057 (long referencing at the middle of sentence), \n2. Table 9 & 10 in page 8."}, "extra_text": {}}, {"review_id": "aFZZzDoQta", "reviewer": "Reviewer_WQsK", "summary": "", "questions": "Question A: How were the human evaluators selected? Were they compensated for this task? \n\nQuestion B: What is the 3-point scale? What instructions were given to the annotators?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for your comments concerning our manuscript entitled “Understanding Translationese in Cross-Lingual Summarization”. We thank you for the very helpful suggestions and we provide below with our answers to your comments. We have followed closely the suggestions, and made clarifications and revisions accordingly. We hope the newly provided content could help to further strengthen our work.\n\n**Comment 1**: The authors also include a partial human evaluation, though details are missing.   \n**Response to comment 1**: Thank you for pointing it out!  We tell our evaluators a brief guideline about the three human evaluation metrics we used: (1) Informativeness measures how informative the summary is. (2) Fluency measures how fluent, and grammatical the summary is. Is a summary well-written and grammatically correct? (3) Overall measures the overall quality of each generated summary. It can be judged under the consideration of informativeness, fluency, relevance, consistency and so on.   \nThen, all evaluators are required to give each summary a score selected from “1”, “2” and “3” for each metric. When making the judgments, all summaries of a given document are provided for our evaluators simultaneously to let them make comparisons among different models (note that all evaluators do not know each summary is generated by which model, and the appearance order of summaries is shuffled).\nWe will add these details of our human evaluation in the revised version.\n\n**Comment 2**: At least a distribution of human evaluation scores should be added.  \n**Response to comment 2**: We provide the distributions of our human evaluation scores in the following tables:  \n\nSystem |Metric|proportion of “1”|proportion of “2”|proportion of “3”|average score\n-|-|-|-|-|-\nmBART-mt|inform.|36.0|22.0|42.0|2.06\nmBART-ht|inform.|19.2|37.6|43.2|2.24\nGround Truth|inform.|7.0|44.0|49.0|2.42\nmBART-mt|fluency|62.6|21.8|15.6|1.53\nmBART-ht|fluency|35.2|34.6|30.2|1.95\nGround Truth|fluency|4.4|10.2|85.4|2.81\nmBART-mt|overall|48.2|26.6|25.2|1.77\nmBART-ht|overall|27.2|29.6|43.2|2.16\nGround Truth|overall|10.0|7.0|83.0|2.73  \n\n(The distributions of human evaluation scores in Section 3)   \n\nSystem |Metric|proportion of “1”|proportion of “2”|proportion of “3”|average score\n-|-|-|-|-|-\nmBART-iMT (input MT doc.)|inform.|56.0|26.0|18.0|1.62\nmBART-iMT (input HT doc.)|inform.|42.0|27.0|31.0|1.89\nmBART-iHT (input MT doc.)|inform.|28.6|25.8|45.6|2.17\nmBART-iHT (input HT doc.)|inform.|20.0|21.0|59.0|2.39\nGround Truth|inform.|8.2|5.6|86.2|2.78\nmBART-iMT (input MT doc.)|fluency|43.0|30.0|27.0|1.84\nmBART-iMT (input HT doc.)|fluency|32.4|30.2|37.4|2.05\nmBART-iHT (input MT doc.)|fluency|23.4|24.2|52.4|2.29\nmBART-iHT (input HT doc.)|fluency|19.0|19.0|62.0|2.43\nGround Truth|fluency|6.2|3.6|90.2|2.84\nmBART-iMT (input MT doc.)|overall|51.2|23.6|25.2|1.74\nmBART-iMT (input HT doc.)|overall|30.8|31.4|37.8|2.07\nmBART-iHT (input MT doc.)|overall|29.4|30.2|40.4|2.11\nmBART-iHT (input HT doc.)|overall|29.8|17.4|52.8|2.23\nGround Truth|overall|3.8|4.4|91.8|2.88  \n\n(The distributions of human evaluation scores in Section 4)   \nPlease also note that the human evaluation samples are randomly selected from the test set of CLS data, and there is no order among them.  \n\n**Comment 3**: The reported Fleiss Kappa should be employed to categorical data.   \n**Response to comment 3**: We provide the Fleiss' Kappa scores with respect to different subsets of summaries as following tables:   \n\nFleiss' Kappa |Informativeness|Fluency|Overall\n-|-|-|-\n100 summaries generated by mBART-MT|0.41|0.32|0.48\n100 summaries generated by mBART-HT|0.41|0.33|0.45\n100 ground truth summaries|0.48|0.36|0.62\nall 300 summaries|0.46|0.37|0.52   \n\n(The Fleiss' kappa scores of human study in Section 3 w.r.t different subsets of summaries)    \n\nFleiss' Kappa |Informativeness|Fluency|Overall\n-|-|-|-\n100 summaries generated by mBART-iMT (input MT docs)|0.48|0.40|0.44\n100 summaries generated by mBART-iMT (input HT docs)|0.51|0.52|0.60\n100 summaries generated by mBART-iHT (input MT docs)|0.50|0.48|0.54\n100 summaries generated by mBART-iHT (input HT docs)|0.45|0.41|0.44\n100 ground truth summaries|0.55|0.50|0.60\nall 500 summaries|0.56|0.52|0.58   \n\n(The Fleiss' kappa scores of human study in Section 4 w.r.t different subsets of summaries)   \n\n**Comment 4**: Question A: How were the human evaluators selected? Were they compensated for this task?  \n**Response to comment 4**: (1) We employ five Chinese graduate students who are fluent in English as our evaluators to conduct human evaluation. All evaluators have extensive experience in data annotation (participating many human studies in our lab's previous work), and a small number (about 20%) of their evaluation results are checked by a data expert to ensure the quality of human evaluation.  (2) During the human evaluation, the salary for annotating each summary is determined by the average time of annotation and local labor compensation standard. We would like to add this explanation in Ethical Considerations.  \n\n**Comment 5**: Question B: What is the 3-point scale? What instructions were given to the annotators?  \n**Response to comment 5**: Please refer to response to comment 1.   \n\n**Comment 6**: Apart from Fleiss Kappa, it would be also good to report the percentage agreement. If the data is not categorical, Krippendorff's alpha should be employed. Preferably CI should be reported.  \n**Comment 6.1**: It would be also good to report the percentage agreement.  \n**Response to comment 6.1**: Thank you for pointing it out! Since the human evaluation scores are annotated under the comparisons among different models, we provide the percentage agreements of model comparisons as follows:  \n\nPercentage Agreements |mBART-MT|mBART-HT|Ground Truth\n-|-|-|-\nmBART-MT|-|58%|70%\nmBART-HT|58%|-|65%\nGround Truth|70%|65%|-  \n\n(The percentage agreements of human evaluation in Section 3)  \n\nPercentage Agreements |mBART-iMT (input MT doc)|mBART-iMT (input HT doc) |mBART-iHT (input MT doc)|mBART-iHT (input HT doc)|Ground Truth\n-|-|-|-|-|-\nmBART-iMT (input MT doc)|-|53%|57%|62%|74%\nmBART-iMT (input HT doc)|53%|-|53%|61%|70%\nmBART-iHT (input MT doc)|57%|53%|-|60%|73%\nmBART-iHT (input HT doc)|62%|61%|60%|-|69%\nGround Truth|74%|70%|73%|69%|-  \n\n(The percentage agreements of human evaluation in Section 4)  \n\nEach percentage in the above tables means what percentage of the test samples achieves consistent results among all five evaluators when comparing the corresponding two models (including ground truth). The consistent results mean all five evaluators (1) give the first model a better score, (2) give the second model a better score, or (3) give the same score for both models.  \n\n**Comment 6.2**: If the data is not categorical, Krippendorff's alpha should be employed. Preferably CI should be reported.   \n**Response to comment 6.2**: Our human evaluation score is categorical. Please also refer to the response to comment 1.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The problem addressed in this study is very timely and likely to be an issue in the current systems;\n\n- The experiment choice is motivated by the RQs. ", "reasons_to_reject": "- I am not sure about the human evaluation employed, as there are no sufficient details in the paper raising ethical and methodological questions. For instance, I believe that the ratings should not be displayed as means as the scale employed is likely ordinal (or at least a distribution of scores should be added. This further raises a question about the IAA analysis, as the reported Fleiss Kappa should be employed to categorical data. In other words, the IAA is done for categorical data, the means are reported as if the data was interval/numerical, while most likely the data is ordinal, though it is impossible to say as the details are not reported (or somehow I have missed that).", "questions_to_authors": "Question A: How were the human evaluators selected? Were they compensated for this task? \n\nQuestion B: What is the 3-point scale? What instructions were given to the annotators?", "paper_topic_and_main_contributions": "In this paper the authors explore the effect of translationese on the cross-lingual summarization (CLS). Since in the cross-lingual summarization task the data is often obtained via translation (often MT) this leads to a question of whether the translationese, likely introduced during this process, affects the task itself (CLS)\n\nThe authors investigate how translationese, when present in source documents or target summaries, affects the modeling and evaluation of CLS. The author explore this problem through a series of experiments and give suggestions on how one may deal with this issue in the future. These include controlling translationese in test sets, developing translationese-aware models, and building mixed-quality CLS datasets.\n\nThe authors also include a partial human evaluation, though details are missing.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Apart from Fleiss Kappa, it would be also good to report the percentage agreement. If the data is not categorical, Krippendorff's alpha should be employed. Preferably CI should be reported.", "Justification_for_Ethical_Concerns": "Details on human evaluation are missing, it is not stated how much the evaluators were paid or whether the study design was reviewed by IRB. "}, "extra_text": {}}, {"review_id": "UsE4zEGlq9", "reviewer": "Reviewer_AJcm", "summary": "", "questions": "The authors' responses seem conditioned by the size limitations of the\ntext, both in my case and in that of the rest of the reviewers.\n\nCertainly some of the modifications required imply a significant\ninvestment of time and work, but it is no less true that if this is\nthe case, it is because it refers to aspects that should have been\nconsidered before writing the text. More precisely, we have that:\n\nA. Please, justify the authors' position in relation to the reviewer's\nitem A in the previous section \"Reasons to reject\".\n\nRegarding the authors' responses, it is not clear whether or not some type of\ncomparison with other operating models will be included in the\nreviewed paper, given that the authors limit themselves to stating\nthat:\n\n\"We would like to add this discussion in Limitation section\"\n\nwhen I have no doubt that it is an issue that must be addressed in one\nway or another in an eventual final review, if only to warn of the\npossibility of biases associated with the use of a certain type of\nmodeling, given that this would condition the entire experimental part\nof the work.\n\nB. Please, justify the authors' position in relation to the reviewer's\nitem B in the previous section \"Reasons to reject\".\n\nRegarding the authors' answers at this point, indeed most existing translationese-relevant\nstudies typically focus on MT, which would have precisely facilitated\na comparison like the one we have just commented on. In any case, we\nare talking about a decision of the authors who have adequately\njustified.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for your comments concerning our manuscript entitled “Understanding Translationese in Cross-Lingual Summarization”. We thank you for the very helpful suggestions and we provide below with our answers to your comments. We have followed closely the suggestions, and made clarifications and revisions accordingly. We hope the newly provided content could help to further strengthen our work.\n\n**Comment 1**: I wonder if the fact of considering only m-BART-based MT models for testing does not introduce some kind of bias in the results.  \n**Response to comment 1**: There is no doubt that conducting experiments on more multi-lingual pre-trained models (like mT5, BLOOM or LLaMA) would give more comprehensive analyses. Since the human evaluation is labor-intensive and time-consuming, we only conduct experiments on mBART-based model, and it might involve bias of mBART itself. We would like to add this discussion in Limitation section.\nBesides, we want to emphasize that the mBART model is a commonly used end-to-end baseline in the cross-lingual summarization research field [1-4]. This is why we choose to conduct experiments on this model. We sincerely hope the current settings could be accepted.\n\n[1] WikiLingua: A New Benchmark Dataset for Cross-Lingual Abstractive Summarization (EMNLP 2020 Findings)  \n[2] Models and Datasets for Cross-Lingual Summarisation (EMNLP 2021)  \n[3] A Variational Hierarchical Model for Neural Cross-Lingual Summarization (ACL 2022)  \n[4] Towards Unifying Multi-Lingual and Cross-Lingual Summarization (ACL 2023)\n\n**Comment 2**: I also wonder if the best way to address the impact of translationese in the field of cross-lingual technologies would not be by focusing our attention on the basic MT technologies... and not on each of their possible practical applications (summatization, ...). In this regard, it should be remembered that there are already works published in this line as, for example Machine Translationese: Effects of Algorithmic Bias on Linguistic Complexity in Machine Translation (EACL 2021).   \n**Response to comment 2**: To the best of our knowledge, most existing translationese-relevant studies typically focus on machine translation (MT) which is also a practical application. In this paper, we want to provide analytical research from another lens, i.e., cross-lingual summarization (CLS). Different from MT, CLS is a more complex scene that requires the model to perform both translation and summarization. Though translationese is already studied in MT, the current CLS research field typically neglects this phenomenon, and the traditional evaluation (ROUGE or BERTScore) might not reflect human feelings towards CLS systems in real scenes. Thus, we want to openly discuss these potential risks and they can cause problems. As for your recommended reference, we will cite it in the related work in the revised version.\n\n**Comment 3**: About Typos Grammar Style And Presentation Improvements  \n**Response to comment 3**: Thank you for pointing them out! We will sincerely consider your constructive suggestions in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The impact of translationese in the domain of NLP is a critical\nissue. Here it is important to take into account its growing\ndependence on ML techniques, a reality associated with the need for\ntraining corpus in different fields of knowledge and languages. This\nhas led to the application of TM tools to its generation and,\nconsequently, the generalization of all kinds of phenomena related to\nthe concept of \"translationese\". In turn, given the popularization of\nNLP-based tools, this could even affect language on a longer term. The\nrecent irruption of GPT-like models delves into the importance of\nstudying this type of problems, particularly in relation to\nsummarization.\n\nIn this context, the work is interesting. Regardless of the fact that\nthe authors' conclusions coincide with those intuitively expected, the\ntruth is that the evidence corroborates them and seems to dispel any\npossible doubts.", "reasons_to_reject": "A. I wonder if the fact of considering only m-BART-based MT models for testing\ndoes not introduce some kind of bias in the results.\n\nB. I also wonder if the best way to address the impact of\ntranslationese in the field of cross-lingual technologies would not be\nby focusing our attention on the basic MT technologies... and not on\neach of their possible practical applications (summatization, ...). In\nthis regard, it should be remembered that there are already works\npublished in this line as, for example:\n\nEva Vanmassenhove, Dimitar Shterionov, and Matthew\nGwilliam. 2021. Machine Translationese: Effects of Algorithmic Bias on\nLinguistic Complexity in Machine Translation. In Proceedings of the\n16th Conference of the European Chapter of the Association for\nComputational Linguistics: Main Volume, pages 2203–2213,\nOnline. Association for Computational Linguistics.\n\nSicheng Yu, Qianru Sun, Hao Zhang, and Jing\nJiang. 2022. Translate-Train Embracing Translationese Artifacts. In\nProceedings of the 60th Annual Meeting of the Association for\nComputational Linguistics (Volume 2: Short Papers), pages 362–370,\nDublin, Ireland. Association for Computational Linguistics.\n\nthe latter of which is one of the references included in this work.", "questions_to_authors": "The authors' responses seem conditioned by the size limitations of the\ntext, both in my case and in that of the rest of the reviewers.\n\nCertainly some of the modifications required imply a significant\ninvestment of time and work, but it is no less true that if this is\nthe case, it is because it refers to aspects that should have been\nconsidered before writing the text. More precisely, we have that:\n\nA. Please, justify the authors' position in relation to the reviewer's\nitem A in the previous section \"Reasons to reject\".\n\nRegarding the authors' responses, it is not clear whether or not some type of\ncomparison with other operating models will be included in the\nreviewed paper, given that the authors limit themselves to stating\nthat:\n\n\"We would like to add this discussion in Limitation section\"\n\nwhen I have no doubt that it is an issue that must be addressed in one\nway or another in an eventual final review, if only to warn of the\npossibility of biases associated with the use of a certain type of\nmodeling, given that this would condition the entire experimental part\nof the work.\n\nB. Please, justify the authors' position in relation to the reviewer's\nitem B in the previous section \"Reasons to reject\".\n\nRegarding the authors' answers at this point, indeed most existing translationese-relevant\nstudies typically focus on MT, which would have precisely facilitated\na comparison like the one we have just commented on. In any case, we\nare talking about a decision of the authors who have adequately\njustified.", "paper_topic_and_main_contributions": "In the authors' words, the aim of the work is to confirm that\ndifferent approaches of constructing cross-lingual summarization (CLS)\ndatasets will lead to different degrees of translationese. Then they\nstudy how such translationese affects CLS model evaluation and\nperformance when it appears in source documents or target summaries.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Is such a display of bibliographic references really necessary ? Their\npresence is overwhelming and makes it difficult to understand the text\n(see for example Lines 040 or 058). Such display does not at all favor\nthe reading and comprehension of the text.\n\nPlease, avoid redirects in the text, particularly in the Introduction\n(see Line 063, 083, 084). Instead, include a \"road-map\" of the paper\nat the end of Section 1, including a brief look of all the other\nSections.\n\nMost of the last paragraph (Contributions) of Section 1 retake the\nsame contents included in the previous one as well in the Abstract. Is\nthis really necessary."}}]

Author rebuttal:

Meta-review: The paper presents a study of the impact of “translationese” on Cross Lingual Summarization (CLS), a problem relevant to the CLS area. An evaluation based on human annotators is presented, including Inter Annotator Agreement values. Some interesting conclusions are drawn from the work.

Some unresolved questions remain: 
- Is Fleiss' Kappa appropriate for calculating IAA when using an ordinal scale (R3)?
- Does it make sense to study the effect of translationese in a particular application, such as summarization, or does it make more sense to study it in Machine Translation methods in general (R4)?
- It is important to better describe the annotation task that was performed (R1 and R2).
Decision: Accept-Findings