Revisiting Cross-Lingual Summarization: A Corpus-based Study and A New Benchmark with Improved Annotation
Year: 2023

Abstract:
Most existing cross-lingual summarization (CLS) work constructs CLS corpora by simply and directly translating pre-annotated summaries from one language to another, which can contain errors from both summarization and translation processes. To address this issue, we propose ConvSumX, a cross-lingual conversation summarization benchmark, through a new annotation schema that explicitly considers source input context. ConvSumX consists of 2 sub-tasks under different real-world scenarios, with each covering 3 language directions. We conduct thorough analysis on ConvSumX and 3 widely-used manually annotated CLS corpora and empirically find that ConvSumX is more faithful towards input text. Additionally, based on the same intuition, we propose a 2-Step method, which takes both conversation and summary as input to simulate human annotation process. Experimental results show that 2-Step method surpasses strong baselines on ConvSumX under both automatic and human evaluation. Analysis shows that both source input text and summary are crucial for modeling cross-lingual summaries.

heading: 1 Introduction
text: With the advance in deep learning and pre-trained language models (PLMs) (Devlin et al., 2019; Lewis et al., 2020; Raffel et al., 2020), much recent progress has been made in text summarization (Liu and Lapata, 2019; Zhong et al., 2022a; Chen et al., 2022a). However, most work focuses on English (En) data (Zhong et al., 2021; Gliwa et al., 2019; Chen et al., 2021), which does not consider crosslingual sources for summarization (Wang et al., 2022b). To address this limitation, cross-lingual summarization (CLS) aims to generate summaries in a target language given texts from a source language (Zhu et al., 2019), which has shown values
∗Yue Zhang is the corresponding author.
to both academic and industrial communities (Bai et al., 2021; Perez-Beltrachini and Lapata, 2021).
Most existing work (Zhu et al., 2019; Bai et al., 2021; Feng et al., 2022) constructs CLS corpora by translating summaries from existing mono-lingual summarization datasets into other languages, which is de facto a “pipeline” annotation protocol (first summarize, then translate) as shown in Figure 1. However, such an annotation method can suffer from two major problems: First, summaries from mono-lingual summarization corpora (summarization process) can contain errors (Liu et al., 2022), which are likely to be preserved in translated summaries. For example, the English summary in Figure 1-(a) contains unmentioned con-
9332
tent/hallucination (red text), which leads to the same discrepancy as in the translated summary (Figure 1-(b), red text). Second, the translation process can further introduce errors, in particular for polysemous words. For example, in Figure 1- (b), the term, “Ex-Viking” (which refers to previous members of the Minnesota Vikings team), is mistakenly translated into “前海盗” (which means “expirate/buccaneer”). To determine proper translation, it require more information beyond the scope of short summaries.
To qualitatively understand the above problems, we conduct human evaluation and error analysis on existing popular CLS corpora. Empirical results show that existing corpora suffer from the two aforementioned problems, containing significantly many hallucinations and factual errors.1 In particular, we find that overall 20 ∼ 67% of summaries in CLS datasets contain errors, where 7 ∼ 46% and 13 ∼ 47% of summaries suffer from summarization and translation processes, respectively. This suggests that the pipeline protocol, which is widely used in CLS research, can result in low-quality data and negatively impact the validity of modeling research. In addition, fine-grained error analysis shows that 55.6 ∼ 89.1% of translation errors can be resolved with the help of input context.
Motivated by the above findings and to address this issue, we propose the protocol that crosslingual summaries should be sourced from original input text where mono-lingual summaries can serve as a quick review for salient information. With this concept, we annotate cross-lingual summaries (Stgt) by relying on source text (Dsrc) and sourcelanguage summaries (Ssrc) as shown in Figure 1- (c). Such an annotation protocol brings three advantages: First, compared with translation only given Ssrc, rich context information from Dsrc helps annotators to disambiguate word senses and comprehend Ssrc accurately, e.g., “前维京人队” (which means “ex-viking team player”) in Figure 1-(c); Second, Dsrc is more reliable and can provide ground-truth information to correct potential errors in Ssrc, e.g., red text in Figure 1-(a); Third, compared with writing Stgt only given Dsrc, Ssrc can serve as supplement guidance to help annotators be aware of what should be involved in the summaries, ensuring that salient information in Ssrc and Stgt is aligned. 1The term error later in this paper refers to errors that are hallucinations or can cause factual misunderstandings, except when otherwise specified.
Using CLS protocol, we build ConvSumX, a new benchmark to facilitate future CLS research. ConvSumX focuses on conversational text in a few-shot setting. Compared with monologue (e.g., news), conversational text is less explored yet is also practically useful in real-world scenarios (Chen et al., 2022b). ConvSumX contains two sub-tasks, namely DialogSumX and QMSumX, based on two English conversation summarization datasets DIALOGSUM (Chen et al., 2021) and QMSum (Zhong et al., 2021), respectively. Each covers three language-directions, taking En as the source, and Mandarin (Zh), French (Fr) and Ukrainian (Ukr) as target languages. We empirically compare different annotations using the pipeline protocol and our CLS protocol with human evaluation. Analysis shows that by considering input context, our protocol can significantly reduce annotation errors, suggesting ConvSumX is a high-quality benchmark in terms of cross-lingual faithfulness.
Based on the same intuition that Dsrc and Ssrc
can serve as a critical complement to each other, we propose a 2-Step framework for CLS, which fine-tunes a multi-lingual PLM using concatenated Ssrc and Dsrc as input, and Stgt as output. Experimental results show that our conceptual framework yields surprisingly better performance over strong baselines on ConvSumX. Analysis and human evaluation show that our method can effectively generate more faithful cross-lingual summaries in a low-resource setting, and verify that source input text and summaries are supplementary to each other in modeling cross-lingual summaries.
To summarize, our contributions are the following:
1. We systematically review the pipeline annotation protocol and show that such a protocol can result in low-quality data (§ 2); 2. We propose the concept that CLS should be sourced from both source input text and source-language summaries and under our protocol, we present ConvSumX benchmark (§ 3), where QMSumX is the first queryfocused CLS dataset. 3. Under the same concept, we propose a simple yet effective 2-Step framework for CLS (§ 4), which demonstrates the necessity of both source input text and mono-lingual summary for CLS modeling.
We release ConvSumX at https://github.com/ cylnlp/ConvSumX.

heading: 2 Analyzing Existing CLS Corpora
text: We conduct a corpus-based study on existing popular human-annotated CLS corpora, namely NCLS, XSAMSum and XMediaSum, covering both monologue and dialogue texts.
NCLS (Zhu et al., 2019) is the first large crosslingual news summarization corpus, which is constructed by automatically translating existing monolingual summarization datasets and using a roundtrip strategy with human post-editing on test sets.
XSAMSum and XMediaSum are both from CLIDSUM (Wang et al., 2022a), where they manually translate summaries from two English dialogue summarization datasets, namely SAMSum (Gliwa et al., 2019) and MediaSum (Zhu et al., 2021), into Mandarin and German.
2.1 Error Analysis on Pipeline Annotation Since all 3 corpora have the task of summarizing English (En) documents into Mandarin (Zh) summaries, we perform human evaluation on this language direction. For each corpus, we randomly extract 100 instances from its training and testing sets, respectively, resulting in a total of 600 instances to evaluate. Each instance consists of English document (Den) and summary (Sen), and Mandarin summary (Szh).
We invite two expert translators, who are native in Mandarin and professional in English, as our judges and ask them to first evaluate whether the Szh contains errors or not, by evaluating the Szh against Den (IAA2: 0.67, substantial agreement). If Szh is found errors, the judges are asked to identify where such errors come from (IAA: 0.80, substantial agreement). Specifically, if this error is also found in Sen, we regard that it is caused by the mono-lingual summarization process; if this error is only found in Szh but not in Sen, we regard that it is caused by the translation process. In this process, we only focus on factual errors, and minor syntax errors are ignored.
Table 1 shows the evaluation result. Overall, we see that all CLS corpora show high error frequencies (20 ∼ 67%), indicating existing CLS can be less accurate. In particular, all mono-lingual summarization annotation contains errors (7 ∼ 46%), which are preserved in the CLS corpora. Moreover, the cross-lingual annotation process can in-
2We measure Inter-Annotator Agreement (IAA) by calculating their Pair-wise Cohen kappa score on 60 quiz instances.
vite more errors (13 ∼ 47%). This verifies our assumption that the pipeline annotation protocol, which ignores valuable input context, can lead to poor data quality.
In particular, NCLS contains the most errors, which can be because in addition to the different quality of their original mono-lingual summaries, Szh in NLCS are automatically translated by MT systems. Although human post-editing is conducted on the test set, factual errors are still frequent in the test set compared with the training set. This can be because their post-editing focuses on poor fluency and translationese, while correcting factual errors or hallucinations requires information from the source text, which is not presented to human editors. In addition, the averaged number of words in NCLS is much larger than in XMediaSum and XSAMSum,3 making translation more difficult.
The major contradiction between frequent errors according to our analysis and the high data quality reported by (Zhu et al., 2019) and (Wang et al., 2022a) can be explained by different reference sources, where our results show that these datasets have limitations in the choice of source for reference. For example, when only given Sen (“Fifty Five Percent... Ex-Viking...”) as reference, an Szh (“55%的美国人...前海盗”) can be considered as a correct translation (Figure 1-b). However, when evaluated against Den, Szh is considered to have hallucination errors (“55%的美国人(fifty five percent...)”) and impropoer translation (“前海盗(expirate)”, which should have been translated into to “前维京人队员(ex-viking team member)”).
3Avg. token. length in English summaries: NCLS (55.2) XMediaSum (14.4), XSAMSum(20.3).

heading: 2.2 In-depth Analysis on Translation Errors
text: To further understand why directly translating English summaries can invite so many errors, we perform an error analysis on summaries containing translation errors and categorize them. In particular, the two judges first identify whether the translation error can be resolved by considering the input context, or not, assuming that the errors can be caused by lacking input context (e.g., polyseme translation), and other translation errors (e.g., inconsistent translation). We categorize the former error types based on their linguistic typologies (avg. IAA: 0.62, substantial agreement):
Word Sense (W.S.): the translation of a word/phrase is incorrect under source input context. Terminology (Ter.): the translation of a word/phrase can be semantically correct but is improper in source input domains. Coreference (C.): the translation of coreference expressions refer to incorrect objectives. Sentence Relation (S.R.): The relation between two sentences/clauses is induced incorrectly or the translation of a sentence is incorrect because of misunderstanding the interrelation/structure of a sentence. Others (Oth.): simple errors such as typos or less accurate translation.
Table 2 presents the error types and their error counts. First, we see that errors (W.S, Tem., C. and S.R. together: 8 ∼ 41) caused by lacking input context are more than other translation errors (Oth.: 5 ∼ 12). This further suggests the necessity of considering input text when annotating CLS corpora. In addition, word sense sees overall most errors (26.32 ∼ 51.02%, avg. 41.81%), which is in line with the intuition that lacking context can
mostly lead to word sense ambiguity. Moreover, all categories see error instances, suggesting that such problematic summaries can confuse humans at multiple levels of language understanding.
Appendix A shows detailed information about our judges and Appendix B shows cases of different translation error types and their analysis.

heading: 3 ConvSumX
text: To address the aforementioned issues in pipeline annotation, we propose ConvSumX with a new annotation protocol, focusing on few-shot CLS. ConvSumX contains two cross-lingual summarization scenarios, namely daily dialogue summarization, and query-based summarization, covering 3 language directions: En2Zh, En2Fr and En2Ukr.

heading: 3.1 Data Source
text: We choose DIALOGSUM (Chen et al., 2021) and QMSum (Zhong et al., 2021) for ConvSumX by considering their potential to build real-world applications, and annotating their test and dev sets.
DIALOGSUM DIALOGSUM (Chen et al., 2021) is a real-life scenario dialogue summarization dataset, including various types of task-oriented dialogues.
QMSum QMSum (Zhong et al., 2021) is a querybased meeting summarization dataset, covering the academic, product and committee domains. We select data from academic and product for annotation.

heading: 3.2 Annotation
text: As discussed in § 2, the final quality of CLS corpora can be influenced by both summarization process and translation process, most of which can be resolved with the information from input documents. Therefore, instead of merely focusing on summaries in source languages, we ask annotators to write summaries in target languages (Stgt) directly by considering both input documents (Dsrc) and pre-annotated summaries (Ssrc). We refer to our protocol as CLS protocol.
We take English as the source language and choose Mandarin, French and Ukrainian as target languages because they are from different language families, and have different morphological variations and syntactic structures, with the potential to benefit other languages in their families. We invite expert translators, who are native in target
languages and professional in English, as our annotators (Appendix A). We ask annotators to first comprehend Dsrc, and then write Stgt with the help of Ssrc. In addition to the standard annotation criteria of DIALOGSUM and QMSum, we ask our annotators specifically pay attention to the following aspects featuring the CLS:
• Cross-lingual Consistency: Although being in different languages, the semantic information of Stgt should be consistent with Dsrc, in particular for polysemous words or phrases. • Language Style and Terminology: Annotators should write Stgt in the same language style of Ssrc, and use proper terminologies in some certain domains, such as academic meetings. • Translationese: The annotated summaries should be natural in the target languages.
For QMSum, annotators are additionally asked to write a query in target languages (Qtgt) with the help of the query in source language (Qsrc), where Qtgt and Stgt form a QA pair.
Before annotation, we ask each annotator to label training samples (10% of each dataset) until all annotated instances meet our requirements. After annotation, each instance is reviewed by an editor, who is also an expert translator. Editors are asked to first read the annotated summary to identify whether it is natural and readable in target languages, and then evaluate it against source input document to identify whether there are any factual errors. If any errors are found, we ask the corresponding annotator to re-annotate the whole batch and repeat this checking and re-annotation process until all summaries are correct. As mono-lingual summarization process can also contain large errors (§ 2.1), we additionally require annotators to modify English summaries/queries if any errors are found. Table 3 presents the percentage of summaries that contain errors in the original datasets.
Finally, we split the original dev sets into our new training and dev sets and keep the test set unchanged (DialogSumX: 400/100/500 and QMSumX: 157/40/209).

heading: 3.3 Comparison between ConvSumX with
text: Pipeline Annotation Data
To qualitatively compare CLS and pipeline annotation protocols in a fair setting (e.g., to remove the influence of different data sources), we additionally annotate instances using the pipeline approach, i.e., directly translating English summaries into Mandarin. We randomly sample 100 instances from dev/test sets of DIALOGSUM and QMSum, referring to them as DialogSum-P and QMSum-P, respectively. Overall, we have 400 instances to annotate and 800 instances to evaluate.
These data are annotated by the same annotators, using the same quality control process as ConvSumX. To avoid priori knowledge from input context for pipeline annotation, this process is conducted before ConvSumX annotation. Then, we perform human evaluation on those translated data and corresponding data in ConvSumX using the same method as described in § 2.1 in an anonymous way. For ConvSumX, we take corrected English summaries as pseudo translation for evaluation. Table 4 shows the human evaluation results.
Consistent with our findings (§2.1), DialogSumP and QMSum-P contain errors (11 ∼ 31) from both the summarization and translation processes. In contrast, ConvSumX contains fewer errors (0 ∼ 2),4 indicating the necessity of our CLS annotation protocol.

heading: 3.4 Characteristics of ConvSumX
text: Table 5 presents a comparison between ConvSumX and other CLS corpora, highlighting the unique features of ConvSumX. Firstly, ConvSumX is de-
4All errors that we find in § 3.3 are further corrected in ConvSumX. The final ConvSumX, which is used for training and evaluating models in § 5, contains no errors that we can find.
signed for spoken conversation summarization and encompasses two real-world scenarios. Notably, QMSumX is the first corpus addressing querybased CLS. Secondly, ConvSumX includes multiple languages from diverse families (French: Romance; Mandarin: Chinese; Ukrainian: Slavic; English: Germanic), positioning it as a valuable resource for studying cross-lingual generalization and language transfer. Furthermore, ConvSumX is the pioneering benchmark for CLS research involving the low-resource language, Ukrainian. Last, ConvSumX is the first CLS benchmark that forsakes the pipeline annotation protocol, which is essentially different from all existing human-crafted corpora. The low error frequencies demonstrate its cross-lingual faithfulness.

heading: 4 Method

heading: 4.1 Setting
text: Generally, the task of few-shot CLS is defined as: given a source input text Dsrc, few-shot CLS is to generate a summary in a target language Stgt by learning a limited number of gold-annotated ⟨Dsrc, Stgt⟩ data, with the help of external knowledge, which can be from mono-lingual summarization data, machine translation data and PLMs.
Specifically, for query-focuse CLS, the system is asked to generate Stgt given Dsrc with a query in the target language Qtgt.

heading: 4.2 Models
text: We evaluate two standard CLS baselines, namely pipeline method and End2End method, and propose a novel 2-Step framework, which differ from each other in the way the cross-lingual summary is generated. Figure 2 summarizes the main difference between their workflows.
Pipeline Method Previous work decomposes CLS into mono-lingual summarization and machine translation (Zhu et al., 2019), by deploying first-summarize, then-translate (S-T) or firsttranslate, then-summarize (T-S) strategies.
We compare with S-T as it can benefit from large mono-lingual summarization and monologue translation data, while T-S has been proven much worse (Feng et al., 2022) as both dialogue translation and non-English summarization data are very limited. For QMSumX, we additionally translate Qtgt into Qsrc before mono-lingual summarization and translation, to which we refer as T-S-T.
End2End Method Previous work models the CLS task and has shown better performance on previous datasets compared with pipeline methods (Zhu et al., 2019; Xu et al., 2019).
We compare two End2End methods: First, we directly fine-tune a multi-lingual model on ⟨Dsrc, Stgt⟩ (DialogSumX) and ⟨{Qtgt;Dsrc}, Stgt⟩ (QMSumX), marked as E2E; Second, inspired by Bai et al. (2021), where an End2End model first generates mono-lingual summary and then cross-lingual summary in an auto-regressive way and shows good performance in few-shot setting, we fine-tune a multi-lingual model on ⟨Dsrc, {Ssrc;Stgt}⟩ (DialogSumX) and ⟨{Qtgt;Dsrc}, {Ssrc;Stgt}⟩ (QMSumX), marked as E2M (M means mixed).
2-Step Method Inspired by our data analysis (§ 2) that mono-lingual summary can help guiding salient information for cross-lingual summary, and generating proper translation requires information from source input text, we proposed a 2-Step method. Conceptually, 2-Step is designed to simulate human annotation, where we ask an end2end model to generate Stgt given concatenated Ssrc
and Dsrc. Compared with pipeline methods, 2- Step method can explicitly make use of information from source input. Compared with End2End methods, 2-Step can focus on relevant information with the help of mono-lingual summaries.
Similarly, for QMSumX, we obtain the source language summaries by first translating Qtgt into Qsrc and then using mono-lingual summarizers. During inference, we use model-generated source summaries as Ssrc, which are obtained using the same way of pipeline methods.
Note all individual models are in a seq2seq manner. The terms “pipeline”, “End2End” and “2-Step” are stated from the perspective between source input text and output cross-lingual summaries.

heading: 5 Experiments
text: Metrics For automatic evaluation, we use ROUGE (Lin, 2004)5 and BERTSCORE (Zhang et al., 2020)6. ROUGE measures the n-gram overlap between generated and reference summaries. BERTSCORE calculates the pariwise cosine similarity between BERT (Devlin et al., 2019) token embeddings of generated and reference summaries. We report the F -1 scores of ROUGE1 (R1), ROUGE-2 (R2), ROUGE-L (RL) and BERTSCORE (BS).
Implementation Details For mono-lingual generation, we use UNISUMM7 for model initialization, further pre-training it on original training sets of DIALOGSUM and QMSum, and then
5https://github.com/csebuetnlp/xl-sum 6https://github.com/Tiiiger/bert_score 7https://github.com/microsoft/UniSumm
prefix-tuning it on our few-shot training data. For cross-lingual generation (MT or CLS), we use mBART-large-50-many-to-many-mmt8 for model initialization and then fine-tune it on our crosslingual data. All experiments are conducted on NVIDIA A100 GPU. We conduct a hyperparameter search for learning rate and batch size, from [1.5e-4, 1e-4, 5e-5, 3e-5, 1e-5] and [8, 16, 32, 64], and choose the best checkpoint based on R2 score on our few-shot dev sets.

heading: 5.1 Main Results
text: The main results on DialogSumX (DX) and QMSumX (QX) are shown in Table 6. In general, we find that our 2-Step system achieves the best results in most languages and the best averaged results on both tasks. In particular, 2-Step system outperforms pipeline method (S-T) (avg. improvement: 0.19 R2 and 0.24 BS scores on DX; 0.61 R2 and 1.39 BS scores on QX). It also outperforms End2End models by a large margin (avg. improvement: 4.73 ∼ 5.78 R2 and 2.36 ∼ 2.79 BS scores on DX; 1.65 R2 and 2.69 BS scores on QX). Note that 2-Step system is additionally presented with source summary and input text information compared with E2E and S-T systems. Thus, the superiority of 2-Step demonstrates that the source document and source summary are crucial in modeling cross-lingual summaries, and are complementary to each other.
Moreover, S-T outperforms End2End models. The contradiction between our results and previous finding (Zhu et al., 2019; Bai et al., 2021) can
8https://huggingface.co/facebook/ mbart-large-50-many-to-many-mmt
be explained by the fact that the summarizer and translator we use are much stronger and the error propagation problem is less severe. Also, S-T can benefit from our high-quality parallel cross-lingual summary pairs (Ssrc and Stgt) as few-shot translation data, while previous work ignores such valuable data and only uses a fixed MT system without fine-tuning (Zhu et al., 2019).
All CLS systems perform better at En2Zh and En2Fr than En2Ukr. The high performance on En2Zh and En2Fr can be explained by that both Zh and Fr are highly-rich resource data on which mBART-50 is pre-trained (Tang et al., 2021), and mBART-50 can easily bridge the alignment between texts in Zh/Fr and En. In contrast, Ukr is a low-resource language, on which the mBART-50 performs poorly. All systems have higher performance on DX compared with QX, which is because QX is more challenging w.r.t the task of querybased summarization for long text and more extreme few-shot setting, and its domain is very different from mBART-50’s pre-training data.
We notice that all models perform better on QX En2Fr than En2Zh and En2Ukr. A possible reason can be that QX contains many professional in-domain words whose word sense can be multiple and very different from its general ones. The sense of these words can be different lexical items, in particular for Zh or Ukr, which are typologically different from En (Chen and Ng, 1989; BudzhakJones, 1998). In contrast, Fr and En both use Latin script and are more similar in terms of morphology and lexicon rules (Kirsner et al., 1984; Pacton and
Deacon, 2008; Fan et al., 2021) compared with Zh and Ukr. For example, “discourse” can be mapped into “论文(academic paper)/讲述(talk)/...” in Zh and “diskusicq (discussion)/diskurs (linguistic discourse)” in Ukr, while “discours (discussion/linguistic...)” in Fr.
We also conduct experiments on pipelined datasets, XSAMSum and XMediaSum (Appendix C). Experimental results show that, with a fine-tuned translator, S-T method outperforms best-reported systems on most tasks. Moreover, 2-Step does not show better performance than S-T, which can be because 2-Step systems are trained to only translate source summaries instead of comprehending source input text. The high performance of S-T emphasizes that cross-lingual summaries in those pipelined datasets do not rely on source input text, which can rather be a translation task. This confirms our motivation that the pipeline annotation protocol has important limitations.

heading: 5.2 Human Evaluation
text: To comprehensively understand CLS systems, we conduct human evaluations of the model outputs, as multi-dimensional assessment offers a more robust and holistic perspective (Zhong et al., 2022b).
Following previous work (Kryscinski et al., 2019; Fabbri et al., 2021), we evaluate generated summaries from the following dimensions: Fluency evaluates the quality of generated sentences, including grammar and whether it is natural; Coherence evaluates the collective quality of generated summaries; Relevance evaluates the importance of information in generated summaries; Consistency evaluates factual alignment between generated sum-
maries and source input texts. We randomly extract 50 summaries from S-T and 2-Step outputs on ConvSumX for each language, and ask native speakers to give scores from 1 to 5. Higher scores indicate higher qualities.
The result is shown in Table 7. Generally, all metrics see low scores, suggesting the challenge of few-shot CLS. Both models see higher scores on DX compared with QX, which is consistent with our automatic evaluation. Compared with S-T, 2- Step achieves similar Relevance scores on all tasks. This is because the input source summary for both models is identical, thus the information in it is the same. However, 2-Step achieves higher Fluency, Coherence, and Consistency scores, which justifies our assumption that source input text information is critical, in particular for consistency.
We present case study of model outputs in Appendix D.

heading: 6 Related Work
text: CLS Corpora Existing CLS corpora construction can be categorized into two main protocols. 1) Pipeline annotation: translating summaries from MLS corpora into other languages and; 2) Automatic alignment: aligning summaries and input texts of different language versions.
Zhu et al. (2019) construct the first large-scale CLS dataset by automatically translating monolingual summaries using MT systems with a roundtrip strategy and manual post-editing on test sets. Bai et al. (2021) construct an En2De dataset using the same method. Feng et al. (2022) automatically translate summaries from SAMSum (Gliwa et al., 2019) into Russian, De and Zh. Wang et al. (2022a) manually translate summaries from SAMSum (Gliwa et al., 2019) and MediaSum (Zhu et al., 2021) into De and Zh. Different from them, we propose a new annotation protocol, which helps
annotators to comprehend documents quickly and accurately. To our knowledge, we are the first to address such human annotation issues for CLS research and present a new benchmark, ConvSumX.
A different line of work constructs CLS datasets by linking different language versions of online articles, such as Wikipedia (Perez-Beltrachini and Lapata, 2021) and WikiHow (Ladhak et al., 2020). Despite their cheap cost and large scale, there can be misalignment and hallucination problems. For example, Wikipedia articles and their leading paragraphs (pseudo summaries) of the same person in different languages can contain different contents.
CLS Models Early work on CLS focuses on a pipeline paradigm by first summarizing, then translating, or vice versa. However, due to the poor performance of early MT and summarization systems, such methods can often suffer from error propagation. With the advance of deep learning and PLM technologies, recent work deploys end-to-end methods. Zhu et al. (2019), Xu et al. (2020), Bai et al. (2021) and Wang et al. (2022a) propose multi-task learning or pre-training on large in-domain CLS, mono-lingual summarization and translation data. Different from them, we propose a 2-Step method under the same concept of sourcing from source input text with the guidance of source summary, which is free of pre-training on large and thus can be easily adapted to other tasks and languages.

heading: 7 Conclusion
text: We conducted data analysis on 3 typical corpora and showed that the pipeline annotation protocol suffers from errors from both the summarization and translation processes. To address these issues, we proposed that cross-lingual summaries should be sourced from source input text. Based on this principle, we annotated a more faithful CLS benchmark, ConvSumX by relying on both sourcelanguage texts and summaries. Based on the same intuition, we proposed a 2-Step method that takes both source text and source summaries as input. Experimental results showed that 2-Step method outperforms strong baselines on ConvSumX, demonstrating that both source-language texts and summaries are crucial in modeling cross-lingual summaries and are complementary to each other. To our knowledge, we are the first to show that summary translation has limitations for CLS, giving a more faithful solution.
Limitations
The limitation of this paper can be stated from three perspectives. First, although using our CLS annotation protocol can label more faithful data, the annotation cost is higher because annotators need to comprehend the full source text instead of only the source summary. Second, ConvSumX only covers 3 typical languages, while languages from different language families and have different morphology and lexical-/syntactic rules require further investigation. Third, although the proposed 2-Step method is effective, we simply concatenate the source input text and mono-lingual summary at the token level as the model input but do not make further exploration. We believe that more smart and sophisticated designs to integrate features from source input text and mono-lingual summary can further improve the CLS performance, which, however, we leave for future work.
Ethics Statement
Data Usage and License ConvSumX is based on two public English conversation summarization datasets, namely DIALOGSUM and QMSum. Both datasets are freely available online under the MIT license, which has no constraint to academic use, modification, and further distribution. We will follow the MIT license to make our data (annotated target summaries/queries and corrected English summaries/queries) freely available online.
Human Annotation The construction of ConvSumX involves human annotation. We hire 4 expert translators as our annotators and editors for each target language. The total cost is around 6, 500 USD, which applies to our annotation (including quiz annotation) and review. The hourly salary is equal. The total annotation time (including training annotation and editing) for Zh, Fr and Ukr is around 96, 96, and 120 hours (according to our annotation cost/hourly salary). Detailed information about our annotators/judges/editors can be found in Appendix A.
Content Safety During our annotation, annotators are explicitly asked to not involve any personal/violent information and to write summaries strictly limited to the scope of source input text. Also, if any violent or uncomfortable information is found in source input text, annotators are asked to report such issues. All data are further reviewed by editors. With careful checking and evaluation,
ConvSumX (including source input text) contains no personal/violent content, and is safe to use.

heading: Acknowledgement
text: We thank reviewers from ACL2023 for their suggestions. We extend special thanks to our metareviewers for their indispensable and exceptional contributions. We also appreciate expert translators from Lan-bridge who have played a crucial role in the development of ConvSumX. This work is funded by the Ministry of Science and Technology of China (grant No. 2022YFE020038) and National Natural Science Foundation of China (grant NSFC No. 62161160339).

heading: A Human Judges and Annotators
text: For human evaluation in § 2, we invite 2 expert translators as judges to conduct human evaluation and analysis of existing CLS corpora. For crosslingual summary annotation and mono-lingual correction (§ 3), we invite 3 translators as our annotators and 1 as an editor to evaluate human annotation and model outputs (§ 5.2) for each language direction. Additionally, we invite one senior translator as the project manager to monitor the whole annotation progress.
All expert translators are from Lan-bridge, a qualified institution for translation service9, recognized by the ISO10. All annotators, editors and judges are native in the target language (i.e., Chinese, French or Ukrainian), and professional in English. They are competent in translation, linguistic research and related information processing. They also have a good understanding of the textual background of certain culture, technology and domain. Our annotators and editors either got graduate certificates in translation major or got graduate certificates in other fields but have more than 2 years of full-time professional experience in translating. Besides the above requirements, the manager has more than 5-year experience in multi-lingual translation projects that cover the language directions as described in this paper.

heading: B Analysis and Cases of Translation Errors
text: As shown in Table 9 and Table 10, we present cases of each error type as discussed in § 2.2.
In Table 9, “Sheen” refers to an actor, while annotators translate it into “高光/Highlight”, and the term “queer group” into “同性恋群体/gay group”. Although “queer” has a meaning of “gay”, the proper translation should be “酷儿群体”. Also, in the Coreference case, “the date” refers to the day when “go do groceries together”, which is incorrectly translated into “聚会的日期/the date of party”. In Table 10 Sentence Relation, annotators confuse the meaning and relation between two sentences, and the translation is completely incorrect at the sentence semantic level.
All those translation cases together with summarization case (Figure 1) suggest the pipeline anno-
9Requirements for translation services: https://www. iso.org/standard/59149.html.
10International Organization for Standardization: https: //www.iso.org/home.html.
tation can contain a large number of errors.

heading: C Experiment on Pipelined Datasets
text: We conduct experiments on two pipelined datasets, namely XSAMSum and XMediaSum from the CLIDSUM benchmark and compare our pipeline and 2-Step methods with best-reported systems from (Wang et al., 2022a):
Summ-Trans Pipeline They use BART(Dall) for mono-lingual summarization (Feng et al., 2021), and Google Translate 11 for summary translation.
Trans-Summ Pipeline They use Google Translate to first generate cross-lingual dialogues, and then use mBART-50 to generate target language summaries.
mBARTE2E They directly fine-tune an mBART50 on cross-lingual ⟨Ssrc, Stgt⟩ pairs, which is also an E2E baseline in our § 4.
mDialBARTE2E They further pre-train an mBART-50 model using multiple tasks, including action filling, utterance permutation, mono-lingual
11https://cloud.google.com/translate
summarization and machine translation, on MediaSum (Zhu et al., 2021).
For more fair comparison, we fine-tune BART-large on corresponding mono-lingual data for mono-lingual summarization and fine-tune mBART-50 for translation and our 2-Step model. The result is shown in Table 8.
We see that without fine-tuning, our pipeline method (S-T∗) shows low performance. However, equipped with fine-tuned mBART as translator, ST outperforms all previous methods on all tasks (e.g., S-T outperforms the best-reported mDialBART on En2Zh XMediaSum by 1 R1) except for the S-T pipeline on En2De XSAMSum, which can be because that Google Translate is better than mBART-50 at En2De translation. However, our 2-Step method, which is explicitly presented with source dialogue information and outperforms ST on ConvSum, only shows comparable or even worse results compared with S-T on XSAMSum and XMediaSum. The contradiction of such model performance on CLIDSUM can be explained by that such pipelined datasets focus more on how to directly translate the mono-lingual summary, while adding more source dialogue information is less useful and sometimes harmful.

heading: D Case Study
text: To qualitatively demonstrate the advantage of 2- Step method, we present cases from S-T and 2- Step on ConvSumX. Figure 3 shows a sample from DialogSumX and Figure 4 shows another sample from QMSumX.
As shown in Figure 3, both summaries generated by 2-Step and S-T contain errors (“donson”, which should be “dawson”). However, compared with S-T (“下台”, which means “step down”), 2-Step method can improve the faithfulness of summaries (“发了一 份备忘录, which means “send a memo”). Similarly, as shown in Figure 4, S-T method suffers from generating unnatural language(e.g., a string of d-s in En2Zh case) and it has trouble generating some not-commonly used words (e.g., the counterpart word of cepstra in 3 target languages), while 2-Step method can significantly ameliorate such problems.
Moreover, we also find that 2-Step method not only focus on “translating” summaries from source language to target language, but also rewriting them by referring to the original input text. In the En2Zh example in Figure 4, 2-Step method properly generates “告诉” (which has a sense of “tell”) for the
word “inform”, which is more natural considering its textual background. In contrast, S-T method simply translates it into “通知”, which is more of the sense “announce/notify”, and is not natural in Mandarin.
These cases indicate that source-language texts are essential in cross-lingual summarization tasks, which further demonstrates our conclusion.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Limitations
3 A2. Did you discuss any potential risks of your work? Ethics Statement
3 A3. Do the abstract and introduction summarize the paper’s main claims? Abstract and Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 3, Section 4 and Section 5
3 B1. Did you cite the creators of artifacts you used? Section 3, Section 4 and Section 5
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Ethics Statement
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 3
3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Section 3
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Section 3
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 3
C 3 Did you run computational experiments? Section 4 and Section 5
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section 5
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 5
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 5
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 5
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 2, Section 3, and Section 5
3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Section 2, Section 3, Section 5 and Appendix A
3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Section 2, Section 3, Section 5 and Appendix A
3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Ethics Statement and Appendix A
3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Appendix A
3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Appendix A

heading: Future Work
text: We believe that more smart and sophisticated designs to integrate features from source input text and mono-lingual summary can further improve the CLS performance, which, however, we leave for future work