CrossSum: Beyond English-Centric Cross-Lingual Summarization for 1,500+ Language Pairs
Year: 2023

Abstract:
We present CrossSum, a large-scale crosslingual summarization dataset comprising 1.68 million article-summary samples in 1,500+ language pairs. We create CrossSum by aligning parallel articles written in different languages via cross-lingual retrieval from a multilingual abstractive summarization dataset and perform a controlled human evaluation to validate its quality. We propose a multistage data sampling algorithm to effectively train a cross-lingual summarization model capable of summarizing an article in any target language. We also introduce LaSE, an embedding-based metric for automatically evaluating model-generated summaries. LaSE is strongly correlated with ROUGE and, unlike ROUGE, can be reliably measured even in the absence of references in the target language. Performance on ROUGE and LaSE indicate that our proposed model consistently outperforms baseline models. To the best of our knowledge, CrossSum is the largest cross-lingual summarization dataset and the first ever that is not centered around English. We are releasing the dataset, training and evaluation scripts, and models to spur future research on cross-lingual summarization. The resources can be found at https: //github.com/csebuetnlp/CrossSum.

heading: Future Work
text: We are releasing the dataset, training and evaluation scripts, and models to spur future research on cross-lingual summarization. The resources can be found at https: //github. com/csebuetnlp/CrossSum. 1 Introduction
Cross-lingual summarization (hereinafter XLS) is the task of generating a summary in a target language given a source text in another language. The task is challenging as it combines summarization and translation in one task, both challenging tasks in their own right. Earlier approaches to XLS thus employed pipeline methods such as translate-thensummarize (Leuski et al. , 2003) and summarizethen-translate (Wan et al. , 2010). Not only are they computationally expensive, having to use multiple
∗These authors contributed equally to this work. Input Article: [. ] 新型コロナウイルスに対し、様々な既存の 治療法の効果を試す世界的規模の臨床試験の一貫として、デキ サメタゾンが試された。(Dexamethasone was tested as part of a global clinical trial to test the effectiveness of various existing therapies against the new coronavirus. ) [. ] その結果、人 工呼吸器を必要とする重症患者の致死率が3割下がり。(As a result, the case fatality rate of critically ill patients who require a ventilator is reduced by 30%. ) [. ] ボリス･ジョンソン 英首相は「イギリス科学界の素晴らしい成果」を歓迎し。(British Prime Minister Boris Johnson welcomed "the great achievements of the British scientific community". ) [. ]「しかもこれ は、世界中で手に入る薬だ」("And this is a medicine available all over the world". ) [. ] きわめて安いステロイド剤だった (but a very cheap steroid that has been used for a long time. )
Summary: িবজ্ঞানীরা বলেছন েড�ােমথােসান নােম স�া ও সহজলভয্ একিট ওষুধ কেরানাভাইরােস গুরুতর অসু� েরাগীেদর জীবন রক্ষা করেত সাহাযয্ করেব। (Scientists say a cheap and readily available drug called dexamethasone will help save the lives of critically ill patients with coronavirus. )
Figure 1: A sample article-summary pair from CrossSum, the article is written in Japanese, and the summary is in Bengali. We translate the texts to English inside parentheses for better understanding. Words and phrases of the article relevant to the summary are color-coded. models, but these approaches also suffer from errorpropagation (Zhu et al. , 2019) from one model to another, degrading the overall performance. The success of sequence-to-sequence (seq2seq) models (Cho et al. , 2014; Sutskever et al. , 2014) and the advances in Transformer-based models (Vaswani et al. , 2017) have aided in the emergence of end-to-end methods that can perform XLS with one single model (Zhu et al. , 2019; Cao et al. , 2020b). The availability of XLS datasets (Ladhak et al. , 2020; Perez-Beltrachini and Lapata, 2021) has also helped this task gain popularity in recent times. However, they cover only a few languages, contain a small number of samples for training and evaluation, or use English as the pivot language (i. e. , the target language always remains English), thereby limiting their applicability to a great extent. 2541
To democratize XLS beyond high-resource languages, in this work, we introduce CrossSum, a large-scale XLS dataset containing 1. 68 million article-summary samples in 1,500+ language pairs. We align parallel articles1 written in different languages via cross-lingual retrieval from the multilingual XL-Sum (Hasan et al. , 2021) dataset. We introduce and rigorously study the notions ‘induced pairs’ and ‘implicit leakage’ to increase the coverage of the dataset while at the same time ensuring maximum quality. We also perform a controlled human evaluation of CrossSum spanning nine languages from high- to low-resource and show that the alignments are highly accurate. We design MLS, a multistage language sampling algorithm, for successfully training models that can generate a summary in any target language for an input article in any source language, both from a set of languages present in the training dataset. For the first time, we perform XLS with CrossSum on a broad and diverse set of languages without relying on English as the standalone pivot, consistently outperforming many-to-one and one-to-many models, as well as summarize-then-translate baselines. We propose LaSE, an embedding-based metric for evaluating summaries when reference summaries may not be available in the target language but may be available in another language, potentially opening new doors for evaluating lowresource languages. Furthermore, we demonstrate the reliability of LaSE by its high correlation with ROUGE (Lin, 2004), the de-facto metric for evaluating text summarization systems. To the best of our knowledge, CrossSum is the largest publicly available abdtractive XLS dataset, both in terms of the number of samples and the number of language pairs. We are releasing the dataset, training and evaluation scripts, and models hoping that these resources will encourage the community to push the boundaries of XLS beyond English and other high-resource languages. 2 The CrossSum Dataset
The most straightforward way of curating a highquality XLS dataset is via crowd-sourcing (Nguyen and Daumé III, 2019). However, it may be difficult to find crowd workers having professional command over low-resource languages or distant language pairs. Moreover, scalability issues might arise due to the time and budget constraints for
1We re-purpose the terminology of parallel corpus here. crowd-sourcing. Therefore, synthetic (Zhu et al. , 2019) and automatic methods (Ladhak et al. , 2020; Perez-Beltrachini and Lapata, 2021) have gained traction over crowd-sourcing. Automatic curation of an XLS dataset is simply to pair an article A in a source language with the summary of a parallel article B written in a different target language (Figure 1), assuming the availability of a multilingual dataset having identical contents in different languages. Two contemporary works have compiled large-scale multilingual summarization datasets, namely XL-Sum (Hasan et al. , 2021) (1. 35M samples in 45 languages) and MassiveSumm (Varab and Schluter, 2021) (28. 8M samples in 92 languages). Though substantially larger than the other, MassiveSumm is not publicly available. Since public availability is crucial for promoting open research, we opted for XL-Sum, distributed under a non-commercial license. Additionally, all articles of XL-Sum are crawled from a single source, BBC News. We observed that BBC publishes similar news content in different languages and follow similar summarization strategies. Hence adopting XL-Sum would increase the quality and quantity of the article-summary pairs. Unlike previous automatic methods, there are no explicit links between parallel articles in XL-Sum. Fortunately, language-agnostic sentence representations (Artetxe and Schwenk, 2019a; Feng et al. , 2022) have achieved state-of-the-art results in crosslingual text mining (Artetxe and Schwenk, 2019b), and hence, we use them to search identical contents across languages. For simplicity2, we perform the search over summaries only. To ensure maximum quality, we set two conditions for a summary SA in language A to be aligned with another summary SB in language B:
1. SB must be the nearest neighbor of SA among all summaries in B, and vice-versa. 2. The similarity between SA and SB must be above the threshold, τ. The similarity of a summary pair is measured by the inner product of their Language-agnostic BERT Sentence Embeddings (LaBSE) (Feng et al. , 2022) (a unit vector for an input text sequence). We empirically set the similarity threshold as the average over all languages that maximized their respective F1 score (τ = 0. 7437) in the BUCC mining tasks (Zweigenbaum et al. , 2017). 3
2The entire procedure is described in Appendix A. 3Around 90% F1 is achieved using LaBSE in BUCC, hence not all CrossSum alignments will be correct. Therefore,
Induced Pairs We observed that many summary pairs, despite being nearest neighbors in their language pairs, were filtered out because of the threshold τ. Although interestingly, both were aligned with the same summary in a different language. Moreover, these pairs are prevalent if their languages are distant or low-resource. LaBSE uses contrastive learning (Guo et al. , 2018; Yang et al. , 2019) to rank parallel sentences over non-parallels. Since parallel pairs are mostly found for highresource and linguistically close languages, we hypothesize that LaBSE fails to assign high similarity to sentences from languages that are not. To include these pairs into CrossSum, we introduce the notion ‘induced pairs. ’ Formally, two summaries SA, SB in languages A, B are induced pairs if they are nearest neighbors of each other in A, B, their similarity score is below τ , and both are aligned with SC in language C, or through a chain of aligned pairs (SA, SC), (SC , SD), · · · , (SY , SZ), (SZ , SB) in languages {C, D, · · · , Y, Z}. We thus incorporate the induced pairs into CrossSum through a simple graph-based algorithm. First, we represent all summaries as vertices in a graph and draw an edge between two vertices if the summaries are aligned. Then we find the connected components in the graph and draw edges (i. e. , induced pairs) between all vertices in a component. Again to ensure quality, before computing the induced pairs, we use the max-flow min-cut theorem (Dantzig and Fulkerson, 1955) considering the similarity scores as edge weights to limit the size of each component to 50 vertices (since ideally, a component should have at most 45 vertices, one summary from each language) and set their minimum acceptance threshold to τ ′ ← τ − 0. 10. in the following section, we further assess the quality of the alignments using human evaluation. We finally assembled the originally aligned pairs and induced pairs to create the CrossSum dataset. Figure 6 (Appendix) shows the article-summary statistics for all language pairs in CrossSum. As evident from the figure, CrossSum is not centered only around the English language but rather distributed across multiple languages. Implicit Leakage We initially made the traindev-test splits respecting the original XL-Sum splits and performed an initial assessment of CrossSum by training a many-to-one model (articles written in any source language being summarized into one target language). Upon evaluation, we found very high ROUGE-2 scores (around 40) for many language pairs, even reaching as high as 60 for some (Figure 2). In contrast, Hasan et al. (2021) reported ROUGE-2 in the 10-20 range for the multilingual summarization task. We inspected the model outputs and found that many summaries were the same as the references. Through closer inspection, we found that their corresponding articles had a parallel counterpart occurring in the training set in some other language. During training, the model was able to align the representations of parallel articles (albeit written in different languages) and generate the same output by memorizing from the training sample. While models should undoubtedly be credited for being able to make these cross-lingual mappings, this is not ideal for benchmarking purposes as this creates unusually high ROUGE scores. We denote this phenomenon as ‘implicit leakage’ and make a new dataset split to avoid this. Before proceeding, we deduplicate the XL-Sum dataset4 using semantic similarity, considering two summaries SA, S′A in language A to be duplicates of one another if
4XL-Sum has been deduplicated using lexical overlap methods only. But due to the risk of implicit leakage, which is not lexical, we further perform semantic deduplication. their LaBSE representations have similarity above 0. 95. We take advantage of the component graph mentioned previously to address the leakage and assign all article-summary pairs originating from a single component in the training (dev/test) set of CrossSum, creating an 80%-10%-10% split for all language pairs. Since parallel articles no longer appear in the training set of one and the dev/test set of another, the leakage is not observed anymore (Figure 2). We further validated this by inspecting the model outputs and found no exact copies. 3 Human Evaluation of CrossSum
To establish the validity of our automatic alignment pipeline, we conducted a human evaluation to study the quality of the cross-lingual alignments. We selected all possible combinations of language pairs from a list of nine languages ranging from high-resource to low-resource to assess the alignment quality in different pair configurations (e. g. , high-high, low-high, low-low) as per the language diversity categorization by Joshi et al. (2020). We chose three high-resource languages, English, Arabic, and (simplified) Chinese (categories 4 and 5); three mid-resource languages, Indonesian, Bengali, and Urdu (category 3); and three low-resource languages, Punjabi, Swahili, and Pashto (categories 1 and 2), as representative languages and randomly sampled fifty cross-lingual summary alignments from each language pair for annotation. As a direct evaluation of these pairs would require bilinguallyproficient annotators for both languages, which are practically intractable for distantly related languages (e. g. , Bengali-Swahili), we resorted to a pivoting approach during annotation for language pairs that do not contain English. For a language pair (l1− l2), where l1 ̸= en and l2 ̸= en, we sampled alignments (x, y) such that ∃(x, e) ∈ (l1−en) and ∃(y, e) ∈ (l2 − en), for an English article e. In other words, we ensure that both the articles of the sampled cross-lingual pair have a corresponding cross-lingual pair with an English article. An alignment (x, y) would be deemed correct if both (x, e) and (y, e) are correct. This formulation thus reduced the original problem to annotating samples from language pairs (l1−en) and (l2−en), where l1 and l2 are from the previously selected languages that are not English. We hired bilingually proficient expert annotators adept in the language of interest and English. Two annotators labeled each language pair where one
language is English. We presented them with corresponding summaries of the cross-lingual pairs (and optionally the articles themselves) and elicited yes/no answers to the question:
“Can the provided sequences be considered summaries for the same article?”5
We deem a sequence pair accurate if both annotators judge it as valid. We show the alignment accuracies of the language pairs in Figure 3. As evident from the figure, the annotators judge the aligned summaries to be highly accurate, with an average accuracy of 95. 67%. We used Cohen’s Kappa (Cohen, 1960) to establish the interannotator agreement and show the corresponding statistics in Table 3 in the Appendix. 4 Training & Evaluation Methodologies
In this section, we discuss the multistage sampling strategy for training cross-lingual text generation models and our proposed metric for evaluating model-generated summaries. 4. 1 Multistage Language Sampling (MLS)
From Figure 6, it can be observed that CrossSum is heavily imbalanced. Thus, training directly without upsampling low-resource languages may result in their degraded performance. Conneau et al. (2020)
5We do not explicitly evaluate article-summary correctness as this has already been studied in work on XL-Sum. This was also done to reduce annotation costs. used probability smoothing for upsampling in multilingual pretraining and sampled all examples of a batch from one language. However, extending this technique to the language pairs in CrossSum would result in many batches having repeated samples as many language pairs do not have enough training samples in total compared to the batch sizes used in practice (e. g. , Conneau et al. (2020) used a batch size of 256, which exceeds the training set size of nearly 1,000 language pairs in CrossSum). At the same time, many language pairs would not be sampled during training for lack of enough training steps (due to our constraints on computational resources). To address this, we adapt their method to introduce a Multistage Language Sampling algorithm (MLS) to ensure that the target summaries of a batch are sampled from the same language. Let L1, L2,. , Ln be the languages of a crosslingual source-target dataset, and cij be the number of training samples where the target is from Li and source from Lj. We compute the probability pi of each target language Li by
pi = ∑n k=1 cik∑n
j=1 ∑n k=1 cjk ∀i ∈ {1, 2,. , n}
We then use an exponent smoothing factor α and normalize the probabilities
qi = pαi∑n j=1 p α j ∀i ∈ {1, 2,. , n}
Given the target language Li, we now compute the probability of a source language Lj , represented by pj|i. pj|i = cij∑n k=1 cik ∀j ∈ {1, 2,. , n}
We again smooth pj|i by a factor β and obtain the normalized probabilities
qj|i = pβj|i∑n k=1 p β k|i ∀j ∈ {1, 2,. , n}
Using the probabilities, we describe the training process with the MLS algorithm in Algorithm 1. Note that the proposed algorithm can be applied to any cross-lingual seq2seq task where both the source and target languages are imbalanced. 4. 2 Evaluating Summaries Across Languages
A sufficient number of reference samples are essential for the reliable evaluation of model-generated summaries. However, for many CrossSum language pairs, even the training sets are small, let
Algorithm 1: Multistage Language Sampling (MLS)
Input: Dij ∀i, j ∈ {1, 2,. , n}: training data with tgt/src languages Li/Lj ; cij ← |Dij | ∀i, j ∈ {1, 2,. , n}; m: number of mini-batches. 1 Compute qi, qj|i using cij 2 while (Model Not Converged) do 3 batch← ϕ 4 Sample Li ∼ qi 5 for k ← 1 to m do 6 Sample Lj ∼ qj|i 7 Create mini-batch mb from Dij 8 batch← batch ∪ {mb} 9 Update model parameters using batch
alone the test sets (the median size is only 33). For instance, the Japanese-Bengali language pair has 34 test samples only, which is too few for reliable evaluation. But the size of the in-language6 test sets of Japanese and Bengali are nearly 1,000. Being able to evaluate against reference summaries written in the source language would thus alleviate this insufficiency problem by leveraging the in-language test set of the source language. For this purpose, cross-lingual similarity metrics that do not rely on lexical overlap (i. e. , unlike ROUGE) are required. Embedding-based similarity metrics (Zhang et al. , 2020; Zhao et al. , 2019) have recently gained popularity. We draw inspiration from them and design a similarity metric that can effectively measure similarity across languages in a language-independent manner. We consider three essential factors: 1. Meaning Similarity: The generated and reference summaries should convey the same meaning irrespective of their languages. Just like our alignment procedure from Section 2, we use LaBSE to compute the meaning similarity between the generated (sgen) and reference summary (sref ):
MS(sgen, sref ) = emb(sgen)Temb(sref )
where emb(s) denotes the embedding vector output of LaBSE for input text s. 2. Language Confidence: The metric should identify, with high confidence, that the summary is indeed being generated in the target language. As such, we use the fastText language-ID classifier
6Both article and summary belonging to the same language
(Joulin et al. , 2017) to obtain the language probability distribution of the generated summary and define the Language Confidence (LC) as:
LC(sgen, sref ) = { 1, if Lref = argmaxP (Lgen) P (Lgen = Lref ), otherwise
3. Length Penalty: Generated summaries should not be unnecessarily long, and the metric should penalize long summaries. While model-based metrics may indicate how similar a generated summary is to its reference and language, it is unclear how they can be used to determine its brevity. As such, we adapt the BLEU (Papineni et al. , 2002) brevity penalty to measure the length penalty:
LP(sgen, sref ) = { 1, if |sgen| ≤ |sref |+ c exp(1− |sgen||sref |+c), otherwise
sgen and sref may not be of the same language, and parallel texts may vary in length across languages. Hence, we use a length offset c to avoid penalizing generated summaries slightly longer than the references. By examining the standard deviation of mean summary lengths of the languages, we set c = 6. We finally define our metric, Language-agnostic Summary Evaluation (LaSE) score as follows