PMIndiaSum: Multilingual and Cross-lingual Headline Summarization for Languages in India
Authors: Ashok Urlana
Pinzhen Chen
Zheng Zhao
Shay B Cohen
Manish Shrivastava
Barry Haddow
Venue: EMNLP 2023 Findings (2023)
Keywords: multilingual
cross-lingual
summarization
dataset
benchmark
languages in India
TL;DR: We propose a summarization dataset for languages in India which provides the current widest language pair coverage.

Abstract: This paper introduces PMIndiaSum, a multilingual and massively parallel summarization corpus focused on languages in India. Our corpus provides a training and testing ground for four language families, 14 languages, and the largest to date with 196 language pairs. We detail our construction workflow including data acquisition, processing, and quality assurance. Furthermore, we publish benchmarks for monolingual, cross-lingual, and multilingual summarization by fine-tuning, prompting, as well as translate-and-summarize. Experimental results confirm the crucial role of our data in aiding summarization between Indian languages. Our dataset is publicly available and can be freely modified and re-distributed.

PMIndiaSum: Multilingual and Cross-lingual Headline Summarization for Languages in India
Ashok Urlana".
Pinzhen Chen
Zheng Zhao²
Shay B. Cohen
Manish Shrivastava'
Barry Haddow2
'IIT Hyderabad
University of Edinburgh
ashok. urlana@research. iiit.iiic.... ac. in, {pinzhen. chen, zheng Jeed ac. uk scohen@inf ac. m. m.shrivastavaeiiin ac. in, bhaddow@ed. ac.
Abstract
article_ article___ur 1__r
article
headline 1 ur
This paper introduces PMIndiaSum, multi- lingual and massively parallel summarization corpus focused on languages India. Our cor- pus provides training and testing ground for four language families, 14 languages, and the largest with 196 language pairs. We detail our construction workflow including data acquisition, processing, and quality assurance. Furthermore, we publish benchmarks for mono- lingual, cross-lingual, and multilingual summa- rization by fine-tuning, prompting, as well as translate-and summarize. Experimental results saaaatt-anndssmmriiee....
article
headline 1
article 1_as
headline 1_bn
headline __as
news body
headlines in 14 languages
14 languages
PMIndiaSum supports 14 languages and 196 lan- Figure 1:
guage pairs, by sourcing from massively parallel articles.
confirm the crucial role of our data in aiding summarization between Indian languages. Our dataset is publicly available and can be freely modified and re-distributed."
cross-lingual summarization is desirable for infor- mation access, however, it is challenging due to the absence of reliable resources.
To address these challenges, we present PMIndi- aSum, massively multilingual and cross-lingual summarization dataset sourced from the Prime Min- ister of India website. This website publishes po- litical news, usually available in various languages covering the same content. In accordance with tablished methods (Napoles al., 2012; Rush al., 2015), we use a news article as the document and its headline as the summary. Beyond monolingual data, as Figure 1 depicts, the website's multilin- gualism enables cross cross-lingual document-summany alignment with a firm confidence in quality.
1 Introduction
The era of deep learning has witnessed great ad- vancements in various natural language processing (NLP) tasks. Yet prevalent solutions usually rely on large datasets, which are limited to high-resource languages. This is particularly pronounced in In- dia, where languages spoken by large population have been historically overlooked in research and under-resourced (Kumar et al., 2022).
In text summarization, where system generates brief description of longer text, the availabil- ity of datasets for Indian languages restricted in terms of both language coverage and size (Wang et al., 2022).2 Moreover, some existing datasets are not easily accessible or have been criticized for their quality (Urlana et al., 2022b). Given the multilingual nature of India, having 122 major lan- guages and 22 official ones, the development of
Our efforts lead to an extensive coverage of 196 language directions from 14 languages across four families, making the corpus the current widest col- lection of Indian language pairs for summarization. There are 76,680 monolingual document head-ihedlee pairs and 620.336 cross-lingual pairs in total. We display language family and code information in Table 1. Of partieular note is the inclusion of Manipuri (Meitei), an often neglected language in present-day datasets. The text in our corpus is in Bengali script. Besides the corpus release, we
"Equal contribution.
'The corpus is released under BY license https:/hhuggnngface
"This paper refers to the languages widely used in India as "languages in India" or "Indian languages", but they are not exclusive to India and are spoken worldwide.
'A governmental website that allows content re- distribution. http://wwwmiiiii.oooonn/nheeeeee

---

Family
Language (language code)
Number of English articles
PMIndia Our craw
Dravidian
Kannada (kn), Malayalam (ml). Tamil (ta), Telugu (te)
1500
Indo-Aryan
Assamese (as), Bengali (bn), Gujarati
1000
1806
(gu), Hindi (hi). Marathi (mr), Odia
904
3533
3500
(or), Punjabi (pa), Urdu (ur)
1004
500
840
Indo-Furopean
Finglish (en)
Tibeto-Buren Baretn
Manipuri (mni)
2015
2017
2019 Year
2021
2023
Table Language family and language code information.
Figure 2: PMIndiaSum acquisition statistics for English, where 56% are from PMIndia and 44% newly crawled.
obtain results in multiple metrics for popular sum- marization paradigms like fine-tuning, two-step catpaariiaaioon-aanlatttii and prompting, to serve as a reference for future rescarch.
2.2 Data processing
Though there exist summarization datasets sup- porting languages in India, some concerns need to be addressed with regard to developing datasets for Indian language summarization. Explicitly, data construction should (1) respect copyright permis- sions and ensure transparency in data processing; (2) carry out multifaceted quality checks; (3) offer a broad range of language directions; (4) provide reasonably sized data to facilitate system building. Our PMIndiaSum takes into consideration all of these matters. Following Gebru et al. (2021)'s ad- vice, we include datasheet in Appendix I.
To create a high-quality dataset, the collected headline-body pairs undergo rulc-based process- ing in the context of monolingual summarization. The cleaned version contains 76,680 monolingual instances for all languages, which is 81.5% of the raw size. We describe the filtering rules, and list them in Appendix A Table 12 detailing the number of invalid samples removed at each step.
Language mismatch Despite confidence in the website's 's efforts towards language correctness, we set our language filtering. We discard entire data instance if either the document or the headline contains text outside of the Unicode range of its designated language.5 We notice that large num- ber of samples removed, especially from bn and en, are code-mixed data.
2 Preparation of PMIndiaSum
2.1 Data acquisition
The Prime Minister of India website posts articles that consist of a headline and news body. These ar- ticles are available in multiple languages, with En- glish being the default option. The HTML structure of each article includes language indicator and a pointer the English version. We gather web- site data for all articles in all available languages, which are sourced from two channels:
Duplication and empty To maintain dataset with unique document--uummary pairs, remove all duplicates. We also eliminate samples that have identical summaries to steer clear of any headline- article mismatch errors from the website. In addi- tion, we take out instances if either their document or summary is empty.
1. We ingest readily crawled HTML data from the PMIndia parallel corpus release (Haddow and Kirefu, 2020), which correspond to articles published between 2014 and 2019.
Prefix To enforce that PMIndiaSum is abstractive
in nature, remove all samples where the sum- mary is repeated as the initial or first few sentences in the document. For cases where the summary appears later, the model still needs to "understand" the document in order to "select" a summary. Here, such extraction becomes an easier (but valid) case of summarization.
2. We newly crawl for more articles up early 2023, using PMIndia crawler."
We design a parser retrieve article bodies and headlines from the HTML. We harvest 94,036 raw article-headliee pairs for all languages, which we preliminarily regard as monolingual document- summary data. Figure 2 outlines the origins of English articles across different years.
Length We filter out data pairs where the docu- ment contains less than two sentences or the sum-
Defined under the South and Central Asia-l languages on wwwp:/icnondooooo//sioo////////o 13.00/
httnss///gthhhhooommmmd////////d

---

bn
gu
hi
or
pa
en
monolingual size
2089
5557
6802
7363
5307
4665
4936
5816
4651
4814 14
6079
6126
4315
8160
doc vocah
73k
130k
125k
94k
166k
217k
143K
147k
98k
80K
150k
202k
61k
68k
sum vocab
6k
14k
10k
12k
12k
13k
16k
8k
token/doc
505
719
405
336
544
688
344
430
sentídoc
24.7
31.2
31.0
27.0
26.0
27.1
30.2
26.7
34.0
22.2
token/sum
11.9
15.4
12.3
10.1
13.5
11.4
11.7
14.1
17.7
13.4
sent/sum
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
compression
91.5
92.1
89.7
91.3
91.00
90.8
90.4
90.9
91.77
90.
90.5
9T.6
91.6
density
6.06
3.91
5.45
8.67
4.86
3.98
7.97
4.50
3.74
8.12
5.35
4.75
8.13
7.31
novelty, 1gram
27.9
30.0
27.4
12.7
28.5
30.7
22.4
29.5
29.2
15.2
24.7
26.9
13.1
22.9
novelty, 2gram
52.5
59.8
53.5
38.7
58.4
59.2
44.9
59.0
60.8
41.3
52.9
55.8
39.7
44.9
novelty, 3gram
63.5
73.3
65.9
54.1
70.9
72.2
59.8
71.1
73.5
57.6
66.1
71.9
56.3
56.9
novelty, 4gram
71.4
81.3
73.6
62.7
78.5
79.9
67.5
78.8
81.4
68.4
74.0
79.7
66.7
64.9
redundancy, Igram
1.4
1.3
1.5
5.5
2.8
2.5
5.1
1.5
1.3
4.9
2.1
2.6
5.7
4.0
redundancy, 2gram
0.1
0.1
0.2
0.3
0.4
0.4
0.8
0.2
0.1
0.9
0.3
0.3
0,4
0.4
Table 2: PMIndiaSum monolingual documenttsurumaryy data analysis.
mary contains less than three tokens, as these pairs may not provide sufficient information for training summarization model and can become outliers that negatively impact system's performance. We used the indic_nlp_libre library for both word and sentence segmentation (Kunchukuttan, 2020).
Novelty To evaluate the originality of summarics in our corpus, we compute the percentage of n- grams that present in summary but not in its corresponding document. We report 1-to-4-gram novelty scores averaged across all samples in each language. Here, unigram novelty equivalent to coverage (Grusky et al., 2018).
2.3 Monolingual statistics
Redundancy Hasan et al. (2021) described re- dundancy as the amount of repeated information in summary. It is calculated as the ratio of repet- itive n-grams all n-grams in the summary. We measure the average redundancy for unigrams and bigrams, where lower is more desirable.
Table presents the demographics of PMIndiaSum for qualitative inspection. We use based mea- sures (Grusky et al., 2018; Narayan et al., 2018). Since many are not well-defined for cross-lingual data, we report the figures for monolingual pairs.
Vocabulary We list the count of unique tokens in documents and summaries for each language, under the same tokenization as earlier.
2.4 Multilingualism and parallelism
On top of monolingual data, PMIndiaSum features massive parallelism because the source website publishes a single article in multiple language ver- sions to cater for its audience. This means that document-headllnng pairs in various languages de- rived from the same article enable cross-lingual summarization. As depicted in Figure 3, most of the articles are available in at least two languages, and 232 articles available in all languages. This allows for the creation of cross-lingual and multi- lingual summarization data pairs.
Length and compression. Our average docu- ment length is 27 sentences with 518 tokens with ur being the longest, and m1 being the shortest, Summaries are on average 12 tokens, with nearly all being single sentence. We then compute the compression ratio, which quantifies how concise summary is given its document D as Ten (1) in (Bommasani and Cardie, 2020). For each lan- guage, we display the average over all samples. A high compression of around 90% implies that the headlines are extremely abstractive and condensed.
len(S)
Technically, every document is paired with sum- maries in other languages from the same article, and vice versa. Matching is done via the default English pointer in HTML. Such multi-way par- allelism results in 14 13 182 cross-lingual pairs in addition to monolingual data. The average data size 5,477 for monolingual and 3,408 for cross-lingual summarization. Appendix B Table 13 details the sizes for 196 language directions.
Density Grusky et al. (2018) introduced ex- set of shared
tractive fragments F(D,S) sequences in document D and summary S. We use their density metric which reflects the degree which the summary can be com- posed of texts from the document: density lent (eu

---

2000
en
te
Number aarticles
109
1073
972
1965
3001
Headlines
47
49
First sentences
28
45
35
500
01
570
4077
7:2
651
550
Table 3: Number of times, out of 50, headlines and first sen- tences considered summary by evaluators.
2
11
13
headline is a summary. We record majority votes for each item in Table 3, which suggests that head- lines are consistently regarded as summary; on the contrary, first sentences may not always func- tion summary, which is also hard to be auto- matically identified. In Appendix D, outline the evaluation protocol and supply few examples of problematic first sentences. Based on the eval- uation outcome, we determine that it is more ap- propriate to use the headlines as summaries our corpus. We argue that having the high parallelism between articles, findings from three languages can generalize to the whole corpus.
Number languages available
Figure 3: Degree article parallelism PMIndiaSum.
2.5
Data split
To prevent test data leakage in multilingual models, where model sees an article in one language and is tested on the same article in a different language, we isolate the 232 articles that are available in all languages for validation and testing. We divide the data equally into validation and test sets, resulting in 116 instances for each language pair in each set. This approach provides consistent validation and test splits for each target summary language, regardless of the language directions. All other data are the training split.
2.6.2 Parallelism
Finally, we assert the validity of cross-lingual align- ments. We measure the degree of parallelism by calculating cosine similarity between neural pre- sentations of texts in two languages (LaBSE, Feng et al., 2022). We compute LaBSE scores between summaries as well as between entire documents, from the same article, for each language pair. The average cross-lingual LaBSE score is 0.86 for sum- maries and 0.88 for documents. These scores indi- cate the high parallelism between summaries and between documents in different languages; they also notably exceed the 0.74 summary-summ summary threshold Bhattacharjee et al. (2023) used to extract CrossSum from XL-Sum. Hence, given mono- lingual document-summaay pairs, our choice of substituting the summary (or the document) with one in another language maintains the integrity of cross-lingual pairs. We enclose all pairwise LaBSE scores in Appendix C Table 14 for reference.
2.6 Quality considerations
We have high confidence in text quality since the corpus crawled from a governmental website and we have specifically designed an HTML parser to remove extraneous elements. In addition, we discuss candidate summaries and data parallelism.
2.6.1 Summary choices
We take Rush et al. (2015)'s approach of treat- ing the headline as article's summary, whereas some works opt for the first sentence instead, e.g. XSum and XL-Sum (Narayan et al., 2018; Hasan et al., 2021). Although the lead sentence can be an overview of an article, this paradigm has received criticism due to potential concerns (Zhao et al., 2020; Urlana al., 2022b):
1. The first sentence can be part of multi-sentence writing, which is difficult to isolate.
3 Benchmark Experiments
2. Further, the second sentence onwards, when employed as document, may not always con- tain all the information in the first sentence. We conduct empirical study on the suitability of headlines versus first sentences as summaries. For each of English, Hindi, and Telugu, we invite three native speakers to perform a manual inspec- tion on 50 random samples. Given an article, we ask if the first sentence a summary and if the
3.1 Task and evaluation
Formally, given a source document D. the process of summarization should produce target summary S with shorter length, yet conveying the most important message in D. We explore three types of models defined by language directions:
1. Monolingual: document Dr and summary St are the same language

---

2. Cross-lingual: document Dr and summary Sy are in different languages L and L'.
data for all language pairs, because the data sizes are of the same magnitude across all directions.
3. Multilinnuall: monolingual and cross-lingual summarization from DI...t to
In cross- lingual settings, is practical to leverage translation system for language conversion. Two common pipelines are 1) summariiz-tthen-traallaae where a document is summarized in its original lan- guage and then the summary is translated into the target language, and 2) translate- censslattthee-smmeee -ummmriumeee which means first translating the document into the target language followed by summarizing it in that language.
cuammariattion-an--taaala
9(12.122....m))) within a single model.
In our context, cross-lingual models summarize from one single language to another. Monolingual and cross- cross-lingual models could be more accurate as they concentrate on language (pair), whereas multilingual models can significantly save storage and computational resources, and likely transfer knowledge across languages.
For evaluation, we report FI scores of ROUGE- 2/L (Lin, 2004), as implemented in XL-Sum, with default language-specific settings such segmen- tation and stemming. 6 We also use BLEU-4 from sacrebleu (Papineni et al., 2002; Post, 2018). We abbreviate these R2, RL, and BL hereafter.
Zero-shot fine-tuning We fine-tune a PLM on all monolingual data only. This serves ablation study which only allows same-language document- headline data, but not cross-lingual. We then per- form cross-lingual summarization on the test set with this model to measure its zero-shot capability.
3.2 Methodology
We intend to provide benchmark for conventional summarization approaches to cover all language scenarios. We introduce the paradigms below and list the language settings tested with each method in Table 4.
Prompting Large language models (LLMs) have demonstrated potential in automatic summarization via prompting (Zhang et al., 2023). On our data, we give a preliminary evaluation of two instruction- finetuned LLMs based on LLaMA-7B (Touvron et al., 2023): Alpaca (Taori et al., 2023) and Vi- cuna (Chiang et al., 2023). We query an LLM with prompt "Article: $(article). Summarize the article above into a headling in $ $(language} language. Summary:", and parse the model's completion as the candidate summary.
Mono
Cross
Multi
Extractive: lead
Extractive: oracle
Summarize-ther ata
Translate teeiaattt-hee--tee then--henneeee summariee Fine-tuning: full Fine-tuning: zero-shot LLM prompting
Other techniques Lastly, we provide a prelim- inary discussion on using adapters and summa- rization models trained on other datasets Ap- pendix F.
Table 4: Summarization approaches and language directions.
Extractive baseline We employ two training- free baselines: 1) selecting the lead sentence, and scoring each sentence the document against the reference and picking the best in an oracle way.
3.3 Systems
Our fine-tuning paradigm is tested on two PLMs: IndicBART (Dabre et al., 2022) and mBART-50 (Tang et al., 2021). 8,9 We follow each PLM's s con- vention to add language identification tokens to inform the PLM of the source and target languages.
Fine-tuning Fine-tuning pre-trained language models (PLMs) has shown promising results in summarization for Indian languages (Taunk and Varma, 2022; Urlana et al., 2022a). The paradigm is to load PLM and further train it for summa- rization with our data. In monolingual and cross- lingual settings, a PLM is only fine-tuned for a single language direction. On the other hand, in the multilingual setting, we simply mix and shuffle
The PLMS support various languages, yet neither supports Manipuri; therefore, ize an embedding entry as the mni language token in both IndicBART and mBART.
randomly initial-
"httss//hugginnffce BetpaibhharaanniBBARRRRRS We the IndicBARTSS variant, which deals each language in its script without the need of mapping to from Devanagari. "hups://uuginggfaee https///wgginggeeee.eieeeeoooo// coffacehook/mbart-bbee -agge-
aun/tre//naster/muliiinnguu _aruge rouge_scoring
10 mnn and ur are not covered by IndicBART, as, kn, mni, or, and pa are not supported by mBART.
'reefs: llcase:mixedlefffoooi hat.:/ooooooooaaaaaooooooeoeeoen Bablmoonth:cxppeersion2...31

---

Lead
Oracle
IndicBART
mBART
R2
RL
BL
R2
RL
BL
R2 (A)
RL (A)
(L)
(2)
(L)
BL)
as
31.7
41.6
18.2
34.5
44.7
23.1
41.7
(15.41)
56.1 (18.51)
35.2 (17.4))
bn
27.4
40.3
13.5
31.0
43.7
18.8
27.8
(10.5t)
46.2 (16.91)
20.5 (10.11)
30.5 (1.31)
48.2 (0.31)
21.5 (1.11)
gu
33.7
45.3
22.7
38.4
50.9
24.6
49.8
(4,4t)
64.6
(6.71)
44,6 (8.01)
50.1 (1.81)
64,8 (0.31)
45.2 (4.01)
hi 44.8
58.2
30.1
49.1
62.5
33.2
53.1
(4.6t)
66.9
(6.3t)
46.0 (7.31)
55.9 (1.41)
69.4 (0.41)
48.6
kn 26.4
37.4
18.1
31.1
42.8
19.7
39.9
(4.31)
56.2
(7.81)
36.3 (7.8))
ml 23.8
37.1
12.7
32.7
45.8
15.4
30.3
(6.31) 47.5
(7.51)
15.3 (5.11) (3.1
(1.8))
14.6 (1.24)
mni 38.3
50.5
26.4
42.0
54.2
26.0
38.7
(0.74) 53.0
(1.21)
32.0 (0.81) (1.5(
56,4 (0.91)
35.0 (1.61)
mr 31.0
45.2
20.8
33.5
48.7
21.8
(10.5))
59.6
(14.41)
39.7 (14.71)
(3.01)
61.1 (3.11)
39.8 (4.01)
or 21.8
37.3
11.2
26.2
42.0
15.3
38.6
(3.01)
55.4
(6.21)
28.7 (5.11)
pa 44.0
57.7
30.7
47.1
60.7
30.3
54.2
(3.41)
68.5
(4.31)
47.5 (4.51)
ta 27.9
41.6
22.9
38.0
52.8
25.5
47.5
(6.5t)
62.2
(5.81)
37.4
(5.81)
63.3 (5.8
37.8 (2.71)
te 31.2
41.0
18.0
34.4
45.2
19.5
16.3
(3.01)
32.7
(3.01)
9.9 (1.91)
(0.40)
(3.4
9.8 (0.8t)
ur 21.6
30.2
18.1
39,9
52.7
24.9
52.6 (4.31)
64.8 (2.61) 44.0 (3.1f)
en 33.3
42.1
18.6
38.7
48.2
22.8
63.4
(5.51) 74.5 (5.31) 53.1 (7.71) 66.9 (2.91)
77.8 (1.01) 57.2 (5.61)
Table 5: Monolingual benchmarks: separate models for each language. Bold indicates the best result; (A) refers the change relative to the corresponding multilingual result in Table 9 Table 10.
R2
RL
due linguistic features, vocabulary, etc, but not the data quality because we also found that Telugu summarization naturally has a lower score than other languages in prior work (Dabre et al., 2022).
IndieBART
63.4
74.5
53.1
fine-tuning
mBART
66.9
77.8
57.2
Alpaca-7B
20.5
35,8
14.6
prompting
Vicuna-7B
30.7
48.1
16.4
The performance of monolingual English sum- marization from prompting is listed in Table 6 along with fine-tuning baselines. Both Vicuna and Alpaca prompting underperform fine-tuning. Also, we discover that although the L.LMs can generate some Indian language texts, the quality subpar, potentially due to the lack of exposure to these languages during pre-training, We hence suggest that our benchmark cannot be trivially solved by LLM prompting because of language inadequacy and domain specificity. Further research is required to overeome the problems.
Table I.IM prompting results English.
In the aunmarizatioo-and--raansaa monolingual summarization is done using our monolingual fine-tuned PLMs introduced above. The translation process is delegated to a public en- gine that supports all the involved languages." detailed in Ap- pendix E. All trainable experiments are conducted three times to obtain the mean and standard devia- tion statistics. We report average scores, and attach standard deviations in Appendix G.
workflow,
Training configurations
3.4.2 Cross-lingual
3.4 Experimental results and discussions
Fine-tuning all 182 cross-lingual directions in- feasible given our resource constraint. Thus, we shortlist language pairs to fulfil as much as pos- sible: 1) high and low data availability, 2) com- binations of language families as in Table 1, and 3) languages supported by both IndicBART and mBART for comparison.
3.4.1 Monolingual
We list monolingual results in Table 5. Comparing the two extractive baselines, the oracle performance is better than using the first sentence, yet both are nowhere near perfect scores. This indicates that the summary information is scattered across doc- ument and the headline is abstractive in nature.
According to results in Table 8, we observe that outperforms fine- tuning, but the gaps are not wide. It is worth noting that the comparison is not strictly fair be- cause the summarization-ant and-translation pipeline an external translation model which presum- ably ingests much more data. Specifically, the order of translation and summarization matters in reghur-resource scenarios: for Hindi and En- glish, translation-sumnn summarization achieves 10 more
summaaizatiion-andtrasssste
Our PLM fine-tuning yields significantly higher numbers than extractions, implying non-trivial headline summarization task. Generally, mBART is ahead of IndicBART, but it supports fewer lan- guages. In terms of automatic metrics, most lan- guages have reasonable performance except for Telugu. The score lags behind perhaps probably 11 "ttpp:://ssmt.i iitt.ac in/translate

---

en-en
hi-hi
te-te
en-hi
hi-en
Mono
Multi
Mono
Multi
Mono
Multi
Cross
Multi
Cross
Multi
IB
mB
IB
mB
IB
mB
IB
mB
IB
mB
IB
mB
IB
mB
IB
mB
IB
mB
IB
mB
Comprchensibility
39
0
4
2
31
Grammar & Fluency
0
0
1
0
1
1
2
Factuality
2
6
6
6
5
22
26
3
7
21
18
6
10
Omission
11
23
12
19
12
18
16
17
11
12
1
15
25
21
7
17
Redundancy
5
2
5
2
13
8
3
9
5
1
3
8
10
7
No error
28
35
30
34
16
20
17
24
31
25
28
31
13
16
6
20
16
17
11
26
Table 7: Error analysis on different models and different language scenarios. IB denotes IndicBART and mB denotes to mBART.
points on all three metrics than summarization- translation. The zero-shot columns reveal that both PLMs are unable to perform cross-lingual sum- marization even though they have been fine-tuned on monolingual data in all languages. This obser- vation indicates the necessity of our true parallel for practical cross- tross-lingual headline summarization between languages in India.
In monolingual cases, the prevalent error is omis- sion for both IndicBART and mBART models. Conversely, have different discoveries in cross- lingual models. Dedicated cross-lingual models for single language pair usually make factual- ity and omission mistakes. Regarding multilin- gual systems, IndicBART suffers from language mismatch badly, whereas mBART is associated with factuality, omission and redundancy. Ap- proximately 53% monolingual summaries and 30% cross-lingual summaries are considered to be cor- rect. Although decent automatic scores previously, manual inspection suggests that there is ample room for model improvement on our dataset.
3.4.3 Multilingual
We fine-tune both PLMs on all data where the lan- guages are supported. Results are documented in Table 9 for IndicBART and Table 10 for mBART. Regarding cross-lingual cases, multilingual In- dicBART seems to only produce sensible numbers for summarization into hi, pa, and en; mBART performs remarkably better than IndicBART.
seen
4.2 Languages and PLMs
comparison with In- dian languages, we see superior summarization scores for English in the monolingual case, prob- ably due to larger training size. Multilingual models, too, performed better for languages that have more monolingual data the target side, for instance, en, hi, and gu.
Resource availability.
Referring to the differences in results (A) in the monolingual result Table multilingual models are inferior to monolingual models for both In- dicBART and mBART. In clear contrast, for 11 out cross-lingen lingull directions Table 8, multi- lingual mBART surpasses separate mBART mod- els fine-tuned for each direction, implying that the availability of data in multiple language pairs helps cross- lingull summarization.
Unseen language. Even though Manipuri was not seen during the pre-training of either PLM, it has results close to other languages in monolin- gual summarization, as well as with multilingual mBART. We hypothesize that this is because the language is written in Bengali script introduced earlier.
4
Analysis and Discussions
4.1
Error analysis
To assess and interpret errors made by different models in different language settings, we con- ducted an error analysis by comparing system out- puts with references. We establish еттоr categories and invite human annotators to label the errors in outputs, with setup and explanations of error categories detailed in Appendix H. We randomly sample 50 outputs from all models that deal with monolingual en, hi, te, as well as en<shh cross-lings lingullingual summarization. Annotation outcome is presented in Table
Language family. agreement with previous work (Hasan et al., 2021), 1), both IndicBART and mBART work better for Indo-Aryan languages compared to other families.
IndieBART versus mBART. In monolingual and cross-lingual settings, IndieBART is only slightly behind mBART, being only one-third of the size of mBART. However, for multilingual summa- rization, mBART is clearly preferred in our context.

---

Table 8: Cross-lingual benchmarks: separate models for each language pair. Bold indicates the best result; (A) refers to the change relative to the corresponding multilingual result in Table 9 or
79
78
10.7
10.3
134
129
86
76
11.7
211
45.4
en
18.6
14.6
24,0
266
298
28.9
228
199
24.5
788
69.2
128
100
16.
168
202
088 20.0
135
17
57.9
BL
0.2
0.1
0.1
0.1
0.1
0.0
0.1
0.1
mBBAKE
RL.
35
1.0
0.0
1.2
0.0
000
0.0
00
1.2
13
166
0.0
0.5
0.
0.
1.0
19
05
1.3
80
08
BL.
28.9
30.3
31.4
29,
30.3
29.1
29.5
30.6
51.5
9.7
9.7
14,8
16.
21.3
154
139
144
1599
29.7
14.7
Zero-slot
R2
0.3
0.0
01
0.0
000
00
0.1
0.1
0.2
0.0
te
80
RL.
59.1
61.
62)
015
6.0
597
59.7
60.5
767
R2
42
38
6,4
69
97
7.1
6.7
677
78
133
67
B1.
0.1
00
0.1
0.0
00
0.0
00
0.1
R2
398
42.9
4399
413
41.
40.7
40.5
41.9
64.0
IndioBARY
0.3
0.3
0.7
1.7
13
14
00
1.4
1.4
0.0
31.6
0.1
1.2
R2
RL.
1.0
0.0
1.0
124 (133g) 01
1.0
0.0
0.0
0.1
00
70 (32.) 00
0.0
0.0
111
8.2 (3.4.) 011
14
8.1
00
00
0.0
53
1.5
280
4.7
75
59
96
7.0
35
56.4
1.0
6.9
BL.
19.3
20.0
20.4
19.0
185
18.4
40.8
213
(2.24) 241 (1.
55.6 (1.3,1 26.8 (0.57)
255 (6.4)
0.5 (40
28 (0.5))
7.2 (19.8))
1.5 (866,)
0.6
24
41
35
47
3.4
22
41.0
4.0
= 169 supported language pairs.
UF
RL.
45.6
45.8
46.5
45.2
48.5
44.2
46.7
62.2
48.0
Table 10: Multilingual benchmark with ART: a single model for 10 = 100 supported language pairs.
B1. (3))
DL
6.4
42
11.2
87
12.9
1311
11.5
430
7.
1.
73
R2
27.9
26.2
28.7
27.3
25.
26.7
26.7
48.5
29.9
29
pa
16.1
81
25.2
20
27.9
0.9
27.9
22,8
64.2
17.0
48
184
11.
33
344
37
36
28
39
32
9.0
32
40
IBART
RL.A3
30.5 (0.85)
20.4 (12 40.7 (200))
18.9 (2.5))
63 (21111)
72 (635) 128 (2.8 315 (3.0
16.5 (2.6g) 35.7 (40.))
294 (041))
22.2 (3521)
14.9 (8.4.)
R2
10.5
1088
51
17.2
141
19
0.6
19.3
15.8
50.8
11.7
30
12.2
50
RL.
28.5
29.5
31.7
29.6
27.
30.9
29.8
334
29.9
30.
54.
1.3
0.7
1.7
1.5
19
0.5
3.0
0.9
1.5
0.1
0.9
R2
12.6
13.11
150
13.6
13.7
14.4
15.6
143
14.5
(8.04)
37.9 (0.22))
48 51 7.6 18.34)
(0-4)
0.(2888))
5.
OF
RL.
7.6
44
9.3
72
107
13.5
55
7.2
11
60
BL.
10.7
11.3
112
10.4
86
10.7
35.1
11.
11.2
119
8212)
5.0
211
5.
39
57
80
31
4,4
011
311
RL.
39.1
39.1
40.4
38.5
39.5
59.2
39.7
39.6
40.6
353
Fine-
BL (A)
117
23. 23.2(16.7 (16.7)
11.6 (11(11277)
19c 1977
114 (347)
25 (121)
123
36 (0.1)
RF
B1.
4.2
1.1
5.5
1.1
61
2.6
13.8
55
11
1.5
82
0.1
45.2
25.0
2.0
88
5.3
1.0
0.1
09
822
1.6
Table 9: Mulrilnguul benehmark with IndieBART: a single model for 13
13
BL
R2
1222
18.
13.5
19.1
19.8
149
18.6
12.3
17.1
18.8
3588
13.7
19
19.1
12.1
134
20.2
33
97
R2
233
33
6.6
28
3W
0.3
324
5.2
5.
2.7
0.1
38
EF
R1
39.0
39.8
40.8
38.4
36.5
5%.0
57.8
39.5
39.0
59.4
RL (4)
56 73444471
51.3 (L313000)
(20.7)
37 (33.5)
18.9 (421)
(1)
30-4 12332)
(3000)
17.9 (467) 348 (5.1))
20911500)
(333)
18.4 (1.71)
499
54
277
39
34
53
388
51
26
322
13
3.
R2
2044
21.3
22.4
19.6
40.2
21.1
20.3
20.7
ani
RL
23.4
11.9
162
15.7
147
17.5
189
141
14.
66
167
BL
83
888
933
89
84
62
9.3
777
10
R2 (A)
33.9
(23J 1
169
2.00
4,7 (4.7)
12,911049)
(7.9
18.1 (1642)
7.8 (0.61)
R2
10.0
1033
51
24
711
62
78
58
6.4
28
7.2
and
R1.
31.7
31.4
318
31.4
31.4
31.7
30.6
333
14.3
0.1
05
0.7
03
10.2
0.0
0.
04
07
0.7
00
022
33.9
m1
29
241
255
6.
34
40.0
0.1
9.5
4.9
34
5.1
0.9
433
R2
154
15.6
15.7
15.9
15.6
16.2
14.5
16.9
11.
158
27.4
52
67
09
77
20.3
74
113
R2
1.6
10
28
17
24.0
00
4.5
22
1.2
21
022
2.1
BL
6.9
6.9
64
15.8
5.1
6.4
6.0
6.6
6.2
6.0
IBAKT
R1.
56.8
57.5
22.
16.4
14.7
312
43.0
30.
316
23.9
24
22
12
42
285
37
0.5
3.0
24
2.0
01
28
AL
R1
31.7
31.1
314
45.6
31.
20
30.2
29.5
31
R2
36.3
39.5
9.1
87
88
33
11.5
17.8
25.1
14.5
14.2
km
6.5
50
14
48.5
14,0
11.7
88
8,2
14
10.4
R2
14.1
14.5
14.0
27.1
14.4
13
12.7
1366
B1.
270
26.5
1244
55
13
711
19.2
11.
9.5
R2
33
24
5.7
35.6
644
64
49
4.7
04
35
BL
22.9
25.2
47.3
24.6
28.3
23.1
23.5
23.0
26.3
Trunskate
IodicAAK
RL.
56.5
36.0
162
194
31.
41.
6
22.0
5.9
32
38.7
6.
6.3
0.3
13.6
8.5
76
7.1
10
65
h
R1
53.9
55.3
09.0
5488
51.
57.0
52.9
55.1
51.
56.9
R2
38.8
180
58
127
24.3
19.3
12.9
hi
12.5
64
60.6
138
17.
0.5
338
178
16.0
16.7
35
15.3
R2
343
36.6
54.5
36.2
38.2
34,0
36.1
34,7
38.1
10.
0.3
18.7
2.0
9.6
3.9
9.5
9.5
13.6
4.0
18.0
R2
9.0
61
39
48.5
9.3
11.4
21.
11.5
11.3
11.0
22
108
BL.
14.6
41.2
15.6
13.9
15.1
126
13.7
15.3
13.5
Summeria -ben-trunntatte
IIBRRT
R1
43
412
148
23.7
22
33.0
31.1
384
3000
35.3
3.4
366
1.9
41
43
49
1.7
49
3.0
R1
44.4
65,1
46.0
4344
45.7
42.7
41.6
464
84
RL
10.7
57.9
95
144
170
195
156
91
1688
37
137
R2
25.
26.7
44
135
74
14.1
144
13.5
22.3
16.2
23,7
R2
233
48.3
248
22.7
21.5
21.2
216
23.8
25.1
R2
7.3
45.4
5.0
7.7
93
11.6
86
51
9.5
1.5
27
M1.
5.4
16.9
211
12.5
44
87
9.5
13.4
2.9
18.2
B1.
20.4
9.7
9.7
93
711
10.2
8.4
9.4
8.8
955
Indie8AKI
BL.
1.2
0.3
0.2
044
044
0.3
0.9
08
0.4
0.4
0.2
0.2
RL
416
39.6
14.6
24.7
22.3
32.2
50.5
284
35.7
bm
99
43
4.0
51
48
43
72
75
4.
4.9
19
50
6m
RE
47.9
35.1
33.9
34.7
30.3
34.5
350
32.9
32.8
34,3
R2
249
24.1
444
14.
75
13,9
134
14.3
22.
184
24.0
3.0
1.3
0.9
1.6
1.3
0.5
2.5
28
1.7
1.7
0.5
1.1
R2
20.2
16.4
1399
15.4
Ar 15.6
15.2
to 15.1
14.5
15.7
hi-en
en-hi
to-gu
al-ani
ani
nr-bn
bn-er
te-ta
tarte
ami-on
en-ani
DC
17.7
0.2
0.7
0.4
0.5
0.6
0.3
0.3
0.0
0.2
bn
m1
Kni
en
35
RL
37.6
25
5.5
455
4.7
52
2.9
2.6
10
2.1
26.3
04
16
1.9
0.7
0.8
0.2
0.6
Table 10.
35
EU
kn
m1
pa
on

---

5 Related Work
5.2 Coverage for languages in India
The current availability of Indian language sum- mariztaion data is limited. TeSum is monolingual, ILSUM covers three Indian languages, XL-Sum supports eight, and M3LS covers ten. Covering the largest number of Indian languages are Mas- siveSumm and Värta, but these works release URLs and processing scripts instead of the data; copyright holders' conditions and licenses remain unknown. Such practices transfer the risk and responsibility to dataset users. Our data can be freely used, modi- fied, and re-distributed.
Document-summary pairs are the key components of summarization dataset. A typical acquisition workflow is to utilize publicly available articles, given their high availability. To form a document- summary pair, an article can be paired with either human-written written summaries, or accompanying texts such as the headline or the first sentence.
human-
5.1 Data acquisition
Earlier works like the CNN/DM corpus utilized En- glish news articles as documents and highlights as summaries (Hermann et al., 2015; Nallapati et al., 2016), followed by Scialom et al. (2020) to con- struct MLSUM in five Iado-Europent languages. Hasan al. (2021) created XL-Sum with BBC ar- ticles' first sentence as a summary and the rest as a document. Varab and Schluter (2021) developed MassiveSumm by scraping a wide range of web- sites. The ILSUM dataset crawled several leading newspapers in India (Satapara et al., 2022), and TeSum used Telugu news with summaries from hu- man annotators (Urlana et al., 2022b). Verma et al. (2023) built multilingual multi-modal dataset M3LS from the BBC website. Recently, a con- current work sourced data from news aggregator to form a multilingual headline dataset in Indic languages (Värta, Aralikatte et al., 2023).
Moreover, the above multilingual datasets not be used for cross-lingual research. Existing data for summarization between Indian languages are scarce. WikiLingua supports one Indian lan- guage, and to the best of our knowledge, before ours, CrossSum is the largest which contains 56 language pairs from pairing up 8 languages in XL- Sum. In comparison, even excluding English, our PMIndiaSum contains 13 Indian languages and 156 cross-lingual pairs. In a zero-shot fashion, one can train on multilingual datasets for penss-lingual ca- pability. Nonetheless, the performance may be in- ferior as we have shown. More importantly, mono- lingual and multilingual datasets cannot provide a benchmark to evaluate cross-lingun research. Our work fills the gap by providing both training data and trustworthy test suite for large number of Indian language directions.
Exploiting public resources usually results in same-language document-summan pairs as intro- duced above. Zhu al. (2019) translated sum- maries monolingual datasets into another lan- guage while ensuring that the round-trip trip translation of the summary did not deviate from the original one. The CrossSum corpus (Bhattacharjee et al., 2023) was derived from XL-Sum by thresholding similarity between cross-lingual summaries, and pairing summary with the document of paral- lel summary. The naturally arising cross- cross-lingual datasets include WikiLingua which aligned sum- maries and documents using image pivots (Ladhak et al., 2020) and EUR-Lex-Sune from a platform hosting legal documents (Aumiller et al., 2022).
Conclusion and Future Work
We have presented PMIndiaSum, a headline sum- marization corpus focused on 14 languages in India, supporting monolingual, cross-lingual, and multi- lingual summarization in 196 directions. We out- lined the processing steps, explained data statis- tics, discussed quality considerations, and made the dataset publicly accessible. We also published benchmarks for extractive baselines, summarize- and-translate and fine-tuning, involving two pre- trained language models. Experiment results em- phasize the value of our dataset for summarization between languages in India.
Our PMIndiaSum falls in the category of using public headcle- pairs. The data itself is mas- sively cross-lingual due to the multilingualism of the published news. We greatly benefit from the materials released by the PMIndia parallel corpus (Haddow and Kirefu, 2020), which is a machine translation dataset between English and Indian lan- guages, extracted from the same website.
A natural extension is to continuously integrate new articles from the source website to update the dataset. However, to ensure that experimental com- parisons remain fair, one needs maintain consis- tent data sizes in model training. Another direction is to ingest more websites to expand the size, do- main coverage, language availability, and modality.

---

Limitations
Dennis Aumiller, Ashish Chouhan, and Michael Gertz. 2022. EUR-lex-sum. A multi- and tross-linguet dataset for long-form summarization in the legal do- main. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing.
Our data sources are scraped from a government website, leading to a domain bias towards political news, and a style bias towards hcadline-lin like sum- maries. Also, we selected the articles available in all languages to be validation and testing instances, to prevent information leakage. While we believe this to be a sensible decision, those articles might carry distributional drift, such being more im- portant for all readers or easier to translate.
Abhik Bhattacharjee, Tahmid Hasan, Wasi Uddin Ah- mad, Yuan-Fang Li, Yong-Bin Kang, and Rifat Shahriyar: 2023. CrossSum: Beyond English-centrie cross-lingual summarization for 1,500+ language pairs. In Proceedings of the 61st Anmual Meeting of the Association for Computational Linguistics.
Rishi Bommasani and Claire Cardie. 2020. Intrinsic
evaluation of summarization datasets. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing.
Ethics Statement
We place trust in the Indian government website to eliminate inappropriate content, but we could not inspect each data instance ourselves. On the other hand, we note that the values promoted by governmental agency might not align with every potential user of the data. Also, the provision of data in widely spoken languages might further edge out lesser-used languages in research.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. 2023. Vieuna: An open- source chathot impressing GPT-4 with 90%* Chat- GPT quality. Imsys.org.
impressin
Elizabeth Clark, Shruti Rijhwani, Sebastian Gehrmann, Joshua Maynez, Roee Aharoni, Vitaly Nikolaev, Thibault Sellam, Aditya Siddhant. Dipanjan Das, and Ankur P Parikh. 2023. SEAHORSE: A multilingual. multifaceted dataset for summarization evaluation. arXiv preprint.
Acknowledgements
We thank Dennis Aumiller and the anonymous reviewers for giving feedback. We are grateful to Pruthwik Mishra, Lokesh Madasu, Kalahasti Ganesh Srivatsa, and Nikhilesh Bhatnagar for help- ing with human evaluation. We also thank Marcio Fonseca for providing the code for LLM prompt- ing, and LTRC, IIT Hyderabad for providing the translation engine for our summarization and- translation experiments. Our work would not be possible if the Prime Minister's ' Office of the Gov- ermment of India or the authors of the PMIndia parallel corpus did not allow the re-distribution of their materials.
Raj Dabre, Himani Shrotriya, Anoop Kunchukuttan, Ratish Puduppully, Mitesh Khapra, and Pratyush Ku- mar. 2022. IndicBART: pre-trained model for indic natural language generation. In Findings of the Asso- ciation for Computational Linguistics: ACI. 2022.
Jacob Devlin, Ming-Wes Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
Fangxiaoyu Feng, Yinfei Yang. Daniel Cer, Naveen Ari- vazhagan, and Wei Wang. 2022. Language-agnoss BERT sentence embedding. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics.
Pinzhen Chen and Barry Haddow are funded by UK Research and Innovation (UKRI) under the UK government's Horizon Europe funding guar- antee Igrant numbers 10052546 and 10039436]. Zheng Zhao is supported by the UKRI Centre for Doctoral Training in Natural Language Processing (EP/S022481/1).
Timnit Gebru, Jamie Morgenstern, Briana Vec- chione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. 2021. Datasheets for datasets. Commun. ACM, 64(12).
Max Grusky, Mor Naaman, and Yoav Artzi 2018. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies.
Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies.
References
Rahul Aralikatte, Ziling Cheng, Sumanth Doddapaneni. and Jackie Chi Kit Cheung. 2023. Varta: A large- scale meadline-generation dataset for Indic languages. In Findings of the Association for Computational Linguistics: ACL 2023.
Barry Haddow and Faheem Kirefu. 2020. PMIndia A collection of parallel corpora of languages of India. arXiv preprint.

---

Tahmid Hasan, Abhik Bhattacharjee, Md. Saiful Is- lam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bi Bin Kang, M. Sohel Rahman, and Rifat Shahriyar. 2021. XL. sum: Large-scale multilingual abstractive summariza- tion for 44 languages. In Findings of the Association for Computational Linguisties: ACL-IICNLP 2021.
Kishore Papineni. Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics.
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefen- stette, Lasse Espeholt, Will Kay, Mustafa Suleyman. and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems.
Alexander M. Rush, Sumit Chopra, and Jason Weston, 2015. A neural attention model for abstractive sen- tence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Lan- guage Processing.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter- Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning.
Shrey Satapara, Bhavan Modha, Sandip Modha, and Parth Mehta. 2022. FIRE 2022 ILSUM track: In- dian language summarization. 14th Annual Meeting of the Forum for Information Retrieval Evaluation.
Proceedings of the
Terry K Koo and Mae Y Li. 2016. A guideline of selecting and reporting intraclass correlation coeffi- cients for reliability research. Journal of chiropractic medicine. 15())
Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, and Jacopo Staiano. 2020. MLSUM: The multilingual summarization corpus. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing.
Aman Kumar, Himani Shrotriya, Prachi Sahu, Amogh Mishra, Raj Dabre, Ratish Puduppully, Anoop Kunchukuttan, Mitesh M. Khapra, and Pratyush Ku- mar. 2022. IndicNLG benchmark: Multilingual datasets for diverse NLG tasks in Indic languages. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing.
Patrick E Shrout and Joseph L Fleiss. 1979, Intraclass correlations: uses in assessing rater reliability. Psy- chological bulletin, 86(2).
Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Na- man Goyal, Vishrav Chaudhary, Jiatao Gu, and An- gela Fan. 2021. Multilingual translation from denois- ing pre-training. In Findings of the Association for Computational Linguisties: ACL-IJCNLP 2021.
Anoop Kunchukuttan. 2020. The IndicNLP Library. github.com.
Faisal Ladhak, Esin Durmus, Claire Cardie, and Kath- leen McKeown. 2020. WikiLingua: new bench- mark dataset for cross- lonssllnngua abstractive summariza- tion. In Findings of the Association for Computa- tional Linguistics: EMNLP 2020.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford Al- paca: An forructtooo-ollowing LLaMA model. github.com.
Chin-Yev Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out.
Dhaval Taunk and Vasudeva Varma. 2022. Summariz- ing indian languages using multilingual transformers based models. In Working Notes of FIRE 2022 Forum for Information Retrieval Evaluation.
Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Çaglar Gülçchre, and Bing Xiang, 2016. Abstrac- tive text summarization using
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. LLaMA: Open and efficient foun- dation language models. arXiv preprint.
Proceedings of the 20th SIGNLL Conference on Computational Natural Lan- guage Learning.
RNNs and beyond.
Courtney Napoles, Matthew Gormley, and Benjamin Van Durme, 2012. Annotated Gigaword. In Proceed- ings of the Joint Workshop on Automatic Knowledge Base Construction and Web-scale Knowledge Extruc- tion.
Ashok Urlana, Sahil Manoj Bhatt, Nirmal Surange, and Manish Shrivastava. 2022a. Indian language sum- marization using pretrained sequence-to-sequence models. In Working Notes of FIRE 2022 Forum for Information Retrieval Evaluation.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don't give me the details, just the summary! topic-aware convolutional neural networks for ex- treme summarization. In Proceedings of the 2018 Conference on Empirical Methods Natural Lan- guage Processing.
Ashok Urlana, Nirmal Surange, Pavan Baswani. Priyanka Ravva, and Manish Shrivastava. 2022b. TeSum: Human-generated abstractive summarization corpus for Telugu. In Proceedings of the Thirteenth Language Rexources and Evaluation Conference.

---

Daniel Varab and Natalie Schluter. 2021. siveSumm: a very large-scale, very multilingual, news summarisation dataset. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.
Mas-
A Data filtering details
We manually inspected the data and discovered that articles labeled as English may contain non-English content, despite having English URLss We used these URLs as references to match cross-lingual pairs, but we didn't t assume the content was always in English. Therefore, we cleaned our data strictly by language. Table 12 shows the number of invalid samples removed at each step. together with the raw size and the cleaned size.
Yash Verma, Anubhay Jangra, Raghvendra Verma, and Sripara Saha. 2023. Large scale multi-lingual multi- modal summarization dataset. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics.
Jiaan Wang, Fandong Meng, Duo Zheng, Yunlong Liang, Zhixu Li, Jianfeng Qu, and Jie Zhou. 2022. Survey on Cross-Lingual Summarization. Transac- tions of the Association for Computational Linguis-
tics, 10:1304 1323.
B Data size breakdown
Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Katbleen McKcown, and Tatsunori B Hashimoto. 2023. Benchmarking large language models for news summarization. arXiv preprint.
Table 13 lists the total size for cach language pair, Rows correspond to the document language and columns correspond to the summary language. Val- idation and test sizes are 1 16 each, and the training split employs the rest.
Zheng Zhao and Pinzhen Chen. 2022. adapt or to fine-tune: A case study on abstractive summariza- tion. In Proceedings of the 21st Chinese National Conference on Computational Linguistics.
C LaBSE similarity breakdown
Zheng Zhao, Shay B. Cohen, and Bonnie Webber. 2020. Reducing quantity hallucinations in abstractive sum- marization. In Findings of the Association for Com- putational Linguistics: EMNLP 2020.
Technically, we obtain LaBSE representations by feeding a summary or document into the model, truncated at the maximum length if necessary. Ta- ble 14 presents the LaBSE scores between cross- lingual documents and between cross-lingual sum- maries. Rows and columns indicate the language pair. The upper right triangle contains scores for summary pairs, and the lower left triangle contains scores for document pairs. Due to the lack of sup- port for Manipuri in multilingual BERT (Devlin et al., 2019), scores involving Manipuri are consid- erably lower than other language combinations.
Junnan Zhu, Qian Wang, Yining Wang, Yu Zhou, Ji- ajun Zhang, Shaonan Wang. and Chengqing Zong. 2019. NCLS: Neural cross-lingual summarization. Proceedings of the 2019 Conference on Empirical Methods in Natural Language P Processing and the 9th International Joint Conference Natural Language Processing.
D Human evaluation details and samples
We carry out a human evaluation to compare head- lines and first sentences as summaries. For each of English, Hindi, and Telugu, we ask three native speakers to consider the accuracy, informativeness, and quality of the potential summaries from 50 samples based on the following guideline.
hi
te
Headlines
0.80
0.52
0.54
First sentences
0.82
0.88
0.67
Table 11: Inter rater reliability (ICC).

---

In this task, you will look at a set of articles and summaries. Each article will be presented with two (2) potential summaries. The aim of the eval- uation is to identify whether the presented text is a summary. Below are the steps you need to take.
English, so our prompting experiments are for English only. In terms of technical details, In- dicBART and mBART-large-50 models have 244M and 610M parameters each. For fine-tuning, we set a training budget of 100 epochs and apply early stopping after three consecutive non-improving cross-entropy during validation. We use an effec- tive batch size of 96 by combining different batch sizes, numbers of GPUs, and gradient accumula- tion. For document- Hocumeent-summrry pairs, we set the max- imum lengths to 1024 and 64 respectively. All other configurations follow the Hugging Face trainer de- fault. 12 We do not perform hyperparameter search. Specifically, our GPUs include Nvidia GelFrce RTX 2080 Ti (1 GBB, GeForce RTX 3090 (24GB), and A100 (80GB). Models mostly converge within 10 epochs.
1. Read the article to understand the context.
2. Read both summaries carefully. Consider the accuracy, informativeness, and quality of the summaries. We provide the definition of these criteria below.
Accuracy: how closely the summary re- flects the factual content of a news ar- ticle. An accurate summary should not include any false information or misrep- resentations of the content.
Informativenes:: how much information the summary provides about the main points of the news article. An informa- tive summary should provide enough de- tail give the reader a good understand- ing of the article's content.
F
Other summarization techniques
F11 Adapters
Quality: the overall standard of the sum- mary. A high-quality summary should be fluent, well-written, free of errors, and easy to read.
Adapters are small trainable modules inserted into giant PLM; during PLM fine-tuning, the en- tire model can be frozen except for the adapters (Houlsby et al., 2019). This paradigm achieves great parameter efficiency as only the adapter weights need to be updated and saved.
3. Provide your binary decision whether each text summary. Base your decision solely on the quality, accuracy, and informativeness of the content, without being influenced by factors such as writing style, personal pref- erences, etc. Please note that an incomplete sentence is acceptable as long as it does not compromise accuracy, informativeness, or quality.
Adapters are relevant summarization research, but we omit experiments on these because Zhao and Chen (2022) show that with a few thousand training data, the performance of adapters is not comparable to fine-tuning PLM. We, nevertheless, encourage future research to try this out.
For each sample document, we record the ma- jority voting (yes/no) from three annotators in Sec- tion 2.6.1 Table 3. Alongside the results, in Ta-
F.2 Models trained on another dataset
An effortless approach apart from extracting the first sentence from document is to use a sum- marization model that has been trained on another dataset. In this regard, we experiment with an mT5- based multilingual summarization model trained on the XL-Sum dataset comprising 44 languages. 13 We tested it our PMIndiaSum monolingual test sets. As Table 15 shows, the performance of the XL-Sum pre-trained model is subpar on our data, potentially due domain mismatch. This implies that our test sets cannot be easily solved by training on other datasets.
ble 11, we compute the inter-rater reliability us- ing Intra-class Correlation Coefficient as per (ICC, Koo and Li, 2016; Shrout and Fleiss, 1979). We observe that the annotators are in moderate to good agreement for all languages and both potential sum- maries.
We also note samples from the dataset show potential problems with using first sentences as summaries in Table 17. Sometimes the first sen- tence is part of a speech expressing gratitude.
E Experimental setup
12 hhte:://wwssiingecccooooossssaor classes trainer hhtp:://ttugiiigeecccoocccccarap
For prompting, our initial finding is that open- source LLMs do not work very well for non-
XL.Sum

---

as
gu
hi
kn
or
pa
ta
ur
en
size
2925
92.0
7393
7642
5743
5106
5432
6034
4760
7315
6314
6656
5622
13844
language mismatch
68
3418
302
80
361
344
271
93
40
2367
68
422
1144
$288
duplicate pairs
4
11
11
7
12
8
11
23
duplicate summaries
8
28
34
110
25
14
54
34
202
empty
0
0
0
0
0
prefixes
37
28
52
92
73
46
133
169
sentences
216
5
19
<3 tokens
0
3
0
final (monolingual)
2089
5557
6802
7363
5307
4665
4936
5816
4651
4814
6079
6126
4315
8160
Table 12: Data preprocessing and tiltering statistics for document-summary pairs in cach language.
Sij
Dr.
as
bn
gu
hi
kn
m1
mni
mr
or
pa
ta
te
ur
en
as
2089
1153
1728
1716
1411
1166
1737
1415
1711
1332
1607
1570
1571
1743
bn 1153
5557
4655
3987
3573
3332
2898
4268
3079
3479
3845
4239
2345
4075
gu 1728
4655
6802
4896
4411
3901
3998
4940
4021
4052
4731
5111
3185
5216
hi
1716
3987
4896
7363
3680
3723
3163
4340
3784
3426
4817
4376
3061
5899
kn 1411
3573
4411
3680
5307
3176
3405
3800
3439
3526
3720
4151
2742
4325
ml 1166
3332
3901
3723
3176
4665
2423
3547
3037
2881
3957
3737
2299
3769
mni 1737
2898
3998
3163
3405
2423
4936
3120
2901
3198
3181
3596
3199
3807
mr 1415
4268
4940
4340
3800
3547
3120
5816
3450
3573
4219
4488
2582
4411
or 1711
3079
4021
3784
3439
3037
2901
3450
4651
3224
3722
3667
2646
3873
pa 1332
3479
4052
3426
3526
2881
3198
3573
3224
4814
3356
3772
2489
3615
ta 1607
3845
4731
4817
3720
3957
3181
4219
3722
3356
6079
4514
3139
4909
te 1570
4239
5111
4376
4151
3737
3596
4488
3667
3772
4514
6126
2847
4955
1571
2345
3185
3061
2742
2299
3199
2582
2646
2489
3139
2847
4315
3415
1743
4075
5216
5899
4325
3769
3807
4411
3873
3615
4909
4955
3415
8160
Table 13: PMIndiaSum data sizes for all language pairs. Each cell corresponds the size of dataset (Dr., Sr.)), with source document D in language Li and target summary S in language L.j.
G Standard deviations
et al., 2023). In analysis, human annotators identify errors in the system outputs by referring the gold output rather than the source document. To ensure accuracy, our annotators are native speakers of the language of the summaries. In the case cross-lingual summaries, they were also bilingual in the source language.
Standard deviations listed in Table 16 for monolingual summarization, Table 19 for cross- lingual summarization, Table 20 for multilingual InRT and Table for multilingual mBART, These correspond to the mean ROUGE and BLEU scores reported in Section 3.4. We do not notice any peculiar numbers.
H Error analysis details
We conduct an error analysis on the outputs of monolingual and multilingual models for English, Hindi, and Telugu. Additionally, analyzed the outputs of cross-lingual and multilingual models for English-to- Hindi and Hindi-to- lindi-to-English summa- rization. We define our error categories in Table 18 which displays each error type with an explanation and an example. As per our definition, single summary output may fall into multiple error cate- gorics.
is important to highlight the distinction be- tween this error analysis and summary quality anal- ysis in previous works (Grusky et al., 2018; Clark

---

Lj
bn
hi
kn
m1
mni
ta
te
0.88
0.86
0.82
0.87
0.87
0.80
0.85
0.90
0.87
0.85
0.88
0.85
0.87
0.89
0.86
0.85
0.87
0.86
0.73
0.87
0.88
0.87
0.85
0.86
0.87
0.86
gu
0.85
0.90
0.92
0.91
0.88
0.70
0.90
0.90
0.88
0.90)
0..9
0.81
0.89
0.93
0.89
0.86
0.65
0.91
0.87
0.91
0.88
0.88
0,90
0.89
0.86
0.91
0.94
0.92
0.89
0.72
0.90
0.89
0.91
0.89
0.91
0.89
0.90
0.88
0.89
0.73
0.89
0.87
0.88
0.88
mni
0.84
0.75
0.68
0.75
0.69
0.74
0.71
0.71
0.74
0.69
0.74
0.90
0.93
0,90
0.71
0.89
0,88
0.89
0.89
0.89
or
0.90
0.91
0.92
0.90
0.92
0.76
0,88
0.89
0.88
0.90
pa 0.87
0,90
0.93
0,91
0.73
0.92
0.90
0.91
0.90
0,85
0.90
0.90
0.91
0.73
0.91
0.89
0.88
0,87
0.91
0.92
0.90
0.94
0,92
0.75
0.91
0.92
0.93
0.92
0.89
0.90
ur 0,84
0.89
0.92
0.90
0,90
0.72
0.90
0.90
0,90
0.91
0.90
en 0.84
0.89
0.90
0.91
0.91
0.90
0.72
0.90
0.90
0.90
0.90
0.90
0.89
Table 14: LaBSE scores between parallel documents (D ts, Dzy bottom left) and parallel summaries (S1x, S2,, upper right).
R2
BL
bn
16.9
30.0
9.8
gu
21.2
33.3
15.0
23.4
40.1
16.4
21.1
36.2
14.8
pa
29.7
45.1
23.2
19.7
35.6
12.8
ur
26.3
40.4
20.1
en
11.2
24.0
Table Results multilingual model trained XI.-Sum and tested on our PMIndiaSum.
IndicBART
mBART
R2
RL
BL
R2
RL
BL
as
0.6
0.5
0.6
bn
1.0
1.0
1.0
0.2
0.3
0.7
gu
1.5
1.4
0.1
1.3
1.5
1.6
hi
0.8
0.8
0.6
0.5
0.4
1.8
kn
0.6
0.8
0.2
ml
0.8
0.6
0.3
1.6
1.4
1.1
ani
0.3
0.7
0.5
1.9
2.7
1.7
1.4
1.4
2.7
1.3
1.1
1.2
1.5
1.5
pa
0.7
0.6
0.7
ta
0.9
0.5
0.2
0.7
0.4
2.1
0.4
0.0
1.3
2.8
2.1
2.7
en
1.8
2.1
Table 16: Monolingual benchmark: standard deviations.

---

Example 1
document: Your Excellencies President Xi Jinping, President Jacob Zuma, President Michel Temer, President Vladimir Putin, Let begin by sincerely thanking President Xi again for his reception and the excellent organisation of this Summit, Our interaction during the restricted session was constructive, It enriched our mutual understanding and perspectives. Afler more than docade of existence, BRICS has developed a robust framework for cooperation.... headline: PM's Intervention at the Pleniry Session
BRICS Summit, Xiamen,
document: महाнलिमप सायपीि शी दिागरीप सιzपी डेकर SHT सiयfी मiकetl टमर अायपी कशारिाीीर पfिन it सιयyीी oी का नारदर Ho अंर şe रिरर सपोer is शामददर अापोगर db fer gाbby अंब क66- लाहब el हपारी feed सबी il बातनी
साक रहl शTI. अरनन हमसा परयरर शमे अंर होिकीप का मपाद किाी él अपम भंितत म. अगन कF uकp दरा बाद (िरश भहपाा @ [ि ue Hasd बाबा (िकाte पि El
headline: सो सियामन अायीनित नोर (िem सिररर शमीरस के qof अर अपsायs का संयोगन
te document: eopo eocigbee & à aoooo sorsee 25525 aod sog Borsee & Ths wa5 eo goo t Jaabo 2055 &a Jad DO aooo ead soodso woow ei (ane))) de Saaio deo 20 2o
wó and la a $ dewd diwww adobaky os & (oomm) (roo So ar Joos 3ome you Sar3to Sodolom zofi i Nogpa Dol roboo awnkoo, ead är addao oeee Serea, 30320 eamr مر50āر5 o whod....
headline: 2 So soinais & 2017 w weoab 43 abn 93 (05) aed Sabyoioo clvs) Ndd Res Rally 5f (ooos Soole ( 3o0l 3nd Lasoon 2660
Example 2
document: Bharut Mata Ki Jui! Bharat Mata Ki Jui! The Governor of Tripura Shri Satyudev Aryu ji. Tripura's young and energetic Chief Minister Shri Biplab Deb ji, Tripura's Deputy Chief Minister Shri Jishnu Dev Varma ji. my Cabinet collcagues Sister Pratima Bhoumik ji and Shri Jyotiraditya Scindia ji, Ministers in the State Govemment Shri NC Debbarma ji, Shri Ratanial Nath Shri Pranjit Singha Roy ji and Shri Manoj Kanti Deb ji, other public representatives and my dear brothers and sisters who have come in large numbers! Greetings to you all. My best wishes to you all the new year 2022! In the beginning of the yeur itself, Tripuru is receiving three gifts today with the blessings of Mau Tripura Sundari.
en headline: PM's speech at the inauguration of Maharaja Bir Bikram Airport and other projects in Tripura
document: मारत मiबा की दयर मRत मादा की बाe Íिया के पापपपार शी सरादद भnर पी, पही के पaा अीर अपाबाा F[ an ब दब पी, सिपा क अ. मपपा eी Tory दe Tम oी कडीप मयामडयप H. मर सहयपगी बहन पपिास मiाा oीt sf या दि बी, शागी सरकार feat al पमरी द बी, ai सहराल नाय di,, Ai quiaaa लिा रोप wी,, et महाब कifà दब dî, अहय अमपनपीयीााग अरर fagाe सशाा il qeR EQ मर पiर माडप बोर कहना शाार t ममाशकरर शाका Φ a हपार बाड बपर अमन अमन सााा पाला कामाा पा बाशी कंबारीी बा काटा पाकरर भM साब की शहकाा मे ही. Flय का Hi Fay पद के भाीतत स अहग बमन उपहर मिल el
hi headline: मपय e महाागा बोर प दबाड ar अंर अहय पी के अqयालन के अहसरर पर पयानमी का माया
te document: 205 Are é a 20o8 Srer a BBo K3810 & Seads eog mós 10go and #5300 Soarsroeee & la Daes des & (ao as Soor la aed 25 Jot mé, is 3y0 3a Scbee 808 (583" arr of Sooat la a (oodds sogai a oj lagbe & aoojeee 2.5
awor af & des end and at @ (ooze doar ond a Siaas ie Sus soo ae a agd (32 deve, 2e Soaye Sow & (abaa and modsmee do eodds hersose en 2022 Bo 5ose jog Soddjor do woddd n bersos ou! 2033,00 (r0g00 J. ár 183 Koodd a @gó éi per dire STbeee eodisooood....
te headline; G906' 3o30 25 Dis (armoaa agd pas (200 2oddzom Sasole (oooño 3:00
Table 17: Two examples of PMIndiaSum eesument-headlinng pairs in English, Hindi, and Telugu. The documents' first sentences could be part of a speech, which poses problem when used as a summary, whereas the headline is a more appropriate summary.

---

Error type
Error description
Comprehensibility
The output cannot be understood:
wrong language, empty string, etc.
E.g.
Reference:
Cabinet approves creation of one post each of Vice- the-Chairperson and Member in the National
Output:
Cabinet approves creation of two-on-one
of the National Cleaning
Employees Commission
Grammar & fluency
The output has grammatical errors or is unnatural and awkward.
E.g.
Reference:
PM to dedicate naval submarine INS Kalvari to the nation tomorrow
Output:
PM to dedicate Navy Navy Navy Navy Navy Navy Navy Navy Navy Navy
Factuality
The output contains incorrect details as per the reference: named entity, number, etc.
E.g.
Reference:
Cabinet apprised of two Bilateral MoUs hetween India and Cuha, and India and Korea in the area of Biotechnology
Output:
Cabinet apprised of the MoU between India and Mongolia on cooperation
in the field of Biotechnology
Omission
The output does not cover some information the reference. E.g.
Reference:
PM to hold 'Samvad' with Beneficiaries of Pradhan Mantri Bhartiya Janaushadhi Pariyojna and affordable cardiac stents and knee implants on June
Output:
PM to interact with beneficiaries of Pradhan Mantri Bhartiya Janaushadhi Pariyojna on 7th June
Redundancy
The output consists of repeated E.g.
much information compared the reference. PM to confer Awards Excellence Public Administration and address
Reference:
Civil Servants tomorrow
Output:
PM to confer Awards Excellence Public Administration effec- tive implementation of identified Priority Programs and Innovation tiitesiimleemenning units other Central/State organisations Vigyan Bhawan tomorrow
No
The output conveys the same information the reference.
Table 18: Error categories defined in analysis.

---

D1.
0.3
0.3
0.7
0.4
0.7
0.4
0.3
1.2
0.4
2.0
0.9
0.4
1.5
RL
0.4
2.1
122
25
10
0.5
3.8
1.6
R2
0.7
0-4
1.2
0.7
14
0.4
08
16
02
27
10
10
0.3
0.3
0.3
0.6
0.
0.1
0.3
0,1
0.1
0.6
1.1
08
1.3
06
2.0
1.5
088
14
04
0.5
0.4
BL
0.8
23
2.
26
11
0.7
0.4
1.4
25
R2
0.7
0.6
0.9
04
1.0
0.
05
0.6
022
0.3
04
0.2
0.2
ML
1.1
1.3
3.0
2.2
0.6
1.7
1.2
1.1
1.3
BL
0.2
0.3
0.2
0.1
0.0
0.3
0.3
0.2
1.7
24
3.6
29
09
2.1
2.
15
2.3
IBRART
BL
0.1
0.0
0.0
00
0.0
00
00
0.0
RL.
0.6
1.5
0.9
09
1.0
0.8
044
044
05
0.2
16
0.8
21
11
0.8
10
111
03
0.7
06
0.4
4.3
0.2
0.1
00
00
00
2.7
R2
1.0
0.8
06
04
01
06
0.2
10
06
0.6
1.0
1.0
0,7
1.1
0.8
1.1
0.4
R2
0.4
0,0
00
0.0
0.0
0.0
0,0
0.0
0.0
00
0.4
0.0
BL.
0.3
088
0.3
111
088
0.0
1.0
0.7
0.3
09
0.7
11
05
1.0
10
0.0
000
00
00
00
00
00
RL.
0.7
1.3
0.4
25
1.5
0.5
21
08
0.6
10
08
B1.
0.7
1.2
0.5
8.5
0.7
6.3
10
0.1
0.1
0.0
0.0
0.0
0.
0.0
0.0
00
0.0
0.0
R2
1.2
0.3
18
0.1
08
055
06
Table 20: Multilingual benehmark with IndicBARTI standard deviations.
NL.
1.9
1.6
088
0,8
04
18
Tahle 21: Multilingual benchmark with mBART. BA:T: standard deviations.
R2
0.0
0.0
0.0
0.
00
0.0
00
0.0
0.0
0.
0.0
BE
0.3
0.3
0.6
0.3
04
0.3
0.9
0.2
01
0.0
R2
188
1.9
1.8
08
0.3
188
5.9
19
02
03
1.2
23
4.3
Table 19: Cross lingual benchmarks: standard deviations.
RL
0.7
1.3
25
0.
1.5
16
0.7
08
0.
1.0
24
1.2
14
1.8
1,4
0.4
RL
6.0
1.8
1.1
0.2
1.5
311
1.9
24
24
94
3.5
R2
0.7
0.7
0.9
17
03
12
10
15
06
0.2
05
2.0
1.1
08
06
10
1.8
0.5
0.2
08
1.
17
6,0
BL
0.1
0.6
044
0.6
04
0J
02
0.5
0.
08
0.5
R2
1.1
1.1
2.7
1.3
1.2
1.0
1.8
0.1
1.0
0.7
0.4
02
0.3
0.3
01
02
00
00
0.4
02
RL
0.5
0.6
0.6
18
05
09
0.8
04
0.7
BL
1.3
1.7
1.5
044
0.7
1.0
RL
0.2
0.2
0.5
022
03
0.2
01
01
0.5
0.3
R2
02
0S
03
13
04
0.3
12
07
08
04
0.2
06
KL.
0.7
0,8
19
144
11
0.5
06
R2
0.3
0.3
0.3
0.2
02
0.5
BL
0.7
0.2
0.5
06
07
05
06
0.4
07
0.1
KC
1.6
1.4
24
1.5
1.8
1.3
1.4
1.2
1.2
BL
1.2
1.2
03
0.3
0,5
01
0.7
0.6
0.6
0.3
0.9
1.0
RL
0.2
1.2
04
0.8
0.2
1.3
1.0
0.8
08
0.5
ML
02
0.3
1.3
0.3
0.7
1.0
11
0.3
1.0
19
02
0.
18
18
1.2
28
R2
04
06
01
06
02
06
07
04
0.5
09
02
0.5
0.7
0.7
2.0
0.1
06
0.4
06
0.
0.5
R2
0.9
1.3
0.9
0.1
0.4
1.1
11
1.2
1.5
1.7
0.0
0.4
0
0.4
0.
0.
06
0.2
0.3
0.0
0.
15
0.7
0.6
05
0.6
07
0.2
Indi:BART
BL
0.7
0.9
02
0.3
0.3
0.1
05
0.5
01
0.2
RL
1.1
0.3
0.3
12
0,8
0.1
21
03
0.2
0.7
0.4
0.6
1.1
0.6
1.2
2.1
0.9
0.6
1.2
0.6
0.7
0.6
0.8
0.4
06
0.1
08
08
1.0
0.1
0.2
R2
05
0J
04
08
00
09
02
01
04
01
25
08
23
22
1.5
1.
166
14
R2
0.5
0.8
0.4
0,1
0.4
0.5
0.3
0.7
0.7
0.9
011
0.3
0.2
0.6
0.7
03
0.7
0.2
06
0.6
0.6
0.1
0.1
0.2
233
1.7
18
2.1
18
1.
14
1.2
mBART
0.5
0.5
0.2
09
111
04
04
0.3
kn
RL.
0.5
15
03
23
10
06
12
0.9
022
0.
0.7
0:4
1.3
0.5
0.5
0.9
0.4
0.9
0.9
1.0
0.4
0.5
1.0
20
1.5
13
13
04
0.6
0.6
R2
0.6
0.7
1.7
08
0.5
1.0
0.7
0.8
0.4
hi
RL
04
0.1
0.6
10
08
1.0
R2
0.5
0.3
0,3
0.6
15
1.5
08
088
0.2
BL
0.6
1.2
1.1
0.5
0.5
0.2
0.2
21
0.6
0.2
0.9
Summar www.nrrritoonnn-
R2
06
1.6
03
077
L.1
10
10
0.8
0.4
0.5
00
0.3
06
0.4
03
03
0.3
hi
RL
03
122
16
07
17
04
16
43
22
0.5
IndicBARRI
HL.
14
0.5
1.3
L.
12
25
16
1.5
RJ.
0.5
0.8
04
0.2
0.5
06
0.4
0.2
0.2
ta-te 0.
0.6
088
R2
0.3
08
1.3
0:4
0.9
0.3
111
31
16
0.3
1.4
08
1.2
0.5
11
06
0.7
0.4
0.4
0.5
022
ar-bn 0.7
bn 02
te 03
mi 0.5
BL
04
0.5
0.3
0.6
0.1
0.3
04
0.3
0,8
0.1
0.3
R2
044
09
06
0.3
12
1.5
0.7
08
1.4
hi-en
en
gi-te
te-gu
enrmni
RL
1.7
0.7
044
1.5
0,3
1.2
1.3
14
0.5
ani
BL
12
0.6
0.5
0.4
09
0.1
R2
1.3
0:7
0.4
04
0.9
0.7
0.2
0.
0.6
0.9
1.1
0.1
18
1.5
088
0.5
15
1.
1.5
13
1.7
BL
03
0.1
00
00
00
011
00
00
2.0
1.3
1.1
0.5
EF 1.5
1.6
te 1.6
UF 1.2
1.9
bn
RL.
0.5
0.9
0.5
06
0,3
1.2
0.1
1.4
0.3
0.6
0.
0,1
bn
DU
t3
R2
06
0.3
02
03
00
06
02
0.2
02
08
0.4
03
0.1
04
03
00
02
03
00
RE
0.9
1.2
0.1
1.1
05
0.7
011
044
04
0.4
0.5
0.2
01.
bn 016
0.1
06
03
011
oF 02
03
ra 02
to 011
hi
km
sni
en

---

I Datasheet for PMIndiaSum
of instances from a larger set?
A: The dataset consists of all instances derived from the raw data we gathered and processed.
L1 Motivation
Q:
For what purpose was the dataset created? (Was there specific task in mind? Was there spe- cific gap that needed be filled? Please provide description.)
Q: any information missing from individ- ual instances? If so, please provide a description. explaining why this information is missing (e.g., because it was unavailable). This does not include intentionally removed information, but might include, e.g., redacted text. A: No.
A: The dataset was developed to facilitate research on monolingual, cross- cross-lingual, and multilingual headline summarization, specifically for Indian languages. Given the nature of multilingualism in India, cross-lingual summarization can greatly case information access, but there is currently little availability in such datasets that can be used for researching the topic. Refer Section and Section for more details.
Q: Are relationships between individual in- stances made explicit (e.g., users' movie ratings, social network links)? If so, please describe how these relationships are made explicit.)
A: Yes. Data instances are mostly independent of each other. Data instances of document-headline pairs in different languages from the same article convey the same content in the original news articles.
Q: Who created the dataset (e.g.. which team, research group) and on behalf of which entity (e.g., company, institution, organization)? A: Researchers from IIIT Hyderabad and the University of Edinburgh created the dataset.
Q: Are there recommended data splits (e.g., training, developmen//validttion, testing)? If so, please provide description of these splits, explaining the rationale behind them.
Q: Who funded the creation of the dataset? A: Please refer to the author list for the people who created this dataset; the funding sources for authors are stated in the acknowledgement section.
A: Yes. Refer to Section 2.5 for explanation. The split information is in the data file itself. We also provide code zip which has the script create data splits linked in Footnote 1.
Q: Any other comments?
A: No.
1.2
Composition
Q: What do the instances that comprise the dataset represent (e.g., documents, photos, people, coun- tries)? (Are there multiple types of instances (e.g., movies, users, and ratings; people and interactions between them; nodes and edges)? Please provide a description.)
Q: Are there any errors, sources of noise, or redundancies in the dataset?
A: Our expectation is that the articles are pro- fessionally written with no errors, and we take measures to ensure that there are no redundancies by removing duplicated data. However, it is possible that extraneous HTML elements may introduce errors that remain unfiltered from the raw crawled data. It is not feasible to manually inspect all data instances or automatically identify this type of noise.
A: Each instance in the dataset contains news article and corresponding headline either in the same or different languages. Refer to Table 17 for data samples.
Q: How many instances are there in total (of each type, if appropriate)?
Q: Is the dataset self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)? (If it links to relies on external resources, a) are there guarantees that they will exist, and remain constant, over time; b) are there official archival versions of the complete dataset (i.c., including the external resources as they existed at the time the dataset was
A: The dataset consists of 76,680 monolingual and ciiunent-headlin pairs.
620,336 cross-lingual
Please refer to Table 13 for individual language pair counts.
Q: Does the dataset contain all possible in- stances or a sample (not necessarily random)

---

created); <) are there any restrictions (e.g., licenses. fees) associated with any of the external resources that might apply to dataset consumer? Please provide descriptions of all external resources and any restrictions associated with them, as well as links or other access points, as appropriate.)
Q: it possible identify individuals (i.e.. one or more natural persons), either directly indirectly (i.e., in combination with other data) from the dataset? so, please describe how. A: Yes, there are individuals' names present in the data.
A: The dataset is self-contained. The dataset can be downloaded, used, adapted, and re-distributed without restrictions.
Q: Does the dataset contain data that might be considered sensitive in any way (e.g., data that reveals race or ethnic origins, sexual orien- tations, religious beliefs, political opinions or union memberships, or locations; financial or health data; biometric genetic data; forms of government identification, such social security numbers; criminal history)? If so, please provide description.
Does the dataset contain data that might Q:
be considered confidential (e.g., data that is protected by legal privilege or by doctor- patient confidentiality, data that includes the content of individuals' non-public communications)? If so, please provide a description.
A: No, as all articles in the dataset are publicly available from the PMIndia's website.
A: While it is unlikely that the dataset contains sensitive information that is not already public, there is possibility that certain news articles in the dataset may contain details that could be considered sensitive. For example, news articles may refer to individuals' personal information. However, as our dataset derived from a public governmental news website, any sensitive informa- tion that may be present in the dataset is likely to have already been publicly disclosed.
Q: Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, might otherwise cause anxiety? If so, please describe why.
A: Unlikely, but cannot completely rule this out because the data contains news articles from governmental website.
Q: Any other comments?
Does the datasct relate to people? (If not, you may skip the remaining questions in this section.)
Q:
A: No.
1.3
Collection process
A: Yes, the majority of the articles are about the Prime Minister of India who is public figure. The data also contains news about other real people.
Q: How was the data associated with each instance acquired? (Was the data directly observable (c.g.. raw text, movie ratings), reported by subjects (e.g.. survey responses), indirectly inferred/derived from other data (e.g., part-of-speech tags, model- based guesses for age or language)? If data was reported by subjects or indirectly inferred/derived from other data, was the data allidated/verified) If so, please describe how.)
Q: Does the dataset identify any subpopula- tions (e.g., by age, gender)? If so, please describe how these subpopulations are identified and pro- vide description of their respective distributions within the dataset.
A: While the dataset does not explicitly identify any subpopulations based on factors such as age or gender, it is possible that such information may be mentioned in the news articles themselves, such as when referring to specific individuals. As news articles are likely to include details such as gender, age, occupation, etc., it is possible that these factors may be indirectly associated with the data instances in the dataset. However, we do not have any explicit information on the distributions these subpopulations within the dataset.
A: The data is crawled from the Prime Minister of India website followed by processing. It is observable on the website. The data reported directly by the website.
Q: What mechanisms or procedures were used to collect the data (c.g., hardware apparatus or sensor, manual human curation, software program, software APD)? (How were these mechanisms procedures validated?)
A: The data was scraped from a website using a

---

specifically-design open-source crawler and parser. Please refer Section 2.1 for more details. We have manually inspected few samples to verify the sample quality and parallelism.
A: No. The news articles are public.
Q: Did the individuals in question consent to the collection and use their data? (If so, please describe (or show with screenshots or other information) how consent was requested and provided, and provide link or other access point to, otherwise reproduce, the exact language which the individuals consented.)
Q: If the dataset is sample from a larger set, what was the sampling strategy (e.g.. de- terministic, probabilistic with specific sampling probabilities)?
A: The dataset is not sampled from a larger corpus.
A: No. The news articles are public.
Who was involved in the data collection process (e.g., students, crowd workers, contractors) and how were they compensated (e.g., how much were crowd workers paid)?
Q:
Q: If consent was obtained, were the con- senting individuals provided with mechanism to revoke their consent in the future or for certain uses? (If so, please provide a description, well as link other access point the mechanism (if appropriate).)
A: The data set crawled and processed auto- matically, The scripts to crawl and process data are partly open-source, and partly written by the authors of this paper.
A: N/A.
Q: Has analysis of the potential impact of the dataset and its use on data subjects (e.g., a data protection impact analysis) been conducted? (If so, please provide a description of this analysis, including the outcomes, as well as a link or other access point to any supporting documentation.) A: No.
Q: Over what timeframe lected? (Does this timeframe match the creation timeframe of the data associated with the instances (e.g.. recent crawl of old news articles)? not, please describe the timeframe in which the data associated with the instances was created.)
the data col-
A: The data was crawled in early 2023. The crawled data span articles published between 2014 and early 2023.
Q: Any other comments?
A: No.
1.4 Preprocessing, cleaning, labeling
Q: Were any ethical review processes con- ducted (e.g.. by institutional review board)? If so, please provide description of these review processes, including the outcomes, as well as a link or other access point to any supporting documentation. A: No.
Q: Was any preprcesiingllennnng/beeiig of the data done (e.g., discretization or bucketing, tok- enization, part-of-speech tagging, SIFT feature ex- traction, removal of instances, processing of miss- ing values)? (If so, please provide description. If not, you may skip the remainder of the questions in this section.)
Did you collect the data from the indi- viduals in question directly, or obtain it via third parties or other sources (e.g., websites)?
Q:
A: Yes, detailed in Section 2.2.
Q: Was the "raw" data saved in addition the prppooesselcceeneellbbee data (e.g., support unanticipated future uses)? (If so, please provide a link or other access point to the "raw" data.)
A: The dataset was obtained from a third-party website that publishes data or information of individuals.
Were the individuals in question notified about the data collection? (If so, please describe (or show with screenshots or other information) how notice was provided, and provide link or other access point to, or otherwise reproduce, the exact language of the notification itself.)
Q:
A: The "raw" data is saved but not made public at the moment. We plan to do so shortly.
Q: the software used to preprocess/clean/laell the instances available? (If so, please provide a link or other access point.)

---

A: We uploaded code zip which has these scripts.
of the dataset and ensure that it aligns with ethical and legal standards.
Q: Any other comments?
A: No.
Q: Any other comments?
A: No.
1,5
Uses
16
Distribution
Q: Has the dataset been used for any tasks already?
(If so, please provide description.)
Q: Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset created? (If so, please provide a description.) A: Yes, the data will be free to the public to download, use, modify and re-distribute.
A: The dataset has been used by ourselves to fine-tune existing models to perform the headline summarization task. See Section for more details.
Is there repository that links to any or all papers systems that use the dataset? (If so, please provide link or other access point.) A: No.
Q:
Q: How will the dataset be distributed (e.g., tarball website, API, GitHub)? (Does the dataset have digital object identifier (DOI)?) A: The dataset is currently hosted on Hugging Face as indicated in Footnote 1, as well as on Google Drive as zip file.
Q: What (other) tasks could the dataset be used for?
A: The dataset can be utilized for wide range of NLP tasks concerning languages in India: language modelling, named entity recognition, linguistic analysis, etc.
When will the dataset be distributed? A: The dataset is available now.
Q: Is there anything about the composition of the dataset the way it was collected and preproessealeleenne/labbee that might impact future uses? (For example, there anything that future user might need know avoid uses that could result unfair treatment of individuals or groups (e.g., stereotyping, quality of service issues) or other undesirable harms (e.g., financial harms, legal risks) If so, please provide description. Is there anything future user could do to mitigate these undesirable harms?)
Q: Will the dataset be distributed under copyright or other intellectual property (IP) license, and/or under applicable terms of (ToU)? (If so, please describe this license and/or ToU, and provide a link or other access point to, or otherwise reproduce, any relevant licensing terms or ToU, as well as any fees associated with these restrictions.) A: Yes, the dataset is distributed under the CC BY 4.0 license.
Q: Have any third parties imposed IP-based or other restrictions on the data associated with the instances? (If so, please describe these restrictions, and provide link or other access point to, or otherwise reproduce, any relevant licensing terms, well any fees associated with these restrictions.).
A: Yes, we applied strict language filtering which removed code-mixing instances, however, it could be a common and valid language phenomenon in languages of India.
Q: Are there tasks for which the dataset should not be used? (If so, please provide description.)
A: The original website states that "the material has to be reproduced accurately and not to be used in derogatory manner or in a misleading context. Wherever the material is being published or issued to others, the source must be prominently acknowledged." More information available http://wwwwwmmiiiii..ooo/n//////
A: Our dataset, like any large dataset, has the potential to be used in harmful ways. One example of such harm is the creation of models that generate hate speech fake news using the data. Additionally, previously mentioned, the dataset may contain sensitive information that should not be used for discriminatory or harmful purposes. is important to carefully consider the intended use
Q:
Do any export controls or other regula- tory restrictions apply to the dataset or individual

---

(If so, instances? (If so, please describe these restric- tions, and provide a link or other access point to, or otherwise reproduce, any supporting documentation.) A: Not known.
If so, please describe how. If not, why not? there process for communicaing/listibting these contributions to other users? so, please provide a description.)
A: Yes, they can freely do so by any means as long as they abide by the license.
Q: Any other comments?
A: No.
Q: Any other comments?
A: No.
1.7 Maintenance
Q: Who is supportinghoostin//aaataannngggg the dataset?
A: Authors of this paper.
How can the owner//urrtorr/aanger of the dataset be contacted (e.g., email address)? A: Via email or issues the Hugging Face or GitHub repositories as Footnote 1.
Q:
Q: Is there an erratum? (If so, please pro- vide link or other access point.)
A: No.
Q: Will the dataset be updated (e.g.. to cor- rect labeling errors, add new instances, delete instances)? (If so, please describe how often, by whom, and how updates will be communicated to users (e.g., mailing list, GitHub)?)
A: Currently there is no plan to update the dataset.
Q: If the dataset relates to people, are there applicable limits on the retention of the data asso- ciated with the instances (e.g., were individuals in question told that their data would be retained for fixed period of time and then deletedy? (If so, please describe these limits and explain how they will enforced.)
A: No.
Will older versions of the dataset tinue be supportedhosted/maittaiee please describe how. If not, please deseribe how its obsolescence will be communicated to users.)
Q:
A: There is no older version of the dataset. Its obsolescence will be communicated to users at the venue where it is held.
Q: If others want to extend/augment/buiid on/contribute to the dataset, is there mechanism for them to do so? (If so, please provide a descrip- tion. Will these contributions be validated/verified?


Reviews:
[{"review_id": "mtKV7YzHV2", "reviewer": "Reviewer_hbSQ", "summary": "", "questions": "What is the utility of knowing compression of these document-summary pairs? Given the relatively limited headline size it only really gives us an indicator of the document length", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. Reason to reject: Unclear whether this is the appropriate venue for this paper; something like LREC might be a better fit.**\n\nAuthor response:\n- We are encouraged to submit to this venue as the [EMNLP call for papers](https://2023.emnlp.org/calls/main_conference_papers/) welcomes \"New data resources, particularly for low-resource languages\". We also believe the two tracks \"Resource and Evaluation\" and  \"Summarization\" are appropriate for this work.\n\n- Our work provides a summarization dataset for 196 language pairs. Before this, the majority of the 196 pairs were in a “no-resource” state, despite having a large population of speakers. Without data resources, it would not be possible to test any empirical methods. We hope that you could kindly reconsider our contribution.\n\n**2. Question on compression**\n\nAuthor response:\n\n- Compression is a characteristic of a summarization dataset and is often reported by previous works [1,2,3]. Generally, it highlights the difficulty of content selection and aggregation imposed by the summarization task (the higher the more difficult). As in lines 151-153, the high compression of our dataset implies that the summaries are very abstractive and condensed. \n\n[1] Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategies (Grusky et al., NAACL 2018)\n\n[2] Intrinsic Evaluation of Summarization Datasets (Bommasani & Cardie, EMNLP 2020)\n\n[3] Models and Datasets for Cross-Lingual Summarisation (Perez-Beltrachini & Lapata, EMNLP 2021)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Large multilingual dataset for under-resourced language summarization\n- Thorough set of descriptive statistics for the dataset", "reasons_to_reject": "Unclear whether this is the appropriate venue for this paper; something like LREC might be a better fit.", "questions_to_authors": "What is the utility of knowing compression of these document-summary pairs? Given the relatively limited headline size it only really gives us an indicator of the document length", "paper_topic_and_main_contributions": "This paper introduces PMIndiaSum, which is a cross-lingual headline summarization dataset covering 14 different Indian languages. The dataset was extracted from the Prime Minister of India website and contains over 76k monolingual document-headline pairs and 620k cross-lingual pairs in total. \n\nThe authors present a quantitative analysis of the dataset's qualities, including vocabulary size, compression ratio, summary novelty, and redundancy.\n\nFinally the authors conduct fine-tuning of a summarization model over the dataset in both monolingual and cross-lingual settings and present the results.\n\nThe primary contribution here is the large multilingual corpus, and a set of statistics that may be used as the starting point for future research.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "r7Fjp1Uzrv", "reviewer": "Reviewer_2wEo", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. Reason to reject: Some of the language pairs are covered by other corpora but I do not get a feel for how this would compare in the tasks with different data**\n\nAuthor response:\n\n- Some Indian languages are covered by other datasets, in summarization, mostly monolingual. Ours is the first massively parallel and cross-lingual corpus for 196 language pairs. Our motivation is not to improve model performance on other datasets, but more about facilitating research on non-English-centric multilingual and cross-lingual summarization research, especially for Indian languages with no existing resources before our work (e.g. imagine now one can train a model to directly summarise from Assamese to Manipuri).\n\n**2. Reason to reject: As noted by the authors the use of external translation models makes isolating the impact of this methodology and dataset difficult as that model may be and is likely ingesting additional data for some language pairs**\n\nAuthor response:\n\n- Admittedly comparison is not fair because a translation engine sees more data, but this is to the advantage of “translate-and-summarise”, which is a baseline without using our provided cross-lingual data.\n\n- We would like to clarify that your question is exactly the motivation for our work. Without PMIndiaSum, one can perform cross-lingual summarization **only** by adopting the “translate-and-summarise” approach, thus using more compute and data resources. Our work enables cross-lingual summarization without translation, achieving similar performance. In addition, the dataset could promote future research to better tackle the problem.\n\n- Further to your comment on \"translate-and-summarise\", we note that the translation process is a bottleneck itself, since erroneous translations lead to erroneous summaries. This is especially true for language pairs that do not have good translation engines in place due to a lack of parallel data, e.g. Assamese-Manipuri.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Paper is well written and clear and easy to follow. The methodology is repeatable for researchers wanting to extend this work.\n- The research follows standard experimentation and evaluation so it is comparable with work in the field\n- The corpus itself is of sufficient size and scope to have an impact on this language pairs and the research community\n- Supporting monolingual, cross-lingual, and multi-lingual summarization for under-resourced languages is a good contribution by the authors", "reasons_to_reject": "- Some of the language pairs are covered by other corpora but I do not get a feel for how this would compare in the tasks with different data\n- As noted by the authors the use of external translation models makes isolating the impact of this methodology and dataset difficult as that model may be and is likely ingesting additional data for some language pairs\n ", "questions_to_authors": "", "paper_topic_and_main_contributions": "Paper presents the corpus construction of 14 languages focusing on Indian government articles. The 14 languages lead to 196 language pairs with a focus on summarization. The main contribution of the paper is the corpus itself which is by far the largest corpus for this purpose on languages used in India. ", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "zJxEIkuM9T", "reviewer": "Reviewer_Jf8L", "summary": "", "questions": "Line 78-83: if I understand the data correctly, the default language of each article is English. This means all articles should at least be available in English. However, according to Table 2, English has 8160 articles, which is less than the sum of the number of articles in Figure 2. It would be great if the authors could clarify this to avoid misunderstanding. \n\nLine 122-125: If the authors want to ensure the abstractive nature, why do authors only remove all samples where the summary is repeated as the initial or first few sentences in the document? There can be cases where the summary occurs at the middle or even the end of the document, right?\n\nTable 3 shows that the first sentences can be considered as a summary for more than half of the articles out of 50 random samples. In Lines 122-125, the authors already removed quite a few clear repetitions. Does this suggest that many of the remained articles also have some kind of redundancy (headlines and the first sentences can both be considered as summaries)?\n\nLine 338-340: as no PLM support mni language, does the author also train a tokenizer and subword embeddings on monolingual data for mni and merge with the original model? \n\nLine 342-346: The author uses a public external translator to perform the translation in the summarization-and-translation workflow. I have doubts here. Then the cross-lingual performance would be highly dependent on the quality of the external translator. How does the author ensure that, given many languages are indeed low-resource ones?\n\nTable 4: Oracle performance on **te** language is significantly higher than the other models, does the author have any explanation for that?\n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. Reason to reject: The dataset is domain-specific and biased toward political news. Therefore it might be questionable whether it could be used as a proper dataset if one wants to evaluate the general capability of a PLM.**\n\nAuthor response:\n\n- We would like to clarify that the goal of our work is not to evaluate the capability of a PLM, but to enable research on multilingual and cross-lingual summarization for low-resource Indian languages (pairs). We used the two PLMs because we think they provide a good model initialization. On the other hand, an ideal general domain PLM should handle all domains including ours.\n\n**2. Reason to reject: Although the author conducts many experiments, some analyses are not detailed enough (I guess it is due to the page limit). I would suggest the author go into more detail about the analysis in the camera-ready version.**\n\nAuthor response:\n\n- Indeed we are struggling a bit with the page limit. Since this is a dataset contribution, we tried to split the analysis work on both the dataset **itself** (statistics, quality assurance, human inspection), as well as on experimental results (multi/cross/mono-lingual conditions, different PLMs, model error analysis).\n\n- We plan to expand our discussions on the PLM data conditions. It would be interesting to add analysis on domains, such as in-domain and out-of-domain data experiments, inspired by your comment on domains.\n\n**3. Question: Line 78-83: if I understand the data correctly, the default language of each article is English. This means all articles should at least be available in English. However, according to Table 2, English has 8160 articles, which is less than the sum of the number of articles in Figure 2. It would be great if the authors could clarify this to avoid misunderstanding.**\n\nAuthor response:\n- When constructing the dataset, we carried out a manual inspection and found that articles labeled as English could contain non-English content. Although there is an English URL for each article, **the default article language may not necessarily be English**. Essentially the URLs served as pivots/references for us to match cross-lingual data pairs, but we did not assume the content language. Therefore, we have strictly cleaned our data by language (lines 107-112 and Table 9 in Appendix B).\n\n**4. Questions: Line 122-125: If the authors want to ensure the abstractive nature, why do authors only remove all samples where the summary is repeated as the initial or first few sentences in the document? There can be cases where the summary occurs at the middle or even the end of the document, right?**\n\nAuthor response:\n\n- Initially, we followed previous works [1,2,3] to filter out data pairs where the summary is a prefix of the document. We hypothesize the reason behind this is that it removes the position bias where a model can simply overfit to copy the first sentence. For cases where the summary appears later, the model still needs to “understand” the document in order to “select” a summary. Here, such extraction becomes an easier (but valid) case of summarization.\n\n- Nonetheless, your suggestion is definitely a reasonable practice. We could filter out all data pairs where the summary matches a (or a few) sentences anywhere in the document, and update the data once the work can be de-anonymised.\n\n[1] MassiveSumm: a very large-scale, very multilingual, news summarisation dataset (Varab & Schluter, EMNLP 2021)\n\n[2] TeSum: Human-Generated Abstractive Summarization Corpus for Telugu (Urlana et al., LREC 2022)\n\n[3] LR-Sum: Summarization for Less-Resourced Languages (Palen-Michel & Lignos, Findings 2023)\n\n**5. Question: Table 3 shows that the first sentences can be considered as a summary for more than half of the articles out of 50 random samples. In Lines 122-125, the authors already removed quite a few clear repetitions. Does this suggest that many of the remained articles also have some kind of redundancy (headlines and the first sentences can both be considered as summaries)?**\n\nAuthor response:\n\n- Yes, it is totally possible that in the remaining data pairs, the first sentence in the document is a potential summary (but different from the headline summary). This will not hurt the downstream model because we have removed data pairs where the gold summary is the same as the prefix, so the model will not overfit by merely copying the first sentence.\n\n**6. Question: Line 338-340: as no PLM support mni language, does the author also train a tokenizer and subword embeddings on monolingual data for mni and merge with the original model?**\n\nAuthor response:\n\n- The Manipuri data is written in Bengali script as line 59 footnote 5. It is as released by the Indian government website and we did not make any transliteration. As such, as long as a PLM supports Bengali, it will be able to process our Manipuri data. We think your solution works better if the Manipuri data is written in Meitei script. We will improve the explanation in the experimental setup section.\n\n- As line 340, we randomly initialized a single embedding entry for the special token for mni because this is not included in either PLM’s vocabulary. This token acts as a language indicator and lets the model know the correct language to generate. The token is added to the vocabulary, and its embedding is merged with the PLMs' original embedding matrix.\n\n**7. Question: Line 342-346: The author uses a public external translator to perform the translation in the summarization-and-translation workflow. I have doubts here. Then the cross-lingual performance would be highly dependent on the quality of the external translator. How does the author ensure that, given many languages are indeed low-resource ones?**\n\nAuthor response:\n\n- Yes, the \"translate-and-summarise\" pipelines’ performance depends on the quality of the external translator. The translation engine we use is open-source and supports all the languages of interest. Yet it might be the quality bottleneck. **This is exactly the motivation behind our work**: the provision of cross-lingual data in 196 pairs removes the need for an external translator. This is evident in Table 6 where we compared fine-tuned models against \"translate-and-summarise\" approaches. We found that FT achieved similar scores without the need for an external translator. \n\n- Note that \"translate-and-summarise\" serves as a baseline, because without our work, cross-lingual summarization needs to rely on external translators.\n\n**8. Question: Table 4: Oracle performance on te language is significantly higher than the other models, does the author have any explanation for that?**\n\nAuthor response:\n\n- As seen in Table 4, the oracle scores for Telugu are in line with other languages’ oracle scores, thus it is not an issue with our dataset (e.g. no significant overlap between the reference summary and the document). On the other hand, the PLM fine-tuning scores look low for Telugu, but we think these are in the normal range. Referring to Table 6 in the [indicBART PLM paper](https://aclanthology.org/2022.findings-acl.145.pdf), Telugu summarization naturally has a lower score than other languages [4], perhaps due to its linguistic features, vocabulary, etc.\n\n[4] IndicBART: A Pre-trained Model for Indic Natural Language Generation (Dabre et al., Findings 2022)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper is well-written, well-documented, and easy to follow. \n\n- This paper contributes to the summarization by presenting a new dataset that focuses on Indian languages (including low-resource ones).\n\n- This author conducts extensive experiments and provides several benchmark results under different settings.\n", "reasons_to_reject": "- The dataset is domain-specific and biased toward political news. Therefore it might be questionable whether it could be used as a proper dataset if one wants to evaluate the general capability of a PLM.\n\n- Although the author conducts many experiments, some analyses are not detailed enough (I guess it is due to the page limit). I would suggest the author go into more detail about the analysis in the camera-ready version.\n", "questions_to_authors": "Line 78-83: if I understand the data correctly, the default language of each article is English. This means all articles should at least be available in English. However, according to Table 2, English has 8160 articles, which is less than the sum of the number of articles in Figure 2. It would be great if the authors could clarify this to avoid misunderstanding. \n\nLine 122-125: If the authors want to ensure the abstractive nature, why do authors only remove all samples where the summary is repeated as the initial or first few sentences in the document? There can be cases where the summary occurs at the middle or even the end of the document, right?\n\nTable 3 shows that the first sentences can be considered as a summary for more than half of the articles out of 50 random samples. In Lines 122-125, the authors already removed quite a few clear repetitions. Does this suggest that many of the remained articles also have some kind of redundancy (headlines and the first sentences can both be considered as summaries)?\n\nLine 338-340: as no PLM support mni language, does the author also train a tokenizer and subword embeddings on monolingual data for mni and merge with the original model? \n\nLine 342-346: The author uses a public external translator to perform the translation in the summarization-and-translation workflow. I have doubts here. Then the cross-lingual performance would be highly dependent on the quality of the external translator. How does the author ensure that, given many languages are indeed low-resource ones?\n\nTable 4: Oracle performance on **te** language is significantly higher than the other models, does the author have any explanation for that?\n", "paper_topic_and_main_contributions": "This paper presents PMIndiaSum, a multilingual and cross-lingual summarization dataset that supports 14 languages and 196 language pairs across 4 language families. In addition, the authors provide benchmark results for monolingual, cross-lingual, and multilingual summarization for different paradigms: fine-tuning, prompting, and translate-and-summarize.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this study, the authors proposed multilingual headline summarization datasets. These datasets encompass 196 language pairs across 14 languages. The authors have made these datasets publicly available for the community. Reviewers mentioned that the dataset would be a valuable resource for the community and that the paper is well-written and easy to follow.

Reviewer jf8L expressed concerns about the need for a more detailed analysis of the experiments. These concerns should be addressed to give readers a comprehensive understanding of the experiments. Additional information can be included in the appendix.
Decision: Accept-Findings