Vārta: A Large-Scale Headline-Generation Dataset for Indic Languages
Year: 2023

Abstract:
We present Vārta, a large-scale multilingual dataset for headline generation in Indic languages. This dataset includes 41.8 million news articles in 14 different Indic languages (and English), which come from a variety of high-quality sources. To the best of our knowledge, this is the largest collection of curated articles for Indic languages currently available. We use the data collected in a series of experiments to answer important questions related to Indic NLP and multilinguality research in general. We show that the dataset is challenging even for state-of-the-art abstractive models and that they perform only slightly better than extractive baselines. Owing to its size, we also show that the dataset can be used to pretrain strong language models that outperform competitive baselines in both NLU and NLG benchmarks. The data and models are available at https://github.com/ rahular/varta.

heading: 1 Introduction
text: Headline generation is a special case of abstractive summarization where the goal is to create a brief, often single-sentence ‘summary’ of a news article. Unlike traditional summaries, which are a few sentences long and concisely convey the most important information from an article, a headline typically highlights one key fact from the article that is considered to be the most significant. In recent years, headlines have also been written to be catchy and increase click-through rates. This trend has made the task harder, as the lexical overlap between headlines and articles is low, while their compression ratio is high.
There are several datasets available to train and evaluate headline generation and abstractive summarization models, but they are largely limited to English (Narayan et al., 2018; Nallapati et al.,
∗ Equal contribution. Corresponding author: Rahul Aralikatte (rahul.aralikatte@mila.quebec)
2016; Rush et al., 2015; Völske et al., 2017). Although there have been efforts to create multilingual datasets such as MLSum (Scialom et al., 2020) and XLSum (Hasan et al., 2021), the representation of Indic languages in these datasets is still minimal.1 As a result, it is challenging to create good headline generation or summarization systems for these languages, which are spoken by approximately one in five people worldwide.
Motivation Indic headline generation is particularly interesting because this family of languages presents unique challenges. For example, though most Indic languages are closely related, they use different scripts which makes transfer learning harder. Many of the languages are morphologically rich, making headlines compact and thus harder to predict. With few exceptions, most Indic languages are low-resource and are underrepresented in multilingual models, which makes few-shot learning ineffective. Recent efforts have shown that language family-specific pretraining enables better transfer among languages (Doddapaneni et al., 2022) and that quality data is required in large quantities for good downstream performance on text generation tasks like summarization (Zhang et al., 2020, HugeNews).
Thus, there is a strong need for a multilin-
1The only exception to this is the recently released headline-generation dataset proposed by Kumar et al. (2022a). We will compare this with our dataset in §2.
3468
gual, large-scale, high-quality dataset for Indic languages. To fill this gap, we introduce Vārta, a dataset consisting of more than 41 million articleheadline pairs in 15 languages (14 Indic languages + English). The data is crawled from DailyHunt, a popular news aggregator in India that pulls highquality articles from multiple trusted and reputed news publishers. It also covers a wide range of topics such as politics, science, entertainment, sports, business, etc., which makes the dataset diverse both in terms of domains and writing styles. Table 1 compares Vārta with other multilingual summarization and headline generation datasets.
Contributions The main contribution of this work is the Vārta dataset, which we hope will push the research on headline generation and summarization in Indic languages. We run two sets of experiments: (i) The first set of experiments deals only with the task of headline generation under different training and evaluation settings. By analyzing the results of these experiments, we answer several important research questions related to text generation from an Indic perspective. (ii) Owing to the size and multilingual nature of Vārta, the second set of experiments treats the dataset as a pretraining corpus. We pretrain a BERTstyle encoder-only model and a T5-style encoderdecoder model and show that the models outperform strong baselines on IndicXTREME (Doddapaneni et al., 2022) and IndicNLG (Kumar et al., 2022b) benchmarks, respectively. Finally, we release the data and the pretrained models so that the community can build on top of our research.

heading: 2 Related Work
text: English Data There have been several proposed datasets for the generation of English headlines and other similar tasks. The DUC 2003 and 2004 datasets, which contain 500 and 624 articleheadline pairs from the Associated Press and New York Times, respectively, were among the first to be released. Rush et al. (2015) introduced a dataset based on the Gigaword corpus (Napoles et al., 2012), which pairs the first sentence of a news article with its headline for sentence summarization. This dataset has around 4 million pairs in total. Other notable datasets from the literature include XSUM (Narayan et al., 2018) for 1-2 sentence summaries of BBC articles, the Google dataset for sentence compression (Filippova and Altun, 2013), and the TLDR corpus (Völske et al.,
2017) for summarizing Reddit posts.
Multilingual Data The Columbia Newsblaster (Evans et al., 2004) was one of the first datasets to include multilingual data for summarization. It includes around 500 news articles in English, Russian, and Japanese, with annotations for articles, texts, titles, images, and captions. In recent years, larger datasets with a greater variety of languages have been curated. MLSum (Scialom et al., 2020) comprises 1.5 million pairs of articles and summaries in French, German, Spanish, Russian, and Turkish. XLSum (Hasan et al., 2021) follows the format of XSUM and crawls the BBC websites to obtain 1 million pairs in 44 languages.2 This dataset also includes the headlines of the articles, along with their summaries. Kumar et al. (2022a) propose a headline generation dataset specifically for Indic languages, which includes 1.3 million data points in 11 Indic languages.
Modeling Approaches The task of headline generation has been approached in several ways over the years. An early approach was proposed by Banko et al. (2000), who viewed the task as a machine translation problem. Subsequently, as sequence-to-sequence models became more popular (Rush et al., 2015), many works have used encoder-decoder architectures to tackle this problem (Shen et al., 2017). More recently, pretrained language models such as BART (Lewis et al., 2020) and T5 (Raffel et al., 2022) have been used for headline generation and other related tasks with great success.

heading: 3 Data
text: Vārta contains 41.8 million high-quality news articles in 14 Indic languages and English. With 34.5 million non-English article-headline pairs, it is the largest headline-generation dataset of its kind. In this section, we detail how the dataset is collected along with the various processing steps involved, before moving on to some interesting statistics and analysis of the data. Table 9 in the Appendix showcases some randomly selected articles from Vārta.

heading: 3.1 DailyHunt
text: Source DailyHunt is a popular aggregator platform for news in India.3 It curates articles from
2The BBC publishes articles in 44 languages, nine of which are Indic.
3https://m.dailyhunt.in/
over 1773 publishers in English and 14 Indic languages: Assamese, Bhojpuri, Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Nepali, Oriya, Punjabi, Tamil, Telugu, and Urdu. We collect all our data from this platform and restrict ourselves to articles that were published between January 2010 and August 2022.
Data Collection Since DailyHunt does not have an external facing API, we crawl their website using Scrapy,4 a Python-based scraping tool that collects data efficiently without burdening their servers. To maintain the quality of the collected data, we: (i) discard articles that are less than 50 words long, (ii) discard articles where an image or video is prominently placed (since these articles cannot be understood without the aid of the embedded media), and (iii) discard articles which require us to navigate to the publisher website to get the entire text.
Processing We only retain text from the articles and remove other HTML content like embedded media, and hyperlinks to other websites. We also de-duplicate articles by using the headline as the identifier. We do not use the URLs as identifiers since DailyHunt re-aggregates an article if the original article text is changed by the publisher. We find and remove around 7.2 million such ‘stale’ articles, and keep only the latest versions.5

heading: 3.2 Statistics
text: The language-wise sizes of the processed data are shown in Table 2. To better understand the various properties of the dataset, we also compute the following metrics for each language: (i) the ratio of novel n-grams between the headline and the article, (ii) the average article and headline lengths, and (iii) the compression ratio between the headline and the article. Additionally, we also report the number of distinct words (‘Vocab Size’ in Table 2), and the number of publishers (‘Domain Count’ in Table 2) for each language.
Novel n-grams The ratio of novel n-grams between a headline and its article text is a proxy for the abstractiveness of the headline, with higher values indicating more abstract headlines. Across languages, 29% and 63% of the unigrams and bigrams are unique respectively. In 7 of the 15 lan-
4https://scrapy.org/ 5These duplicates themselves form an interesting dataset
of how news articles are edited over time.
guages, the novel unigram ratio exceeds 1/3 indicating a high degree of abstraction and low lexical overlap.
Article and Headline lengths On average, Vārta articles have 17 sentences, and the headlines have just over one sentence. A typical article sentence contains about 18 words, and a headline sentence contains 11 words. This indicates that the headlines in our dataset are usually 39% smaller than the average sentence in an article. This gives us an idea about the extreme nature of summarization in the dataset.
Compression Ratio The ratio between the number of tokens in the headline and the article gives a sense of the conciseness of the headline, with lower values indicating more concise headlines. Overall, Vārta has a compression ratio of 5.72%, with English and Assamese headlines being the most and least concise respectively.

heading: 3.3 Extractive Baselines
text: We evaluate two extractive summarization methods on the data to see how far we can get by only selecting a sentence from the article as the headline: (i) Lead-1: choose the first sentence of the article as the headline. This can be seen as a strong performance lower bound (Nenkova, 2005), and (ii) Extractive Oracle: choose a sentence from the article which gives the highest ROUGE score with respect to the gold headline. This provides an upper bound for extractive models (Nallapati et al., 2017). We report the language-wise ROUGEL scores in Table 2 with more detailed numbers in Table 8 of the Appendix.

heading: 3.4 Data Splits
text: From every language, we randomly sample 10,000 articles each for validation and testing. We also ensure that at least 80% of a language’s data is available for training. Therefore, if a language has less than 100,000 articles, we restrict its validation and test splits to 10% of its size. Table 7 in the Appendix shows the splits for each language.
Since this training set has more than 41 million articles, performing experiments and ablations will be compute- and time-intensive. Therefore, we create a SMALL training set by limiting the number of articles from each language to 100K. This SMALL training set with a size of 1.3M is used in all our fine-tuning experiments.

heading: 4 Experiments
text: Rather than trying to squeeze optimal performance from models, the aim of our experiments is to perform a detailed study of how different training settings affect model behavior. In this section, we first present a list of research questions that our experiments try to answer. We then provide a brief overview of the models used in the experiments. Lastly, we describe the different training and evaluation configurations of our experiments before presenting our findings in §5.

heading: 4.1 Research Questions
text: RQ1: What is the best bridge language? A bridge or pivot language is an intermediary for transferring knowledge learned by training in one language to others. English is commonly used as the bridge language in literature. We test if English is indeed the best bridge language in our data, or if can we use Hindi since it is typologically similar to other languages in Vārta.
RQ2: Are different scripts a hindrance? Vārta contains languages with 11 unique scripts. Can we have a better transfer if we use a single script during fine-tuning? Which script would help the most? Why?
RQ3: Which setup performs the best? We can finetune models under different settings with different data configurations. Which of them would give us the best result? What is required for
effective transfer? Does training language-familyspecific models give us an advantage over massive multilinguality?
RQ4: Can Vārta be used for pretraining? The objective of this work is to curate a highquality dataset for headline generation in Indic languages. Upon realization that the final dataset size is comparable to existing pretraining corpora for Indic languages, we decided to compare the downstream performance of models pretrained on Vārta against those pretrained on similar corpora. Therefore, we ask the following questions: Can Vārta be used to pretrain both NLU and NLG models? If yes, are they competitive? Can our finetuned models generalize to other similar tasks, like abstractive summarization?

heading: 4.2 Models
text: mT5 introduced by Xue et al. (2021), it is the multilingual variant of T5 (Raffel et al., 2022). It is pretrained on the mC4 corpus, comprising 101 languages. The pretraining objective of mT5 is “spancorruption” which is generating spans of text that are masked in the input.
mBERT-seq2seq uses a multilingual variant of BERT (Devlin et al., 2019) pretrained on Wikipedia, on a total of 104 languages.6 Since BERT is an encoder-only model trained on
6It should be noted that both mT5 and mBERT are not pretrained on 3 Vārta languages: as, bh, and or.
masked language modeling, it cannot be used directly for generation. However, Rothe et al. (2020) show that initializing encoder-decoder models with pretrained encoder-only or decoderonly checkpoints or “warm-starting”, yields competitive results in multiple sequence-to-sequence tasks. We warm-start both the encoder and decoder of our model with mBERT weights and the encoder-decoder attention weights are initialized randomly.7 We denote this model as just mBERT from here on.
Another reason for choosing such a model configuration as a baseline is that we can compare and contrast how models learn when their encoderdecoder attention weights are (not) initialized from pretrained weights. Aralikatte et al. (2021) show models with randomly initialized encoderdecoder attention learn data characteristics much better than their pre-initialized counterparts.
Vārta-T5 In the pretraining corpus used for mT5 and mBERT, the size of the Indic languages we are interested in, is relatively small. Previous works have shown that such underrepresented languages in massively multilingual models suffer due to a lack of model capacity (Khanuja et al., 2022), and poor transfer (Ponti et al., 2021, Section 2), and that pretraining only on a group of closely related languages results in better downstream performance (Doddapaneni et al., 2022). Therefore, we use the full training set from Vārta to pretrain a T5 model from scratch.
We use span corruption and gap-sentence generation as the pretraining objectives. Both objectives are sampled uniformly during pretraining. Span corruption is similar to masked language modeling except that instead of masking random tokens, we mask spans of tokens with an average length of 3. In gap-sentence prediction, whole sentences are masked instead of spans. We follow the original work (Zhang et al., 2020), and select sentences based on their ‘importance’. ROUGE-1 F1-score between the sentence and the document is used as a proxy for importance. We use 0.15 and 0.2 as the masking ratios for span corruption and gapsentence generation, respectively.
We use a standard T5-base architecture with 12 encoder and decoder layers, and 12 attention heads. Since data sizes across languages in
7We do not share parameters between the encoder and decoder since early experiments showed that sharing parameters slightly hurt performance.
Vārta vary from 1.5K (Bhojpuri) to 14.4M articles (Hindi), we use standard temperature-based sampling to upsample data when necessary.8 For other training details, see Appendix A.3.

heading: 4.3 Finetuning and Evaluation settings
text: To answer the questions posed in §4.1, we perform a series of experiments. First, we finetune each model described in §4.2 on Vārta in five settings: (i) en: finetune only on English data from the SMALL training set, and evaluate on all language test sets in original scripts. (ii) hi: finetune only on Hindi data from the SMALL training set, and evaluate on all language test sets in original scripts. (iii) latin: finetune on the SMALL training set transliterated to Latin (English) script, and evaluate on all language test sets transliterated to Latin script.9 (iv) dvn.: finetune on the SMALL training set transliterated to Devanagari script, and evaluate on all language test sets transliterated to Devanagari script.10 (v) all: finetune on all languages of the SMALL training set, and evaluate on all language test sets, in their original scripts.
Next, to assess the generalizability of our pretrained model, we conduct the following two experiments on the Vārta-T5 model: (i) evalaute on the XL-Sum dataset (Hasan et al., 2021). XL-Sum is particularly useful as it contains article-headlinesummary triplets, which allows for the evaluation of both headline generation and abstractive summarization tasks. We evaluate the best models from the previous set of experiments in a zeroshot setting. (ii) To determine the generalizability of Vārta-T5 on text generation tasks, we evaluate the model on the IndicNLG benchmark (Kumar et al., 2022b) which contains five diverse generation tasks in 11 Indic languages.
Finally, to determine if Vārta can be used to train good NLU models, we train Vārta-BERT, an encoder-only model trained with the masked language modeling objective and evaluated on IndicXTREME (Doddapaneni et al., 2022), a zeroshot cross-lingual NLU benchmark for Indic languages consisting of nine tasks in 18 languages.
8We use α = 0.7. 9We use the IndicTrans (Bhat et al., 2015) and IndicNLP (Kunchukuttan, 2020) libraries to transliterate text to English and Devanagari respectively. For Urdu, we use IndicTrans in both cases. Though Bhojpuri uses Devanagari, we ignore it while performing the unified script experiments since it does not have a good transliteration system.
10We denote latin (dvn.) model as English (Devanagari) ‘unified model’ since it is trained on a single, unified script.

heading: 5 Results

heading: 5.1 Headline Generation Results
text: The ROUGE-L scores for each of the three models in all five settings can be found in Table 3. Overall, Vārta-T5 finetuned in the all setting performs the best with an average ROUGE-L of 40.48. Although better, it is just six points above the Extractive Oracle baseline. This indicates the difficulty of the Vārta dataset, and that there is much room for improvement even among state-of-the-art models.

heading: 5.1.1 mBERT Observations
text: On average, mBERT has inferior performance compared to other models, which is expected due to it having to learn a fraction of its parameters from scratch. However, when trained on the Devanagari and original scripts, mBERT either outperforms or is comparable to the other models in Bhojpuri and Hindi (both use Devanagari script). It also achieves comparable results to mT5 (with a margin of 0.5 ROUGE-L or less) in five languages when trained on the original script data. But when trained only on English and Hindi data, the model is unable to transfer anything meaningful to other languages, showing that warm starting is not effective in zero-shot cross-lingual settings. This effect is particularly visible in the case of Oriya, which does not share its script with any other language in the dataset, and on which the model has not been pretrained.
However, we observe that when trained only on Hindi data, the model demonstrates faster learning and better transfer capabilities than when trained
only on English. This might be because (i) 4/15 languages in the dataset use the Devanagari script, which is also used by Hindi, and (ii) Hindi is closely related to the other languages in the dataset. This suggests that Hindi might be a better bridge language for Vārta than English. This hypothesis is further supported by the fact that the Devanagari unified model (dvn.) performs significantly better than the Latin unified model.

heading: 5.1.2 mT5 Observations
text: Finetuning on Original Scripts The model performs the best in this setting, which is expected since it is pretrained on 12 out of 15 languages in the dataset. More importantly, it performs much better than mBERT in Assamese, Bhojpuri, and Oriya, the only three languages both models have not seen during pretraining. This indicates better intra-script and inter-script zero-shot transfer in mT5 (Assamese shares its script with Bengali, and Bhojpuri with Hindi, but Oriya is unique).
Bridge Language We find that the mT5 model fine-tuned only on English data demonstrates an average improvement of 3.81 ROUGE-L points over the model fine-tuned only on Hindi. This difference in performance can potentially be attributed to the fact that English is the model’s largest pretraining language, and it also has the highest share in the vocabulary of the model. Further analysis of the Vārta-T5 model will support this hypothesis.
Unified Script On average, the Devanagari unified model demonstrates better performance, with an improvement of 0.67 ROUGE-L, when compared to the Latin unified model. Although the performance of the two models is similar for the majority of languages, the Devanagari model shows significant improvement on Marathi, Nepali, and Hindi, all of which also use the Devanagari script. This observation of improved transfer across languages that share a script is a recurrent theme across models and settings. However, it is noteworthy that the Latin model performed better on Assamese and Tamil.

heading: 5.1.3 Vārta-T5 observations
text: Finetuning on Original Scripts Vārta-T5 finetuned in this setting has the highest performance among all models, with a ROUGE-L score of 40.48. This result is expected, as it is pretrained on the same data and one of its pretraining objectives (gap-sentence generation) is almost equivalent to generating sentences that the Extractive Oracle would extract.
Bridge Language We see that the model finetuned only on Hindi outperforms the model finetuned only on English by a margin of 2.68 ROUGEL points. This supports our previous hypothesis that the language with the largest pretraining data is the most effective bridge language. In addition to being the largest language with the largest vocabulary share, Hindi also shares its script with
three other languages in the dataset and is typologically similar to the majority of other languages in the dataset. This should ideally make the Hindionly models significantly better than the Englishonly models. However, we should note that English is the second biggest language in Vārta with 7.2M articles (half the size of Hindi), and therefore the difference in performance between Englishonly and Hindi-only models, or Latin unified and Devanagari unified models is not as significant as expected.
Unified Script On average, the Devanagari unified model is 2.06 ROUGE-L points better than the Latin unified model. The Devanagari model performs much better on Gujarati, Hindi, Marathi, Nepali, and Urdu, with improvements of more than 2 ROUGE-L points.11 In general, all IndoEuropean languages see improvements with the Devanagari unified model. However, such a claim cannot be made for Dravidian languages. While Kannada and Telugu see improvements, we see a performance drop in Malayalam and Telugu.12

heading: 5.2 Generalization Results
text: We see that the models pretrained on Vārta consistently outperform strong baselines and often by significant margins. We argue (particularly in §5.2.2) that Vārta is a good resource for pretraining large language models on Indic languages, especially since it has data from diverse, highquality sources.

heading: 5.2.1 XL-Sum
text: To test the generalizability of our models on other datasets and tasks, we evaluate models finetuned on Vārta on the Indic subset of the XL-Sum dataset (Hasan et al., 2021), for both abstractive summarization and headline generation. We select the best mT5 and Vārta-T5 models obtained previously and evaluate them on nine Indic languages and English, without any additional finetuning. The results are shown in Table 4. We find that Vārta-T5 consistently outperforms mT5 on all languages in both tasks. On average, we see a gain of around 2 ROUGE-L points on abstractive summarization and 4 ROUGE-L points on headline generation respectively.
11Hindi, Marathi, and Nepali use the Devanagari script. 12Surprisingly, we do not see this drop in mT5 models
where English is the dominant pretraining language.

heading: 5.2.2 Vārta on NLU tasks
text: To verify if Vārta can be used to train good NLU models, we use it to pretrain a masked language model with the BERT-Base architecture. We name this model Vārta-BERT and more information about the pretraining can be found in Appendix A.2. We evaluate our model on the IndicXTREME benchmark (Doddapaneni et al., 2022) which consists of 9 tasks in 19 Indic languages and English.
We compare our models against two strong baselines: IndicBERT v1 and v2. These two BERT-Base models are trained on IndicCorp v1 (Kakwani et al., 2020) and IndicCorp v2 (Doddapaneni et al., 2022), with 8.5B and 20.9B tokens respectively. We are mainly interested in the comparison with IndicBERT v1 since the size of its pretraining corpus is comparable with that of Vārta (9B tokens). Table 5 shows the results averaged across languages on the nine tasks. We see that Vārta-BERT consistently outperforms IndicBERT v1 indicating its quality. We should also note that Vārta-BERT’s performance is not too far behind IndicBERT v2 even though it is trained on 11B fewer tokens. In fact, it outperforms IndicBERT v2 on three tasks and loses out on three other tasks by a margin of less than one point.

heading: 5.2.3 Vārta on NLG tasks
text: Finally, we evaluate Vārta-T5 on IndicNLG and compare its performance against two strong baselines: IndicBART (Dabre et al., 2022) and mT5.13 The comparison is presented in Table 6. It is to
13Here we use the original mT5 model trained on the mC4 corpus.
be noted that, at the time of writing we could not independently reproduce all the results presented in Kumar et al. (2022b). We, therefore, present the results as originally reported along with the ones obtained during our experiments. We see that, overall, Vārta-T5 is the best-performing model in 3 out of 5 tasks. But when compared with the reproduced results only, it performs better in 4 out of 5 tasks.

heading: 5.3 Key Takeaways
text: Based on our experiments and analyses, we now try to answer the research questions posed in §4.1.
RQ1 We find that the largest pretraining language typically acts as the best bridge language. It helps if the language is typologically similar to other languages and shares a common script with them. In the case of dataset Vārta, Hindi is the ideal bridge language as it has all of the above properties (though it shares a common script with only three other languages).
RQ2 and RQ3 The performance of the headline generation models is not negatively impacted by the variety of scripts. In fact, models fine-tuned in this setting yield the best results. It is also interesting to note that the transliterated models perform better than the monolingual models. This suggests that, in general, multilingual training enables positive transfer among related languages, with or without the use of their original scripts.
RQ4 We find that Vārta-T5 generalizes well on other, similar tasks like abstractive summarization. Vārta-BERT and Vārta-T5 generally perform on par or better than strong baselines on both NLU and NLG benchmarks for Indic languages.

heading: 6 Conclusion
text: In this work, we create Vārta, a large-scale headline-generation dataset for Indic languages. We show that this dataset is challenging even for state-of-the-art text generation models. Utilizing the size and quality of the dataset, we answer pertinent research questions about multilingual models, from an Indic text generation perspective. We also show that Vārta can be used as a pretraining corpus to train strong NLU and NLG models.

heading: Acknowledgements
text: We would like to thank the anonymous reviewers for their comments and suggestions. We also thank Google’s TPU Research Cloud (TRC) for giving us free access to their v3-128 TPUs for pretraining our models. We also acknowledge the support of the Natural Sciences and Engineering Research Council of Canada (NSERC) for funding Ziling Cheng through their grant program.
Limitations
This work is mainly dedicated to the curation of a new multilingual dataset for Indic languages, many of which are low-resource languages. During data collection, we face several limitations that can potentially result in ethical concerns. Some of the important ones are mentioned below:
• Our dataset contains only those articles written by DailyHunt’s partner publishers. This has the potential to result in a bias towards a particular narrative or ideology that can affect the representativeness and diversity of the dataset.
• Another limitation is the languages represented in Vārta. Out of 22 languages with official status in India, our dataset has only 13. There are 122 major languages spoken by at least 10,000 people and 159 other languages which are extremely low-resourced.14
None of these languages are represented in our dataset.
• We do not perform any kind of debiasing on Vārta. This means that societal and cultural biases may exist in the dataset, which can adversely affect the fairness and inclusivity of the models trained on it.
14https://en.wikipedia.org/wiki/ Languages_of_India
Ethics Statement
The ethical considerations that arise from the limitations of our data collection process are already detailed in the previous section. In this section, we discuss the implications of releasing the data, how we intend to do so in a safe manner, and the license under which it would be released.
While Vārta has the potential to advance NLP research for Indic languages,15 it can also be used in ways not intended by the authors. Since Vārta can be used to pretrain text generation models, it can be used to build models that generate hate speech, fake news, etc.
Since our data is aggregated from different sources and each source may have different restrictions on distributing their data, we only release a list of URLs pointing to the original articles and not the articles themselves, which is a standard and acceptable way of sharing data.16 However, we provide a sample script that can be used to crawl the URLs and rebuild Vārta. We release the URL list under a CC-BY license17 and dedicate it to the public domain. The released code and models will have an Apache License 2.0.18

heading: B Misc. Information
text: Information about the scripts used by each language in Vārta is presented in Table 7. The table also contains the sizes of the per-language data splits. Table 8 contains the full results (ROUGE-1, ROUGE-2, and ROUGE-L) for the extractive baselines. We also show a few random examples of headline-article pairs from Vārta in Table 9.

heading: C Finetuning Results
text: We present detailed results of our headline generation finetuning experiments in Table 10, 11, and 12. For each model (mBERT, mT5, and Vārta-T5), we present the ROUGE-1, ROUGE-2, and ROUGEL F1-scores in all five settings: en, hi, latin, dvn., and all.
In Table 13, we present the zero-shot results of mT5 and Vārta-T5 on both headline generation and abstractive summarization tasks of the XLSum dataset.
D IndicXTREME Results
IndicXTREME is a cross-lingual benchmark consisting of nine tasks in 18 Indic languages. The nine tasks are as follows: (i) IndicCOPA: for commonsense causal reasoning, (ii) IndicQA: for question-answering, (iii) IndicXParaphrase: for paraphrase detection, (iv) IndicSentiment: for sentiment classification, (v) IndicXNLI: for natural language inference, (vi) Naamapadam: for named entity recognition, (vii) MASSIVE: for intent classification and slot filling (also classification), and (ix) FLORES: for retrieval using semantic similarity.
Vārta-BERT is compared with two other BERT models pretrained on IndicCorp v1 and v2 respectively. For each task, we finetune the three models on English training data and evaluate them on Indic languages. For a comprehensive list of training data, please refer Doddapaneni et al. (2022).
For every task, the per-language results for all three models are presented in Tables 14, 15, 16, 17, 18, 19, 20, 21, and 22 respectively.
E IndicNLG Results
IndicNLG is a benchmark for evaluating NLG in 11 Indic languages. It consists of five tasks: (i)
generate the first sentence of a Wikipedia page, given its infobox, (ii) headline generation, (iii) sentence summarization, (iv) paraphrase generation, and (v) question generation. More information about the datasets can be found in Kumar et al. (2022b).
We compare Vārta-T5 with IndicBART and mT5 results reported in (Kumar et al., 2022b), and also with a version of mT5 finetuned by us. All tasks use ROUGE as their metric, except paraphrase generation which uses BLEU. The task– wise results on all supported languages for mT5 and Vārta-T5 are presented in Tables 23 and 24 respectively.20 For paraphrase generation, we report three BLEU variants: standard BLEU, selfBLEU, and iBLEU (with α = 0.7). We see negative self-BLEU numbers for both models which suggest that, in some languages, there is a large lexical overlap between the input sentences and the generated paraphrases.

heading: F Datasheet
text: We also provide a Datasheet (Gebru et al., 2018) for our dataset in Table 25. This provides an indepth rationale behind the creation, distribution, and maintenance of Vārta, including any underlying assumptions, potential risks or harms, and implications of use.
20These tables only contain the results for Vārta-T5 and the mT5 model we finetuned. The detailed results of the other two models can be found in Kumar et al. (2022b).
Lang. mT5
Wikibio Sentence Sum. Head. Gen. Question Gen. Paraphrase (BLEU)
R1 R2 RL R1 R2 RL R1 R2 RL R1 R2 RL BLEU SelfBLEU iBLEU
as 57.08 37.78 55.78 38.95 23.92 37.5 34.16 18.39 32.44 22.33 8.77 21.17 20.75 -12.70 10.71 bn 56.83 37.97 55.25 31.87 20.76 30.68 32.57 18.02 30.58 27.77 11.09 26.34 13.38 -21.02 3.06 gu - - - 27.3 13.6 26.13 32.96 17.49 31.39 28.59 10.91 27.46 23.71 -100 -13.4 hi 66.68 53.06 65.79 33.52 15.92 29.81 36.61 17.61 32.16 35.57 15.71 32.73 35.13 -100.0 -5.40 kn 42.42 22.42 40.73 73.35 67.18 72.86 47.4 30.64 46.15 24.44 9.35 23.4 17.52 0.0 12.26 ml 43.16 22.13 37.83 42.54 28.28 41.3 40.14 23.52 38.58 22.62 8.35 21.34 10.37 -69.97 -13.73 mr - - - 23.33 10.4 22.61 32.59 16.89 31.48 23.78 9.03 22.92 22.99 -33.43 6.06 or 70.85 57.06 70.07 23.44 12.17 22.6 22.55 10.25 21.56 25.83 10.88 24.51 19.94 -100.0 -16.03 pa 54.78 34.95 53.33 48.78 32.16 45.32 46.85 30.06 43.4 33.19 13.73 31.13 25.32 -63.89 -1.44 ta 52.24 30.11 50.47 40.53 22.99 38.86 47.72 31.3 46.16 23.82 8.74 22.89 18.68 -100.0 16.92 te 53.17 29.56 50.65 30.44 15.45 29.43 32.22 15.77 30.83 26.72 10.64 25.56 18.14 -39.76 0.76 Avg 55.25 36.12 53.32 37.64 23.89 36.1 36.89 20.9 34.98 26.79 10.65 25.4 20.54 -58.25 -0.02
as 60.56 41.4 59.1 35.67 20.05 34.03 37.61 20.8 35.54 23.58 10.19 22.48 15.17 -70.71 -10.59 bn 58.4 39.4 56.65 31.88 17.06 29.93 39.27 20.8 36.27 35.18 15.51 32.93 16.77 -22.59 4.96 gu - - - 27.73 13.78 26.5 36.36 19.32 34.49 30.97 12.52 29.75 31.09 -100.0 -8.23 hi 67.48 54.06 66.61 34.7 16.27 30.46 41.65 21 36.37 41.01 19.56 37.42 42.51 -80.91 5.48 kn 42.57 21.62 40.18 42.89 27.45 41.54 43.56 24.84 41.86 26.72 10.83 25.52 22.87 0.0 16.01 ml 44.32 23.01 38.8 37.75 23.43 36.64 39.18 21.95 37.56 24.73 9.46 23.35 19.95 -100 -16.04 mr - - - 23.69 10.8 22.97 36.38 19.43 35.03 26.56 10.91 25.68 31.08 -50.81 6.52 or 72.78 58.8 71.73 23.62 12.35 22.7 30.13 15.04 28.76 30.77 14.37 29.23 29.03 -70.71 -0.89 pa 55.04 35.33 53.45 48.39 31.76 44.93 50.67 33.44 47.1 36.36 16.21 34.16 32.37 -59.46 4.82 ta 53.26 31.49 51.41 41.06 23.35 39.36 49.35 32.44 47.7 26.04 10.08 24.98 23.97 -45.18 3.23 te 54.7 31.38 52.5 30.47 15.14 29.32 36.7 18.85 35.17 28.92 12.24 27.61 24.04 -9.82 13.88 Avg 56.57 37.39 54.49 34.35 19.22 32.58 40.08 22.54 37.8 30.08 12.9 28.46 26.26 -55.47 1.74
Table 24: Performance of Vārta-T5 on IndicNLG tasks.
Q: Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety? (If so, please describe why.) A: Since the dataset contains news articles, there may be reports of violence, murder, racial discrimination, and other triggering phenomena.
Q: Does the dataset relate to people? (If not, you may skip the remaining questions in this section.) A: Yes, most of the articles talk about events involving real people.
Q: Does the dataset identify any subpopulations (e.g., by age, gender)? (If so, please describe how these subpopulations are identified and provide a description of their respective distributions within the dataset.) A: This is not explicitly identified, though many of the articles explicitly mention the gender of the people described/discussed.
Q: Is it possible to identify individuals (i.e., one or more natural persons), either directly or indirectly (i.e., in combination with other data) from the dataset? (If so, please describe how.) A: Yes; their names are given in the running text.
Q: Does the dataset contain data that might be considered sensitive in any way (e.g., data that reveals racial or ethnic origins, sexual orientations, religious beliefs, political opinions or union memberships, or locations; financial or health data; biometric or genetic data; forms of government identification, such as social security numbers; criminal history)? (If so, please provide a description.) A: Yes, some articles may contain the personal information of individuals. But it is very unlikely that they contain information that is not already public.
Q: Any other comments? A: None.
Collection Process Q: How was the data associated with each instance acquired? (Was the data directly observable (e.g., raw text, movie ratings), reported by subjects (e.g., survey responses), or indirectly inferred/derived from other data (e.g., part-of-speech tags, model-based guesses for age or language)? If data was reported by subjects or indirectly inferred/derived from other data, was the data validated/verified? If so, please describe how.) A: The data was all downloaded directly from the news aggregator DailyHunt.
Q: What mechanisms or procedures were used to collect the data (e.g., hardware apparatus or sensor, manual human curation, software program, software API)? (How were these mechanisms or procedures validated?) A: We use a Python-based crawling program called Scrapy to collect the data. It is publicly available at this URL: https://scrapy.org/. The procedure is manually validated by randomly selecting articles and comparing their content with the processed text.
Q: If the dataset is a sample from a larger set, what was the sampling strategy (e.g., deterministic, probabilistic with specific sampling probabilities)? A: The dataset is not part of a larger corpus.
Q: Who was involved in the data collection process (e.g., students, crowd workers, contractors) and how were they compensated (e.g., how much were crowd workers paid)? A: The data is automatically crawled.
Q: Over what timeframe was the data collected? (Does this timeframe match the creation timeframe of the data associated with the instances (e.g., recent crawl of old news articles)? If not, please describe the timeframe in which the data associated with the instances was created.) A: The data was crawled in August 2022. The crawled articles were published between January 2010 to August 2022.
Q: Were any ethical review processes conducted (e.g., by an institutional review board)? (If so, please provide a description of these review processes, including the outcomes, as well as a link or other access point to any supporting documentation.) A: No ethical review processes were conducted.
Q: Does the dataset relate to people? (If not, you may skip the remaining questions in this section.) A: Yes, most of the articles talk about events involving real people.
Q: Did you collect the data from the individuals in question directly, or obtain it via third parties or other sources (e.g., websites)? A: The data was collected from other sources.
Q: Were the individuals in question notified about the data collection? (If so, please describe (or show with screenshots or other information) how notice was provided, and provide a link or other access point to, or otherwise reproduce, the exact language of the notification itself.) A: No, they were not notified.
Q: Did the individuals in question consent to the collection and use of their data? (If so, please describe (or show with screenshots or other information) how consent was requested and provided, and provide a link or other access point to, or otherwise reproduce, the exact language to which the individuals consented.) A: No; since all the news articles are public.
Q: If consent was obtained, were the consenting individuals provided with a mechanism to revoke their consent in the future or for certain uses? (If so, please provide a description, as well as a link or other access point to the mechanism (if appropriate).) A: Not applicable.
Q: Has an analysis of the potential impact of the dataset and its use on data subjects (e.g., a data protection impact analysis) been conducted? (If so, please provide a description of this analysis, including the outcomes, as well as a link or other access point to any supporting documentation.) A: No.
Q: Any other comments? A: None.
Preprocessing/cleaning/labeling Q: Was any preprocessing/cleaning/labeling of the data done (e.g., discretization or bucketing, tokenization, part-ofspeech tagging, SIFT feature extraction, removal of instances, processing of missing values)? (If so, please provide a description. If not, you may skip the remainder of the questions in this section.) A: Yes; see the ‘Processing’ paragraph in Section §3.1.
Q: Was the "raw" data saved in addition to the preprocessed/cleaned/labeled data (e.g., to support unanticipated future uses)? (If so, please provide a link or other access point to the "raw" data.) A: No, the original raw data is not included in the distribution, due to licensing restrictions.
Q: Is the software used to preprocess/clean/label the instances available? (If so, please provide a link or other access point.) A: Yes, we use Beautiful Soup 4 which is available at this URL: https://www.crummy.com/software/ BeautifulSoup/.
Q: Any other comments? A: None.
Uses Q: Has the dataset been used for any tasks already? (If so, please provide a description.) A: The dataset has been used to finetune existing models to perform headline generation on Indic languages. It has also been used to pre-train language models. See §4.2 for more details.
Q: Is there a repository that links to any or all papers or systems that use the dataset? (If so, please provide a link or other access point.) A: No.
Q: What (other) tasks could the dataset be used for? A: The dataset could be used for mining articles that talk about the same piece of news within and across languages to perform interesting analyses such as: (i) how the same news is reported in different languages?, and (ii) how an article evolves as time progresses, and when new information becomes available.
Q: Is there anything about the composition of the dataset or the way it was collected and preprocessed/cleaned/labeled that might impact future uses? (For example, is there anything that a future user might need to know to avoid uses that could result in unfair treatment of individuals or groups (e.g., stereotyping, quality of service issues) or other undesirable harms (e.g., financial harms, legal risks) If so, please provide a description. Is there anything a future user could do to mitigate these undesirable harms?) A: No.
Q: Are there tasks for which the dataset should not be used? (If so, please provide a description.) A: As with any large dataset, Vārta can be used in a variety of harmful ways. For example, it can be used to build models that generate hate speech, fake news, etc.
Q: Any other comments? A: None.
Distribution Q: Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset was created? (If so, please provide a description.) A: Yes, the dataset will be freely available.
Q: How will the dataset be distributed (e.g., tarball on website, API, GitHub)? (Does the dataset have a digital object identifier (DOI)?) A: The dataset will be made available on GitHub.
Q: When will the dataset be distributed? A: The dataset is distributed after this work is published.
Q: Will the dataset be distributed under a copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)? (If so, please describe this license and/or ToU, and provide a link or other access point to, or otherwise reproduce, any relevant licensing terms or ToU, as well as any fees associated with these restrictions.) A: The dataset is licensed under a CC-0 license.
Q: Have any third parties imposed IP-based or other restrictions on the data associated with the instances? (If so, please describe these restrictions, and provide a link or other access point to, or otherwise reproduce, any relevant licensing terms, as well as any fees associated with these restrictions.) A: The news aggregator from which we collect data grants a non-exclusive and non-transferable license to download. More information can be found here: https://www.dailyhunt.com/user-agreement.
Q: Do any export controls or other regulatory restrictions apply to the dataset or individual instances? (If so, please describe these restrictions, and provide a link or other access point to, or otherwise reproduce, any supporting documentation.) A: Not to our knowledge.
Q: Any other comments? A: None.
Maintenance Q: Who is supporting/hosting/maintaining the dataset? A: All authors will maintain the dataset and the first author will host it on GitHub.
Q: How can the owner/curator/manager of the dataset be contacted (e.g., email address)? A: The E-mail address of the first author will be provided.
Q: Is there an erratum? (If so, please provide a link or other access point.) A: No.
Q: Will the dataset be updated (e.g., to correct labeling errors, add new instances, delete instances’)? (If so, please describe how often, by whom, and how updates will be communicated to users (e.g., mailing list, GitHub)?) A: Currently, there is no plan to update the dataset.
Q: If the dataset relates to people, are there applicable limits on the retention of the data associated with the instances (e.g., were individuals in question told that their data would be retained for a fixed period of time and then deleted)? (If so, please describe these limits and explain how they will be enforced.) A: No.
Q: Will older versions of the dataset continue to be supported/hosted/maintained? (If so, please describe how. If not, please describe how its obsolescence will be communicated to users.) A: Yes, if a new update is released, the older versions will still be maintained.
Q: If others want to extend/augment/build on/contribute to the dataset, is there a mechanism for them to do so? (If so, please provide a description. Will these contributions be validated/verified? If so, please describe how. If not, why not? Is there a process for communicating/distributing these contributions to other users? If so, please provide a description.) A: Since the data is collected automatically, others can easily extend/augment/build on/contribute to the dataset. It does not require any additional verification. There is currently no official channel for communication between the authors and potential contributors to communicate. One will be created if it becomes necessary.
Q: Any other comments? A: None.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Limitations (after the conclusion)
3 A2. Did you discuss any potential risks of your work? We discuss this in the Ethics Statement (after Limitations)
3 A3. Do the abstract and introduction summarize the paper’s main claims? Abstract and Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? We create a dataset whose details can be found in Section 3
3 B1. Did you cite the creators of artifacts you used? We use pre-trained language models as baselines. More information can be found in Section 4.2
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? We discuss this in the Ethics Statement (after Limitations)
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? We discuss the intended use of our dataset in great detail in Section 1.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Our dataset contains only news articles from reputed publishers
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Sections 3.1 and 3.2
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? Section 4
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Various sections of the Appendix
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 4
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? We do report the summary statistics of the benchmarking experiments in Section 5.
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix A
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Q: If consent was obtained, were the consenting individuals provided with a mechanism to revoke their consent in the future or for certain uses? (If so, please provide a description, as well as a link or other access point to the mechanism (if appropriate). ) A: Not applicable. Q: Has an analysis of the potential impact of the dataset and its use on data subjects (e. g. , a data protection impact analysis) been conducted? (If so, please provide a description of this analysis, including the outcomes, as well as a link or other access point to any supporting documentation. ) A: No. Q: Any other comments? A: None. Preprocessing/cleaning/labeling Q: Was any preprocessing/cleaning/labeling of the data done (e. g. , discretization or bucketing, tokenization, part-ofspeech tagging, SIFT feature extraction, removal of instances, processing of missing values)? (If so, please provide a description. If not, you may skip the remainder of the questions in this section. ) A: Yes; see the ‘Processing’ paragraph in Section §3. 1. Q: Was the "raw" data saved in addition to the preprocessed/cleaned/labeled data (e. g. , to support unanticipated future uses)? (If so, please provide a link or other access point to the "raw" data. ) A: No, the original raw data is not included in the distribution, due to licensing restrictions. Q: Is the software used to preprocess/clean/label the instances available? (If so, please provide a link or other access point. ) A: Yes, we use Beautiful Soup 4 which is available at this URL: https://www. crummy. com/software/ BeautifulSoup/. Q: Any other comments? A: None. Uses Q: Has the dataset been used for any tasks already? (If so, please provide a description. ) A: The dataset has been used to finetune existing models to perform headline generation on Indic languages. It has also been used to pre-train language models. See §4. 2 for more details. Q: Is there a repository that links to any or all papers or systems that use the dataset? (If so, please provide a link or other access point. ) A: No. Q: What (other) tasks could the dataset be used for? A: The dataset could be used for mining articles that talk about the same piece of news within and across languages to perform interesting analyses such as: (i) how the same news is reported in different languages?, and (ii) how an article evolves as time progresses, and when new information becomes available. Q: Is there anything about the composition of the dataset or the way it was collected and preprocessed/cleaned/labeled that might impact future uses? (For example, is there anything that a future user might need to know to avoid uses that could result in unfair treatment of individuals or groups (e. g. , stereotyping, quality of service issues) or other undesirable harms (e. g. , financial harms, legal risks) If so, please provide a description. Is there anything a future user could do to mitigate these undesirable harms?) A: No. Q: Are there tasks for which the dataset should not be used? (If so, please provide a description. ) A: As with any large dataset, Vārta can be used in a variety of harmful ways. For example, it can be used to build models that generate hate speech, fake news, etc. Q: Any other comments? A: None. Distribution Q: Will the dataset be distributed to third parties outside of the entity (e. g. , company, institution, organization) on behalf of which the dataset was created? (If so, please provide a description. ) A: Yes, the dataset will be freely available. Q: How will the dataset be distributed (e. g. , tarball on website, API, GitHub)? (Does the dataset have a digital object identifier (DOI)?) A: The dataset will be made available on GitHub. Q: When will the dataset be distributed? A: The dataset is distributed after this work is published. Q: Will the dataset be distributed under a copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)? (If so, please describe this license and/or ToU, and provide a link or other access point to, or otherwise reproduce, any relevant licensing terms or ToU, as well as any fees associated with these restrictions. ) A: The dataset is licensed under a CC-0 license. Q: Have any third parties imposed IP-based or other restrictions on the data associated with the instances? (If so, please describe these restrictions, and provide a link or other access point to, or otherwise reproduce, any relevant licensing terms, as well as any fees associated with these restrictions. ) A: The news aggregator from which we collect data grants a non-exclusive and non-transferable license to download. More information can be found here: https://www. dailyhunt. com/user-agreement. Q: Do any export controls or other regulatory restrictions apply to the dataset or individual instances? (If so, please describe these restrictions, and provide a link or other access point to, or otherwise reproduce, any supporting documentation. ) A: Not to our knowledge. Q: Any other comments? A: None. Maintenance Q: Who is supporting/hosting/maintaining the dataset? A: All authors will maintain the dataset and the first author will host it on GitHub. Q: How can the owner/curator/manager of the dataset be contacted (e. g. , email address)? A: The E-mail address of the first author will be provided. Q: Is there an erratum? (If so, please provide a link or other access point. ) A: No. Q: Will the dataset be updated (e. g. , to correct labeling errors, add new instances, delete instances’)? (If so, please describe how often, by whom, and how updates will be communicated to users (e. g. , mailing list, GitHub)?) A: Currently, there is no plan to update the dataset. Q: If the dataset relates to people, are there applicable limits on the retention of the data associated with the instances (e. g. , were individuals in question told that their data would be retained for a fixed period of time and then deleted)? (If so, please describe these limits and explain how they will be enforced. ) A: No. Q: Will older versions of the dataset continue to be supported/hosted/maintained? (If so, please describe how. If not, please describe how its obsolescence will be communicated to users. ) A: Yes, if a new update is released, the older versions will still be maintained. Q: If others want to extend/augment/build on/contribute to the dataset, is there a mechanism for them to do so? (If so, please provide a description. Will these contributions be validated/verified? If so, please describe how. If not, why not? Is there a process for communicating/distributing these contributions to other users? If so, please provide a description. ) A: Since the data is collected automatically, others can easily extend/augment/build on/contribute to the dataset. It does not require any additional verification. There is currently no official channel for communication between the authors and potential contributors to communicate. One will be created if it becomes necessary. Q: Any other comments? A: None. ACL 2023 Responsible NLP Checklist