SlowGuess/NAACL_2022_OCR huggingface dataset:
Encarna Segarra Vicent Ahuir $^{\S}$  Lluis-F. Hurtado José Angel González
VRAIN: Valencian Research Institute for Artificial Intelligence
Universitat Politécnica de València, Spain
{esegarra, lhurtado, jogomba2}@dsic.upv.es
$\S$  viahes@eui.upv.es
The application of supervised methods to automatic summarization requires the availability of adequate corpora consisting of a set of document-summary pairs. As in most Natural Language Processing tasks, the great majority of available datasets for summarization are in English, making it difficult to develop automatic summarization models for other languages. Although Spanish is gradually forming part of some recent summarization corpora, it is not the same for minority languages such as Catalan. In this work, we describe the construction of a corpus of Catalan and Spanish newspapers, the Dataset for Automatic summarization of Catalan and Spanish newspaper Articles (DACSA) corpus. It is a high-quality large-scale corpus that can be used to train summarization models for Catalan and Spanish. We have carried out an analysis of the corpus, both in terms of the style of the summaries and the difficulty of the summarization task. In particular, we have used a set of well-known metrics in the summarization field in order to characterize the corpus. Additionally, we have evaluated the performance of some extractive and abstractive summarization systems on the DACSA corpus for benchmarking purposes.
Automatic summarization is one of the central problems in Natural Language Processing (NLP). The development of automatic summarization systems is an important issue due to the great amount of information in different formats that is accessible on the web or in other repositories. It is necessary to develop techniques that help us to tackle that huge amount of information. For this reason, there is an increasing interest in the NLP community to develop techniques that allow the users to find, read, understand, or process the documents. In this context, automatic summarization can be an important aid because it provides a condensed version
of documents that reduce the time to explore or analyze them.
Access to large-scale high-quality data is an essential prerequisite for making substantial progress in summarization. The application of supervised methods to automatic summarization, as those based on Neural Networks, requires the availability of adequate corpora consisting of document-summary pairs. The construction of large-scale and high-quality corpora for learning neural summarization models is not an easy task. It is necessary a great human effort to generate thousands of manual summaries, or to design new approaches to obtain these summaries in a semiautomatic way. The first important resource for learning corpus-based summarization models was the CNN/DailyMail summarization corpus (Hermann et al., 2015), originally constructed for the task of passage-based question answering and adapted to the document summarization task. It consists of news stories from CNN and DailyMail and contains 312,077 article-summary pairs. Afterwards, another English corpus was provided to the research community for summarization purposes, the NewsRoom corpus (Grusky et al., 2018). It consists of 1.3 million article-summary pairs that have been written by the authors and the editors of 38 different major news publications. The corpus was created through a web-scale crawling of over 100 million pages from a set of online publishers by gathering the news and using the summaries provided in the HTML metadata. The summaries contained in this corpus combine both extractive and abstractive strategies to describe the content of the articles. Also in 2018, the XSUM corpus (Narayan et al., 2018a) was presented, it is a large scale dataset obtained by harvesting online articles from the British Broadcasting Corporation (BBC) with one-sentence news summary.
As in most NLP tasks, the great majority of available datasets for summarization are in English. The
Human Language Technologies, pages 5931 - 5943
lack of this kind of resources for other languages is an encumbrance to modeling that constraints the impact of language technologies on minority language communities. The creation of a large-scale Indonesian summarization dataset of 215,827 document-summary pairs, has just been published (Koto et al., 2020). Recently, some datasets that aim to fill the gap among English and other languages for the automatic summarization task have been proposed: MLSUM (Scialom et al., 2020), MassiveSumm (Varab and Schluter, 2021), and XL-Summ (Hasan et al., 2021). Although Spanish is the world's second-most spoken native language and is the official language in 21 countries, it has only recently been considered in general domain summarization datasets, as the aforementioned, and in specific domains as in (González et al., 2019). The situation is worse for Catalan, although it is not an endangered language, it is spoken by 10 million people in Spain and other three European countries, it is minority worldwide and is underrepresented or even not considered in summarization corpora.
In this work, we describe the construction of a corpus of Catalan and Spanish newspapers, the Dataset for Automatic summarization of Catalan and Spanish newspaper Articles (DACSA) corpus. With the aim of building a quality large-scale corpus that could be used to learn automatic summarization neural models for Catalan and Spanish, we used a strategy inspired by the construction of the NewsRoom corpus (Grusky et al., 2018). We conducted a crawling process on 30 different newspaper websites to extract articles and summaries in a straightforward way. The crawling included from Spanish mass media to regional newspapers. In order to obtain the summaries, we took advantage of the highlights and summaries, provided by authors or editors of the articles.
To ensure the quality of the DACSA corpus, we perform two subsequent filtering processes on the downloaded articles. The first filter was used to ensure, at least, a minimum length in both the article and the abstract. All the articles or summaries that were considered too short were discarded. Obviously, an article or summary too short implies discarding the article-summary pair. The second filter was used to ensure that the summaries were not almost verbatim copies of the first sentences of the articles. To do this, the article-summary pairs in which the overlapping between the summary and the article prefix of the summary length was high
were also discarded. This way, we try to avoid a positional bias in the summaries by discarding those samples in which the summary is reduced to select the first sentences of the article.
Once both filters were applied, we found that some newspaper sources had very few samples, less than 1000 in some cases. To balance the corpus partitions, we decided to remove the sources with few samples from the training, validation, and tests sets. Nevertheless, we joined together the samples from those sources to create a special test set, a test set with sources not present in the training process. Therefore, the corpus consists of four partitions per language: training, validation, and test sets along with an extra test set. Considering all the partitions, the DACSA corpus consists of a set of 725,184 article-summary pairs extracted from 9 different Catalan newspaper websites and 2,120,649 article-summary pairs extracted from 21 different Spanish newspaper websites. The DACSA corpus contains articles and summaries about politics, economics, sports, culture and other topics usually addressed in journalistic domains. To our knowledge, the DACSA corpus is the largest summarization dataset for both languages.
We have used four well known metrics in the summarization field in order to characterize the corpus. These metrics are: extractive fragment coverage and density (Grusky et al., 2018), abstractivity (Bommasani and Cardie, 2020), and novel n-grams (Krysciński et al., 2018). Additionally, for benchmarking purposes, we have evaluated the performance of 6 automatic summarization systems on the DACSA corpus. Concretely, we have used two unsupervised systems (lead-2 and textRank), an extractive summarization system, SHANN (González et al., 2019), two abstractive summarization systems, mBART (Liu et al., 2020) and mT5 (Xue et al., 2020), and one oracle to compute upper bounds of the performance in the DACSA corpus.
The DACSA corpus can be requested for research purposes at https://xarrador.dsic.upv.es/resources/dacsa.
The automatic text summarization problem has been addressed in the literature using abstractive, extractive, or mixed approaches. On the one hand, extractive approaches compose summaries by selecting sentences or words directly from the documents (Cheng and Lapata, 2016; Nallapati et al.,
2017; Liu and Lapata, 2019; Narayan et al., 2018b; Zhang et al., 2018; Dong et al., 2018; Yao et al., 2018; Chen and Bansal, 2018). Most of these approaches address a sequential binary sentence classification problem in order to select the most salient sentences of the documents, following different criteria such as negative log likelihood on preselected sentences (Cheng and Lapata, 2016; Nallapati et al., 2017; Liu and Lapata, 2019) or ROUGE (Lin, 2004) rewards in reinforcement learning environments (Narayan et al., 2018b; Zhang et al., 2018; Dong et al., 2018; Yao et al., 2018). Other extractive architectures are based on siamese hierarchical attention networks built in terms of Long Short Term Memories and Transformer encoders (Gonzalez et al., 2019, 2020). These models have been successfully applied in summarization tasks of Spanish newspapers and talk shows (Gonzalez et al., 2019). On the other hand, the abstractive approaches build the summaries by paraphrasing the sentences of the documents (See et al., 2017; Paulus et al., 2018; Ive et al., 2019). The vast majority of existing neural abstractive summarization models are based on encoder-decoder architectures (Sutskever et al., 2014). Finally, there are also mixed approaches that combine extractive and abstractive techniques, performed in a decoupled way or simultaneously inside the models (Mendes et al., 2019).
Due to the recent success of self-supervised learning, the focus of text summarization research has exhibited a gradual shift from extractive techniques to abstractive techniques (Lewis et al., 2020; Zhang et al., 2020a; Raffel et al., 2020). These kind of objectives allows to pretrain deep architectures (mainly Transformers) to learn vast amounts of general linguistic knowledge from large corpora, that can be transferred to downstream tasks by means of finetuning. The most successful model of this type is BERT (Devlin et al., 2019), that is pretrained with Masked Language Model and Next Sentence Prediction objectives on raw texts from English Wikipedia and BooksCorpus. Based on BERT, some architectural improvements have been proposed like RoBERTa (Liu et al., 2019) or ALBERT (Lan et al., 2020).
In some recent works, BERT and RoBERTa have been finetuned for extractive summarization (Liu and Lapata, 2019; Zhong et al., 2020), but, although it boosted the performance of the previous extractive approaches, the pretraining+finetuning philos
ophy has shown to be most effective for abstractive systems. Nowadays, the best performing abstractive models are BART (Lewis et al., 2020), T5 (Raffel et al., 2020) and PEGASUS (Zhang et al., 2020a), being all of them Transformers (Vaswani et al., 2017) pretrained self-supervisedly as denoising sequence to sequence autoencoders. Some multilingual variants of these models have been recently proposed, mBART (Liu et al., 2020) and mT5 (Xue et al., 2020). Both of them were pretrained following a multilingual denoising procedure on large-scale multilingual corpora. On the one hand, the mBART model was pretrained by using a corpus of 25 languages, extracted from the Common Crawl (Wenzek et al., 2020) (CC25). On the other hand, a multilingual variant of the Colossal Clean Crawled corpus (Raffel et al., 2020) was used to pretrain mT5.
Self-supervised pre-training requires obtaining large amounts of raw data in order to learn good initializations of deep models from denoising objectives. Also, the fine-tuning of these architectures in downstream tasks like text summarization implies the availability of adequate corpora consisting of document-summary pairs. As we mention above, the great majority of datasets for summarization are in English: CNN/DailyMail, NewsRoom, XSUM (Narayan et al., 2018a), and so forth. Although some multilingual datasets have been recently created, as MLSUM, MassiveSumm, and XL-Summ, they do not provide a large enough portion of Spanish data and only MassiveSumm provides a few samples for Catalan. It is in this context where we propose to build the DACSA corpus.
The most used metrics in the literature to quantify the performance of the models in the summarization task are ROUGE (Lin, 2004) and BertScore (Zhang et al., 2020b). On the one hand, ROUGE measures the performance by counting exact matches. On the other hand, BertScore is a more semantic measure which is based on contextual embeddings provided by a BERT language model. These metrics are convenient to evaluate the performance, but they do not explicitly measure the abstractivity. Measuring the abstractivity of the summaries generated by the models is generally not trivial. In this work, we used a set of metrics as abstractivity indicators to assess the level of abstraction: extractive fragment coverage and density (Grusky et al., 2018), abstractivity $_p$  (Bommasani and Cardie, 2020), and novel n-grams (Krysciński
et al., 2018). Additionally, we also used ROUGE and BertScore to compare the different summarization models.
3 Building the DACSA corpus
The DACSA corpus was collected using a distributed web crawler that captured over 6 million news articles, close to 2 million of articles published in Catalan, and more than 4 million written in Spanish. The articles were captured from 30 newspapers sources, 9 sources for Catalan and 21 sources for Spanish. The range of years of publication was between 2010 to 2020.
We divided the crawling process into two services. The first service was designed to retrieve the list of articles on the website of the newspapers source; we refer to this service as the URLs extractor service. The second one aims to extract the content (article content and summary) of the article; we refer to this service as the content extractor service. The whole crawler was developed with Python 3 and JavaScript (Node.js runtime) programming languages.
For the configurations (one per source) of the content extractor service, we used CSS selectors and the library cheerio (https://cheerio.js.org/). In order to capture the article and summary text, we designed the selectors that captured the visible information that a person would read, avoiding metadata. Using visual information instead of metadata is important because we detected that likely the metadata was automatically created by some naive process that could lose information, such as just extracting the first tokens of the article; meanwhile, the visual text is likely complete, readable and coherent.
We searched websites of electronic newspapers published in Spain, in Catalan or Spanish languages. To find the addresses of each article, we decided to use the list of news that electronic newspapers usually have on their website. The benefit of using the list of articles provided by these websites, contrary to the common crawling approach of following every link, was that we aimed the articles themselves, and there was no need to identify whether the web page is a news article or other kind of content. Thus, from the list of news in that newspapers source, we created two configurations, one for the URLs extractor service and another for the content extractor service.
We intended DACSA to be a large-scale, high-
quality corpus for Catalan and Spanish. Thus, after the massive capture of samples, we defined two requirements that the articles and summaries must satisfy. We first established a threshold in the minimum number of words of the article and the summary, and second, a threshold in the maximum similarity between the summary and the first sentences of the article.
On the one hand, we discarded those samples with a short text in the article or the summary. Specifically, every sample inside the corpus contains at least 100 words in the article and 10 words in the summary. With this restriction, we ensure that the samples have enough content to generate a summary with a reasonable length.
On other hand, we rejected from the corpus those samples in which the summary is generated by simply extracting the first sentences of the article. Specifically, we restricted the overlapping between the summary and the starting sentences of the article by using a similarity metric based on the Levenshtein distance to quantify the degree of overlapping. The Equation (1) presents the definition of this metric.
$$
f (A, S) = 1 - \frac {\text {L e v e n s h t e i n} \left(A _ {[ 1 , | S | ]} , S\right)}{| S |} \tag {1}
$$
where  $A$  is the sequence of words of the article text,  $S$  is the sequence of words of the summary,  $|S|$  is number of words of the summary, and Levenshtein is the operation which returns the well-known Levenshtein distance between two texts. In this corpus, we established a maximum threshold of 0.9 of  $f(A,S)$  between the article and the summary.
After the above processes, the DACSA corpus was built. This corpus provides pairs of news article and its summary from different newspapers for both, the Catalan and the Spanish languages. Regarding the Catalan set, there are 725,184 sample pairs from 9 newspapers, regarding the Spanish set, the corpus provides 2,120,649 sample pairs from 21 newspapers.
The amount of samples by newspapers source is far from being homogeneous. If these distributions would be preserved over the partitions (training, validation, and test set), the models will focus their learning in the predominant newspapers. To avoid
Article	Summary
Source	#Docs	Tokens	Vocabulary size	Sents per doc	Words per sent	Vocabulary size	Sents per doc	Words per sent
Training	636,596	316,817,625	1,206,292	17.39	28.62	206,616	1.17	20.36
Validation	35,376	17,831,029	258,999	16.17	31.17	51,940	1.15	20.93
TESTI	35,376	17,704,387	262,148	16.13	31.03	51,958	1.15	20.89
TESTNI	17,836	15,882,219	247,154	35.38	25.17	45,997	1.56	25.93
Set	725,184	368,235,260	1,326,343	17.71	28.67	223,978	1.17	20.59
Table 1: Statistics of the partitions for Catalan.
Article	Summary
Source	#Docs	Tokens	Vocabulary size	Sents per doc	Words per sent	Vocabulary size	Sents per doc	Words per sent
Training	1,802,919	1,172,626,265	2,920,894	23.94	27.17	454,179	1.24	21.99
Validation	104,052	67,669,381	550,213	23.01	28.27	109,460	1.21	23.36
TESTi	104,052	67,363,994	550,910	22.93	28.23	109,706	1.21	23.34
TESTNI	109,626	59,603,306	447,679	16.25	33.46	116,201	1.35	36.84
Set	2,120,649	1,367,262,946	3,189,783	23.44	27.50	516,307	1.24	22.95
Table 2: Statistics of the partitions for Spanish.
this bias and achieve more general models, we propose that the test and validation sets be created in a way that all newspapers have roughly the same number of samples. To achieve this balance, we discarded some sources in order to guarantee that all sources represent at least  $5\%$  of samples in each one of these two sets. Additionally, we discarded those sources that have lower compression ratio than  $10\%$  in their summaries, since we considered these summaries too long compared to their corresponding articles.
The three sets for Catalan (training, validation and test set) are composed by 6 of the 9 newspapers, the training set contains 636,596 samples, and the validation and test sets have 35,376 samples each one. For Spanish, the three sets are composed by 13 of the 21 newspapers, the training set contains 1,802,919 samples, and the validation and test sets have 104,052 samples each one.
All the sources excluded were used as a separate test set. This partition allows evaluating the generalization capabilities of the summarization models against unseen newspaper sources. In this work, we refer to the test set with newspapers included in the training set as TESTI and to the test set that contains newspapers not included in the training set as TESTNI. The statistics of all the sets are shown in Tables 1 and 2.
In the Appendix A, Tables 7 and 8 show the distribution and the average lengths in terms of sentences and words of the articles and their summaries for Catalan and Spanish sets, detailed by the different newspaper sources.
5 Analysis of Dataset
In this section, an analysis of the level of abstractivity of the summaries of the corpus is done. First, the definition of the different measures used in this work is given, and second, we provide the application of these measures to the DACSA corpus.
5.1 Definition of Abstractivity Metrics
We used a set of metrics as abstractivity indicators to assess the level of abstractivity, they capture the degree of text overlapping between the summary and article. In particular, the following metrics have been selected: extractive fragment coverage and density,  $\text{abstractivity}_p$ , and novel n-grams.
Extractive Fragment Coverage (Grusky et al., 2018): the coverage measure quantifies the extent to which a summary is derivative of a text, that is, it measures the percentage of words in the summary that are part of an extractive fragment of the article.
Extractive Fragment Density (Grusky et al., 2018): contrary to the coverage, the density measure takes into account the length of the extractive fragments. A summary might contain many individual words from the article and therefore have a high coverage, however it might have a low density if the extractive fragments are short.
Abstractivity $_p$  (Bommasani and Cardie, 2020): the abstractivity $_p$  metric measures abstractivity as the absence of overlapping between the summary and the original text. Higher values indicate less overlapping and higher abstractivity. The  $p$  parameter weights the length of each extractive fragment,
the higher value of  $p$ , the more the length of the extractive fragment is penalized.
Novel n-grams: (Krysciński et al., 2018) the novel  $n$ -grams metric quantifies the  $n$ -grams introduced in the summary that did not appear in the original text. The value of the metric is a percentage over the total of  $n$ -grams contained in the summary.
Additionally, we also used the Compression Ratio, that is, the ratio between the length of article and summary. Summarizing with higher compression is challenging as it requires capturing more precisely the critical aspects of the article text.
5.2 Dataset Abstractivity
This section presents the results of the abstractivity metrics described in Section 5.1 for the DACSA corpus. The results are shown separately for both languages; Table 3 shows the average values of the partitions for Catalan and Table 4 for Spanish. Tables 9 and 10 in the Appendix B also show these results for each newspaper source.
As Tables 3 and 4 show, the training and validation partitions have a similar type of summaries regarding their degree of abstractivity. The summaries in the test partitions, except the TESTNI set for Spanish, also show similar degree of abstractivity as the previous partitions.