SlowGuess/NAACL_2022_OCR huggingface dataset:
In order to better characterize the corpus, we also present in Figure 1 the distributions of the samples by combining the values of extractive fragment coverage and extractive fragment density of their summaries, and in Figure 2 the distribution of the samples by combining the values of abstractivity $_p$ $(p = 2)$  and novel 2-grams. These plots help to identify visually the degree of abstractivity of the summaries in the Catalan and Spanish sets. On the one hand, the metrics used in the first plots correlate negatively with the abstractivity; thus, higher abstractivity is shown in the partition when the distribution is centered around the bottom left corner of the plot (where the values are lower on both metrics). On the other hand, the second plots correlate positively with the abstractivity; thus, the distributions are centered near the right top corner if the summaries are highly abstractive. Finally, we should point that due to the outliers, the distributions were hard to visualize. Hence, we exclude the  $10\%$  with the lowest values and the  $10\%$  with the highest values.
Figure 1 shows that the Catalan set mainly con
(a) Catalan
(b) Spanish
Figure 1: Distribution of the samples for the Catalan and Spanish sets. x-axis: Extractive Fragment Coverage, y-axis: Extractive Fragment Density.
(a) Catalan
Figure 2: Distribution of the samples for the Catalan and Spanish sets. x-axis: Abstractivity  $p$  (p=2), y-axis: Novel 2-grams.
tains summaries with short extractive fragments since the distribution centers in  $75\%$  of coverage and a density lower than 2. Also, we observe that the distribution tends to go up and right; thus, the samples of the set diversify to less abstractive summaries. In the case of Spanish, we observe that the extractive fragments are longer than in the first language due to the higher density, and also, the distribution centers in the  $85\%$  of coverage, which indicates that the summaries in the Spanish set reuse more words from the article than in the Catalan set. However, the distribution tends to go down and left, which indicates a big presence of abstractive summaries in this set.
Figure 2 helps to show the diversity of the samples by combining  $\text{abstractivity}_p$  ( $p = 2$ ) and novel 2-grams, which brings us more information. Although in Figure 1 the distributions were different from language to language, in this figure, we observe that the two sets are similar regarding these two metrics; note that the darker zones follow the same pattern around the same range of values.
Based on Tables 3 and 4 and Figures 1 and 2, it can be concluded that the DACSA corpus provides samples that do not contain a predominance
Source	Compression	Coverage	Density	\( Abstractivity_p \) (p=2)	Novel 2-grams	Novel 3-grams	Novel 4-grams
Training	23.12	80.87	3.52	84.13	55.55	73.08	81.26
Validation	22.85	81.16	3.96	82.50	54.02	70.99	79.02
TESTI	22.73	81.24	4.01	82.37	53.85	70.74	78.77
TESTNI	24.01	79.98	5.54	83.51	53.55	70.49	78.14
Set	23.11	80.09	3.62	83.95	55.35	72.80	80.96
Table 3: Average values of the metrics in the Catalan partitions.
Source	Compression	Coverage	Density	\( Abstractivity_p \) (p=2)	Novel 2-grams	Novel 3-grams	Novel 4-grams
Training	27.73	82.84	5.64	80.92	51.33	68.74	76.57
Validation	26.32	83.02	5.53	80.07	49.60	66.23	73.94
TESTi	26.20	83.11	5.58	79.92	49.40	65.96	73.64
TESTNI	13.43	72.07	6.37	86.10	59.65	74.01	79.71
Set	26.85	82.31	5.67	81.10	51.58	68.76	76.46
Table 4: Average values of the metrics in the Spanish partitions.
of extractive summaries, and show great diversity regarding their degree of abstractivity.
6 Summarization models and performance results
We evaluate several summarization systems to understand the challenges posed by the DACSA dataset for summarization tasks. We consider both extractive and abstractive models, along with an extractive oracle to show an upper bound of the extractive performance in the corpora.
Extractive systems: Lead-k, TextRank (Mihalcea and Tarau, 2004) and SHANN (González et al., 2019) have been evaluated. Lead-k is a heuristic that extracts the first  $k$  sentences of a text, being especially well suited to summarize newspaper articles. TextRank is a graph-based system inspired by PageRank, where nodes represent sentences, and edges measure similarities in terms of shared words. Finally, SHANN is a supervised system based on siamese hierarchical attentional networks. The document sentences are scored using sentence-level attentions and those with highest scores are extracted to build the summary. As the average number of sentences in the summaries of DACSA is near to two, we extracted two document sentences by using the extractive systems. We built the extractive systems upon code that is available on Github (Barrios et al., 2016), (González et al., 2019).
Abstractive systems: we considered two representative models with high performance on abstractive summarization, based on encoder-decoder architectures with Transformers as backbone: BART
and T5. Due to there are neither BART nor T5 models pretrained from scratch for the Spanish and Catalan languages, we finetuned and evaluated their multilingual variants, mBART $^{1}$  and  $\mathrm{mT5^{2}}$ . It should be noted that, although both of them considered the Spanish language during pretraining, the Catalan language is not represented in the case of mBART, as this language is not contained in the CC25 dataset. We built the abstractive systems using the HugginFace toolkit (Wolf et al., 2020).
Oracle: we implemented an extractive oracle that aligns each summary sentence with the most similar document sentence using ROUGE. The aligned document sentences are concatenated to build the oracle summary.
In order to evaluate the models, we use ROUGE and BERTScore metrics. ROUGE-1, ROUGE-2 and ROUGE-L are reported to measure lexical overlapping, while BERTScore is used to measure semantic similarity.
Tables 5 and 6 show the performance results of the different models on the Catalan and Spanish DACSA TESTI and TESTNI sets in terms of ROUGE and BERTScore metrics. The oracle outperforms the other systems by a large margin. The worse results obtained by the oracle are in the Catalan TESTI, showing that this partition is the most abstractive test partition in the DACSA corpus. Generally, extractive systems are worse in
$^{1}$ HuggingFace finetuned mBART models: ELiRF/mbart-large-cc25-dacsa-ca ELiRF/mbart-large-cc25-dacsa-es
$^2$ HuggingFace finetuned mT5 models: ELiRF/mt5-base-dacsa-ca ELiRF/mt5-base-dacsa-es
Partition	Model	ROUGE-1	ROUGE-2	ROUGE-L	ROUGE-Ls	BERTScore
TESTI	mBART	28.59	11.89	23.00	23.39	72.03
mT5	27.01	10.70	21.81	22.12	71.55
SHANN	23.56	9.64	17.31	19.16	68.85
TextRank	16.54	5.62	11.98	15.33	58.90
Lead-2	23.41	9.33	17.28	19.04	68.96
Oracle	41.68	25.53	36.29	36.64	75.87
TESTNI	mBART	27.46	11.04	21.13	22.01	70.33
mT5	27.00	11.28	21.27	22.01	70.56
SHANN	30.40	9.64	17.31	19.16	69.72
TextRank	17.16	5.83	12.27	15.93	60.36
Lead-2	31.44	15.74	23.63	26.32	70.30
Oracle	47.16	29.44	40.23	41.82	75.86
Table 5: Average  ${\mathrm{F}}_{1}$  scores of the models in the summarization task in Catalan.
Partition	Model	ROUGE-1	ROUGE-2	ROUGE-L	ROUGE-Ls	BERTScore
TESTI	mBART	31.09	13.56	24.67	25.48	72.25
mT5	31.72	14.54	25.76	26.31	72.86
SHANN	26.93	11.84	20.07	22.12	69.86
TextRank	14.13	4.27	8.13	13.15	57.83
Lead-2	29.00	14.39	22.56	24.45	71.03
Oracle	46.04	30.12	40.85	41.37	77.45
TESTNI	mBART	30.66	12.08	23.13	23.89	71.07
mT5	30.61	12.36	23.53	24.05	71.26
SHANN	35.55	15.22	24.63	27.41	70.83
TextRank	21.78	6.13	11.77	18.97	54.54
Lead-2	36.64	16.79	26.07	28.64	71.81
Oracle	46.49	25.50	36.84	37.54	74.85
Table 6: Average  ${\mathrm{F}}_{1}$  scores of the models in the summarization task in Spanish
the TESTI than in the TESTNI, which suggests a higher extractivity in TESTNI than in TESTI. The high results of Lead-2, especially in the TESTNI sets, show that there is a positional bias in these sets.
Languages other than English have a lack of resources for learning models based on deep learning. This is true for endangered languages but it is also true even for those languages that have millions of speakers but are minority worldwide such as Catalan. In this work, we describe the construction of a corpus of Catalan and Spanish newspapers, the Dataset for Automatic summarization of Catalan and Spanish newspaper Articles (DACSA) corpus. We have included an analysis of the corpus using a set of well-known metrics in the summarization field in order to characterize the corpus. This characterization shows that DACSA provides samples that do not contain a predominance of extractive summaries, and show great diversity regarding their
degree of abstractivity. We have also carried out an evaluation of the performance of some extractive and abstractive summarization systems on the DACSA corpus that could be used for benchmarking. To our knowledge, the DACSA corpus is the largest summarization dataset for Catalan and Spanish languages and is freely available for research purposes.
The main objective of this work was to build a quality large-scale corpus that could be used to learn automatic summarization neural models for Catalan and Spanish. To achieve this objective, we selected a set of Spanish news sites, including from Spanish mass media to regional newspapers, and we collected as many data as possible from them. To increase the quality of the corpus, we filtered the article-summary pairs following basic statistics of the text. However, we did not apply any kind of content filtering. Therefore, our filtering could include biased content such as political tendency,
geographic imbalance or gender biases (Stanczak and Augenstein, 2021). A future direction towards improving the dataset quality would be to alleviate that biases, for example, by means of deduplicating content, augmenting artificially the samples to balance gender (Sun et al., 2019), politics, and geographic aspects, or either manually selecting an unbiased subset of the dataset.
The articles collected in the dataset are under Creative Common or private licenses. Nowadays, we are working on obtaining authorization for the distribution of all sources. Those newspaper sources under Creative Common license or the private ones with authorization are freely provided. DACSA can be requested at https://xarrador.dsic.upv.es/resources/dacsa.
This work is part of the AMIC-PoC project (PDC2021-120846-C44), funded by MCIN/AEI/10.13039/501100011033 and by the European Union "NextGenerationEU/PRTR". It is also partially supported by the Vicerrectorado de Investigación de la Universitat Politècnica de València (PAID-11-21).
Federico Barrios, Federico López, Luis Argerich, and Rosa Wachenchauzer. 2016. Variations of the similarity function of textrank for automated summarization. CoRR, abs/1602.03606.
Rishi Bommasani and Claire Cardie. 2020. Intrinsic evaluation of summarization datasets. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8075-8096, Online. Association for Computational Linguistics.
Yen-Chun Chen and Mohit Bansal. 2018. Fast abstractive summarization with reinforce-selected sentence rewriting. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 675-686, Melbourne, Australia. Association for Computational Linguistics.
Jianpeng Cheng and Mirella Lapata. 2016. Neural summarization by extracting sentences and words. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 484-494, Berlin, Germany. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of
deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Yue Dong, Yikang Shen, Eric Crawford, Herke van Hoof, and Jackie Chi Kit Cheung. 2018. Bandit-Sum: Extractive summarization as a contextual bandit. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3739-3748, Brussels, Belgium. Association for Computational Linguistics.
Jose-Angel Gonzalez, Encarna Segarra, Fernando Garcia-Granada, Emilio Sanchis, and Lluís-F Hurtado. 2019. Siamese hierarchical attention networks for extractive summarization. Journal of Intelligent & Fuzzy Systems, 36(5):4599-4607.
Jose Angel Gonzalez, Encarna Segarra, Fernando Garcia-Granada, Emilio Sanchis, and Lluis-F. Hurtado. 2020. Extractive summarization using siamese hierarchical transformer encoders. Journal of Intelligent & Fuzzy Systems, 39:2409-2419. 2.
J.-A. González, L.-F. Hurtado, E. Segarra, F. García-Granada, and E. Sanchis. 2019. Summarization of Spanish talk shows with siamese hierarchical attention networks. Applied Sciences 2019, 9(18).
Max Grusky, Mor Naaman, and Yoav Artzi. 2018. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 708-719, New Orleans, Louisiana. Association for Computational Linguistics.
Tahmid Hasan, Abhik Bhattacharjee, Md. Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M. Sohel Rahman, and Rifat Shahriyar. 2021. XLsum: Large-scale multilingual abstractive summarization for 44 languages. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4693-4703, Online. Association for Computational Linguistics.
Karl Moritz Hermann, Tomáš Kočisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1, NIPS'15, pages 1693-1701, Cambridge, MA, USA. MIT Press.
Julia Ive, Pranava Madhyastha, and Lucia Specia. 2019. Deep copycat networks for text-to-text generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the
9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3225-3234, Hong Kong, China. Association for Computational Linguistics.
Fajri Koto, Joy Han Lau, and Timothy Baldwin. 2020. Liputan6: A large-scale Indonesian dataset for text summarization. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 598-608, Suzhou, China. Association for Computational Linguistics.
Wojciech Krysciński, Romain Paulus, Caiming Xiong, and Richard Socher. 2018. Improving abstraction in text summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1808-1817, Brussels, Belgium. Association for Computational Linguistics.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A lite BERT for self-supervised learning of language representations. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computational Linguistics.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Yang Liu and Mirella Lapata. 2019. Text summarization with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3730-3740, Hong Kong, China. Association for Computational Linguistics.
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pretraining for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726-742.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
Afonso Mendes, Shashi Narayan, Sebastião Miranda, Zita Marinho, Andre F. T. Martins, and Shay B. Cohen. 2019. Jointly extracting and compressing documents with summary state representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 3955-3966.
Rada Mihalcea and Paul Tarau. 2004. Textrank: Bringing order into text. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing.
Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017. Summarunner: A recurrent neural network based sequence model for extractive summarization of documents. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI'17, page 3075-3081. AAAI Press.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018a. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797-1807, Brussels, Belgium. Association for Computational Linguistics.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018b. Ranking sentences for extractive summarization with reinforcement learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1747-1759, New Orleans, Louisiana. Association for Computational Linguistics.
Romain Paulus, Caiming Xiong, and Richard Socher. 2018. A deep reinforced model for abstractive summarization. In Proceedings of the 6th International Conference on Learning Representations ICLR, pages 1-13, Vancouver, BC, Canada. OpenReview.net.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, and Jacopo Staiano. 2020. MLSUM: The multilingual summarization corpus. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8051-8067, Online. Association for Computational Linguistics.
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational
Linguistics (Volume 1: Long Papers), pages 1073-1083, Vancouver, Canada. Association for Computational Linguistics.
Karolina Stanczak and Isabelle Augenstein. 2021. A survey on gender bias in natural language processing. CoRR, abs/2112.14168.
Tony Sun, Andrew Gaut, Shirlyn Tang, Yuxin Huang, Mai ElSherief, Jieyu Zhao, Diba Mirza, Elizabeth Belding, Kai-Wei Chang, and William Yang Wang. 2019. Mitigating gender bias in natural language processing: Literature review. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1630–1640, Florence, Italy. Association for Computational Linguistics.
Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2, NIPS'14, page 3104-3112, Cambridge, MA, USA. MIT Press.
Daniel Varab and Natalie Schluter. 2021. MassiveSumm: a very large-scale, very multilingual, news summarisation dataset. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10150-10161, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS'17, pages 6000-6010, USA. Curran Associates Inc.
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. 2020. CCNet: Extracting high quality monolingual datasets from web crawl data. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 4003-4012, Marseille, France. European Language Resources Association.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua,
and Colin Raffel. 2020. mt5: A massively multilingual pre-trained text-to-text transformer. CoRR, abs/2010.11934.
Kaichun Yao, Libo Zhang, Tiejian Luo, and Yanjun Wu. 2018. Deep reinforcement learning for extractive document summarization. Neurocomputing, 284:52-62.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020a. PEGASUS: Pre-training with extracted gap-sentences for abstractive summarization. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 11328-11339. PMLR.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020b. Bertscore: Evaluating text generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Xingxing Zhang, Mirella Lapata, Furu Wei, and Ming Zhou. 2018. Neural latent extractive document summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 779-784, Brussels, Belgium. Association for Computational Linguistics.
Ming Zhong, Pengfei Liu, Yiran Chen, Danqing Wang, Xipeng Qiu, and Xuanjing Huang. 2020. Extractive summarization as text matching. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6197-6208, Online. Association for Computational Linguistics.
A Statistics of DACSA
We show in Tables 7 and 8 a more detailed view of the statistics of the DACSA corpus, distinguishing among the sources from which it was built. The sources that were only considered in the TESTNI partitions are marked with an asterisk.