# Abstractive vs. Extractive Summarization: An Experimental Review
###### Abstract
The above works, however, have a series of limitations: (i) only a few of them [4; 20; 21] evaluated the approaches under consideration through a common evaluation framework (e.g., the _ROUGE_ metric); (ii) only a few of them discussed deep learning approaches [2; 20; 21]; (iii) they did not use alternative evaluation metrics, which yield interesting results, e.g., _BLEU_[23], as discussed in [22; 23; 24]; (iv) they did not provide links to the code repositories of their experimental setups and datasets. Several works reported the importance of developing a comprehensive evaluation framework [2; 3; 18; 21]. Specifically, [PERSON]. [2] stressed the need for the proposal of new solutions regarding the automatic evaluation of TS approaches, while [18] pointed out that the automatic evaluation of TS approaches remains a very promising research area with many open issues. Some of these issues include: (i) the lack of metrics that take into account the mismatch of synonymous terms between human assigned summaries and machine generated ones, (ii) the lack of datasets with quality summaries, and (iii) the lack of datasets for evaluation of multilingual approaches. Taking into account the above remarks and overcoming the associated limitations, this paper aims to broaden and bring up to date our knowledge of the TS task. The overall contribution of this work is fourfold: (i) we thoroughly analyze state-of-the-art TS approaches, which build on recent advances in deep learning and language models and compare their basic techniques; (ii) we evaluate the performance of these approaches using popular evaluation scores, such as _ROUGE-1_, _ROUGE-2_, _ROUGE-L_, _ROUGE-LSUM_, _BLEU-1_, _BLEU-2_, and _SACREBLEU_, while comparing them to the best performing extractive approaches (as a baseline); (iii) we offer a set of insights on current TS approaches, and outline a set of open issues and future research directions; (iv) we make the code used for our experimental setup ([[https://github.com/cmastrokostas/Automatic_Text_Summarization](https://github.com/cmastrokostas/Automatic_Text_Summarization)]([https://github.com/cmastrokostas/Automatic_Text_Summarization](https://github.com/cmastrokostas/Automatic_Text_Summarization)) (accessed on 19 June 2023)) and datasets ([[https://drive.google.com/drive/folders/1](https://drive.google.com/drive/folders/1) UJ_LSZYYm52 CQuURixc7 DHs2h6 Yz50F0?usp=sharing]([https://drive.google.com/drive/folders/1](https://drive.google.com/drive/folders/1) UJ_LSZYYm52 CQuURixc7 DHs2h6 Yz50F0?usp=sharing) (accessed on 19 June 2023)) public, aiming to enhance the reproducibility of our work and facilitate future research efforts. For the needs of this review, we used the search term \"text summarization\" along with the terms: \"extractive\", \"abstractive\", \"method\", \"approach\", \"review\", \"survey\", and \"study\". Our search was conducted in Google Scholar, Elsevier Scopus, and the ACMDigital Library. We narrowed down the findings of our search by only taking into account papers that have received more than 10 citations, or papers from a journal with an impact factor greater than 2.0. A detailed list of the sources (journals, conferences, workshops, repositories) of the 61 publications that satisfied these constraints and were analyzed in this work appears in Appendix A. The remainder of this paper Is organized as follows. Background concepts and related work regarding extractive and abstractive TS approaches are thoroughly reviewed in Section 2. The evaluation of these approaches, as well as the datasets and metrics used, are presented in Section 3. Finally, concluding remarks, open issues, and future research directions are outlined in Section 4.