# Abstractive vs. Extractive Summarization: An Experimental Review
## 4 Conclusions, Open Issues, and Future Work  
This paper has thoroughly evaluated a set of prominent extractive and abstractive summarization approaches on heterogeneous datasets, aiming to comparatively access them and point out their strengths and weaknesses. Our experimentations reveal the following set of conclusions:  
* Abstractive models that are finetuned on data similar to the test data produce significantly better results. The rest of the abstractive models, which are finetuned on data different than the test data, in most cases perform similarly or even better than the extractive ones. There is no clear winner among the _BART_, _PEGASUS_, and _DistilBART_ approaches since none of them outperformed the others across diverse datasets. However, it is worth noting that in some cases the _DistilBART_ student outperformed the _BART_ teacher model. * Regarding the extractive approaches, their evaluation scores were relatively close to each other, so no approach stood out. Among the two _TextRank_ variations, the implementation from _sumy_ performed better on the datasets from which we extracted only one-sentence summaries (_XSum, XLSum and Reddit TIFU_), while the version from _pyTextRank_ performed better on the rest of the chosen datasets. We also tested an embeddings-based extractive approach, _e-LexRank_, which in most datasets did not yield better results than the classical extractive approaches. * Regarding our evaluation metrics, we note that the scores produced by applying _BLEU_ are similar to those produced by _ROUGE_. This leads us to recommend the _BLEU_ metric for the evaluation of summarization approaches, even if its original use concerned the field of machine translation. * One may observe that the scores of **RL** match those of **RLSum** in all datasets, apart from the case of the _CNN/Daily Mail_ dataset. With respect to the datasets that have single-sentence summaries, **RLSum** (summary-level) scores were equivalent to **RL** (sentence-level) scores. For the other datasets, which might contain multi-sentence summaries, this happens because the implementation of **RLSum** that we use splits sentences using the newline delimiter (\(\backslash n\)). * The implication of the aforementioned experimental results is that all abstractive models do not perform equally, as also reported in [21]. Thus, there is a constant need for researchers to discover better pretrained language model architectures that generalize more easily and produce summaries closer to the human style of writing. Our work also revealed a list of open issues in TS, which require further attention; these include:  
* Abstractive models need to be retrained or refinetuned each time documents of different languages or different domains are introduced, respectively. This could be solved by creating more non-English datasets of various domains, and then training and finetuning different versions of the models. * Instead of highly accurate but specialized abstractive models, a generic type of abstractive models could emerge. These could be trained and finetuned on vast multilingual corpora, achieving a degree of generalization that is not present in current models. * Current abstractive approaches require a significant amount of training data [9; 55] and training time, even with specialized hardware. This could be solved through the semisupervised nature of large language models (_LLMs_), e.g., _GPT-3_[52], which, due to their extremely large training and number of parameters (in the billion scale), can be finetuned for a specific language or domain through the utilization of a limited number of examples. * As mentioned in Section 2.4, _BLEU_ can be used as an evaluation metric for the TS task. As presented in Section 3.2.2, this metric produces a ranking for the approaches that is similar to that produced by _ROUGE_. To the best of our knowledge, most research works that evaluate summarization approaches utilize only the _ROUGE_ metric.