# Abstractive vs. Extractive Summarization: An Experimental Review
## 2 Related Work
### Abstractive Approaches  
The need for abstractive approaches resulted from a major drawback of extractive approaches, which is a lack of readability and coherence of the produced text, since extractive approaches utilize simple heuristics to extract and concatenate the most relevant sentences, without accounting for grammatical or syntactical rules [18]. To generate a fluent and coherent summary, more contextual information about the tokens of the input text is required, thus a family of models that generate new phrases in a similar manner to the paraphrasing process of a human reader is needed [2; 4]. Many models for abstractive summarization have already been proposed in the literature. As seen in a recent survey [2], these include graph-based [41], rule-based [42], and semantic modelling [43] approaches. These earlier models, however, do not utilize recent advancements in deep learning, which improve many NLP tasks. Newer abstractive summarization approaches build on deep learning models, including: (i) the _convolutional neural networks_ (_CNN_) and the _recurrent neural networks_ (_RNNs_); (ii) _LSTM_ and _GRU_, which improve the original _RNNs_ and are discussed in [44]. Other neural architectures that are not based on _CNNs_ and _RNNs_ include _GAN_ (generative adversarial networks). Certain works use these to build their abstractive approaches, as described in [45; 46]. However, these yield lower evaluation scores (i.e., _ROUGE_) than recent deep learning models, which rely on the model explained in the next paragraph, as validated in [21]. _Transformer_[47] is a deep learning model that consists of a series of encoder and decoder layers, which utilize the attention mechanism to model the global dependencies of sequential data [48]. Specifically, the self-attention mechanism assigns different weights to different parts of the input, according to their contextual significance. These are encoded in hidden state layers when generating the output sequence. In addition, _Transformer_ models use multi-head attention, which means that attention is applied in parallel to capture different patterns and relationships of the input data. _Transformer_ uses the encoder-decoder model, which encodes information into hidden layers and then decodes it to generate output. These models are semisupervised, due to their unsupervised pretraining on large datasets, followed by supervised finetuning. Approaches built on this model achieve state-of-the-art performance on various text generation tasks, including abstractive summarization. Recent surveys [20, 21] discussed and evaluated the differences between earlier abstractive approaches, including those that utilize deep learning models proposed before the introduction of the _Transformer_ architecture. It is stressed here that the work presented in this article focuses on prominent pretrained language model approaches, which rely on the _Transformer_ model and are discussed below. \(T\)5 [49], which stands for _text-to-text transfer transformer_, is an approach that closely follows the _Transformer_ architecture. It provides a general framework which converts multiple NLP tasks into sequential text-to-text ones. To address each task, it uses a task-specific prefix before the given sequence in the input. The pretraining process comprises both supervised and unsupervised training. The unsupervised objective of the approach includes masking random spans of tokens with unique sentinel tokens. The \"corrupted\" sentence is passed to the encoder, while the decoder learns to predict the dropped-out tokens on the output layer. A follow up approach, namely _mT_5 [50], builds on \(T\)5 to provide multilingual pretrained baseline models, which can be further finetuned to address diverse downstream tasks in multiple natural languages. _BART_[51], which stands for _bidirectional auto-regressive transformers_, is a multitask deep learning approach, with abstractive summarization being included in them.