# Abstractive vs. Extractive Summarization: An Experimental Review
## 3 Evaluation
### Experimental Setup and Evaluation Results
#### 3.2.2 Evaluation Results
Among the two _TextRank_ variations, the implementation from _pyTextRank_ performed better on this dataset. _e-LexRank_ is the variation of _LexRank_ that uses sentence transformers embeddings, which did not perform better than _LexRank_. As shown in Tables 4 and 5, the top abstractive models scored much higher than the extractive ones on _XSum_. Specifically, _PEGASUS-XSum_ scored the best across all metrics, followed by _BART-large-XSum_ and _distilBART-XSum_-12-6. The _mT5-multilingual-XLSum_ \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline  & **Model** & **R1** & **R2** & **RL** & **RLSum** & **B1** & **B2** & **SB** \\ \hline \multirow{8}{*}{Extractive} & _LexRank_ & **0.3845** & **0.1888** & **0.2460** & **0.2460** & **0.2663** & **0.1815** & **0.1140** \\  & _TextRank_ & 0.3638 & 0.1735 & 0.2168 & 0.2168 & 0.2477 & 0.1651 & 0.1029 \\  & _TextRank_ & 0.3503 & 0.1799 & 0.2320 & 0.2320 & 0.2437 & 0.1629 & 0.1033 \\  & _PositionRank_ & 0.3601 & 0.1686 & 0.2153 & 0.2153 & 0.2436 & 0.1606 & 0.0996 \\  & _e-LexRank_ & 0.3595 & 0.1750 & 0.2205 & 0.2205 & 0.2397 & 0.1620 & 0.1032 \\  & _TopicRank_ & 0.3568 & 0.1763 & 0.2174 & 0.2174 & 0.2334 & 0.1662 & 0.1036 \\  & _Luhm_ & 0.3521 & 0.1812 & 0.2342 & 0.2342 & 0.2349 & 0.1640 & 0.1041 \\  & _LSA_ & 0.3480 & 0.1406 & 0.2115 & 0.2115 & 0.2288 & 0.1380 & 0.0846 \\ \hline \multirow{8}{*}{Abstractive} & _PEGASUS-large_ & **0.3568** & 0.1480 & 0.2268 & 0.2268 & **0.2312** & **0.1400** & **0.0901** \\  & _BART-large-CNN-SAMSum_ & 0.3186 & **0.1571** & **0.2301** & **0.2301** & 0.1333 & 0.0881 & 0.0562 \\  & _distilBART-CNN-12-6_ & 0.3025 & 0.1465 & 0.2161 & 0.2161 & 0.1303 & 0.0836 & 0.0528 \\  & _PCGASUS-_ & 0.2962 & 0.1405 & 0.2116 & 0.2116 & 0.1311 & 0.0832 & 0.0543 \\  & _BART-large-CNN_ & 0.2954 & 0.1433 & 0.2146 & 0.2146 & 0.1168 & 0.0746 & 0.0495 \\  & _PEGASUS-multi_news_ & 0.2801 & 0.0721 & 0.1661 & 0.1661 & 0.1909 & 0.0864 & 0.0406 \\  & _mT5-multilingual-XLSum_ & 0.1486 & 0.0584 & 0.1155 & 0.1155 & 0.0232 & 0.0132 & 0.0081 \\  & _PEGASUS-XSum_ & 0.1440 & 0.0742 & 0.1171 & 0.1171 & 0.0264 & 0.0152 & 0.0129 \\  & _BART-large-XSum_ & 0.1327 & 0.0641 & 0.0988 & 0.0988 & 0.0159 & 0.0093 & 0.0070 \\  & _distilBART-XSum-12-6_ & 0.1237 & 0.0540 & 0.0960 & 0.0960 & 0.0157 & 0.0091 & 0.0065 \\ \hline \hline \end{tabular}
\end{table} 
Table 8: Experimental Evaluation Results on _BillSum_. model has been finetuned on the multilingual \(XLSum\) dataset, which contains similar data in to \(XSum\) in its English part [58]. Most abstractive models, which are not finetuned on \(XSum\), had higher results than the extractive ones, apart from \(PEGASUS\)-_large_ and \(PEGASUS\)-_multi\(\_\)news_, on most metrics. Among the extractive approaches, _TopicRank_ scored the best on every metric except for **B2**, where _TextRank_ (\(sumy\)) received the best score. Among the two _TextRank_ variations, the implementation from \(sumy\) performed better on this dataset. _e-LexRank_ performed slightly better than _LexRank_ in both Tables 4 and 5; however, it was not the top performing extractive approach. As specified in Section 3.2.1, we are conducting the experimental evaluation on the English part of \(XLSum\), which is highly similar to \(XSum\). As shown in Tables 4 and 5, the abstractive models that are finetuned on \(XSum\) have similar or even better performance than the _mT5-multilingual-XLSum_ model. We believe that this better performance of the \(XSum\) models is achieved due to the fact that they were solely trained on English corpora, unlike _mT5-multilingual-XLSum_, which was trained on a multilingual corpus. Among the extractive approaches, _TopicRank_ scored the best across all the evaluation metrics. Among the two _TextRank_ variations, the implementation from \(sumy\) performed better across all metrics.