# Abstractive vs. Extractive Summarization: An Experimental Review
## 3 Evaluation
### Experimental Setup and Evaluation Results
#### 3.2.2 Evaluation Results

Table 6: Experimental Evaluation Results on _Reddit TIFU_. \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline  & **Model** & **R1** & **R2** & **RL** & **RLSum** & **B1** & **B2** & **SB** \\ \hline \multirow{8}{*}{Extractive} & _e-LexRank_ & **0.2762** & **0.0769** & **0.2059** & **0.2059** & **0.1054** & **0.0525** & **0.0393** \\  & _LexRank_ & 0.1654 & 0.0310 & 0.1136 & 0.1136 & 0.0979 & 0.0309 & 0.0183 \\  & _TopicRank_ & 0.1613 & 0.0313 & 0.1112 & 0.1112 & 0.0958 & 0.0316 & 0.0180 \\  & _LSA_ & 0.1622 & 0.0253 & 0.1101 & 0.1101 & 0.0987 & 0.0262 & 0.0173 \\  & _TextRank_ & 0.1570 & 0.0287 & 0.1063 & 0.1063 & 0.0928 & 0.0282 & 0.0172 \\  & _TextRank_ & 0.1471 & 0.0296 & 0.1027 & 0.1027 & 0.0823 & 0.0282 & 0.0154 \\  & _PositionRank_ & 0.1557 & 0.0272 & 0.1073 & 0.1073 & 0.0949 & 0.0270 & 0.0178 \\  & _Luhn_ & 0.1508 & 0.0306 & 0.1049 & 0.1049 & 0.0850 & 0.0292 & 0.0161 \\ \hline \multirow{8}{*}{_Abstractive_} & _BART-large-CNN_ & **0.4004** & **0.2007** & **0.3112** & **0.3112** & **0.2603** & **0.1742** & **0.1165** \\  & _BART-large-CNN_ & 0.3041 & 0.1009 & 0.2272 & 0.2272 & 0.1487 & 0.0793 & 0.0508 \\  & _PEGASUS-_ & 0.2866 & 0.0833 & 0.2220 & 0.2220 & 0.1433 & 0.0697 & 0.0447 \\  & _distilBART-CNN-12-6_ & 0.2884 & 0.0937 & 0.2149 & 0.2149 & 0.1350 & 0.0697 & 0.0466 \\  & _BART-large-XSum_ & 0.2572 & 0.0514 & 0.1878 & 0.1878 & 0.1542 & 0.0460 & 0.0431 \\  & _PEGASUS-large_ & 0.2589 & 0.0628 & 0.2021 & 0.2021 & 0.1008 & 0.0468 & 0.0429 \\  & _distilBART-XSum-12-6_ & 0.2121 & 0.0328 & 0.1561 & 0.1561 & 0.1276 & 0.0275 & 0.0361 \\  & _mT5-multilingual_ & 0.1787 & 0.0235 & 0.1359 & 0.1359 & 0.1082 & 0.0190 & 0.0313 \\  & _PEGASUS-XSum_ & 0.1428 & 0.0228 & 0.1135 & 0.1135 & 0.0779 & 0.0167 & 0.0260 \\  & _PEGASUS-multi\_\_news_ & 0.1127 & 0.0200 & 0.0804 & 0.0804 & 0.0543 & 0.0149 & 0.0103 \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Experimental Evaluation Results on _SAMSum_. As mentioned earlier, the extractive approaches require a specification of the desired number of extracted sentences. To facilitate the fairness of the evaluation, we set this number across all extractive approaches, taking into account the sentence length of the summaries of each specific dataset. For _CNN/Daily Mail_, we set that number to three sentences, due to the average summary length of the dataset. For _XSum_, we set this number to one, since its reference summaries comprise just a single sentence. XLSum has similar summaries, therefore, we used the same setting as _XSum_. For _Reddit TIFU_, we set the number of sentences for the extractive approaches to one, due to the results of our analysis. In the case of _BillSum_, we kept the default output length setting for the extractive approaches, which is four sentences. To evaluate the extractive approaches on _SAMSum_, we set the number of extracted sentences to three, due to the results of our analysis. In the following tables, **R1**, **R2**, **RL**, and **RLSUM** stand for the macro scores of **ROUGE-1**, **ROUGE-2**, **ROUGE-L**, and **ROUGE-LSum**, respectively. Similarly, **B1**, **B2**, and **SB** stand for the macro scores of **BLEU-1**, **BLEU-2**, and **SACREBLEU**, respectively. We bolded the best scoring approaches among both the extractive and the abstractive approaches. We also underlined the second and third best scoring abstractive approaches. As shown in Table 3, the top abstractive models scored higher than the extractive ones on _CNN/Daily Mail_. Specifically, _distilBART-CNN-12-6_ had the best **R1**, **R2**, **B1**, and **B2** scores, while _BART-large-CNN_ had the best **RL**, **RLSum**, and **SB** scores. _PEGASUS-CNN_dailymail_ and _BART-large-CNN-SAMSum_, which were also finetuned on this dataset, had slightly lower scores. Most abstractive models, which were not finetuned on _CNN/Daily Mail_, demonstrated lower scores compared to the extractive ones. Among the extractive approaches, _PositionRank_ scored the best across all evaluation metrics.