# Attentional Extractive Summarization +
## 8 Evaluation
\begin{table}
\begin{tabular}{l l l l l l} \hline
**System** & **Strategy** & **Category** & **R-1** & **R-2** & **R-L** \\ \hline Lead-3 & Ext & Heuristic & 40.24 & 17.70 & 36.45 \\ SHA-NN-3 & Ext & Attentional & 39.99 & 17.75 & 36.27 \\ SHTE-3 & Ext & Attentional & 39.96 & 17.60 & 36.19 \\ SummaRunner & Ext & Oracle & 39.60 & 16.20 & 35.30 \\ ECS-Ext & Ext & Oracle & 41.70 & 18.60 & 37.80 \\ BertSumEXT & Ext & Oracle & **43.25** & **20.24** & **39.63** \\ Refresh & Ext & Reinforcement & 40.00 & 18.20 & 36.60 \\ DQN & Ext & Reinforcement & 39.40 & 16.10 & 35.60 \\ Latent & Ext & Reinforcement & 41.10 & 18.80 & 37.40 \\ BanditSum & Ext & Reinforcement & 41.50 & 18.70 & 37.60 \\ ECS-Comp & Mix & Oracle & 40.90 & 18.00 & 37.40 \\ Latent-Comp & Mix & Reinforcement & 36.70 & 15.40 & 34.30 \\ PGen + Cov & Mix & Text generation & 39.53 & 17.28 & 36.38 \\ CopyCat & Mix & Text generation & 39.15 & 17.60 & 36.17 \\ \hline \end{tabular}
\end{table} 
Table 2: Results on CNN/DailyMail corpus for full-length Rouge. The suffix of Lead, SHA-NN, and SHTE models refers to the number of extracted sentences, \(k\). Best results are in bold. \begin{table}
\begin{tabular}{l l l l l l} \hline
**System** & **Strategy** & **Category** & **R-1** & **R-2** & **R-L** \\ \hline Lead-3 & Ext & Heuristic & 30.66 & 21.09 & 28.35 \\ SHA-NN-3 & Ext & Attentional & 28.99 & 19.42 & 26.69 \\ SHTE-3 & Ext & Attentional & 29.19 & 19.37 & 26.81 \\ Lead-2 & Ext & Heuristic & 33.98 & 23.30 & 31.14 \\ SHA-NN-2 & Ext & Attentional & 32.78 & 21.86 & 29.85 \\ SHTE-2 & Ext & Attentional & 32.38 & 21.25 & 29.40 \\ ECS-Ext & Ext & Oracle & **39.50** & **27.90** & **36.26** \\ PGen + Cov & Mix & Text generation & 26.43 & 13.76 & 22.90 \\ TLM & Mix & Text generation & 33.30 & 20.06 & 29.26 \\ FastRL & Mix & Reinforcement & 21.93 & 9.37 & 19.61 \\ ECS-Comp & Mix & Oracle & 39.06 & 27.36 & 36.13 \\ \hline \end{tabular}
\end{table} 
Table 3: Results on the full test of NewsRoom. The suffix of Lead, SHA-NN, and SHTE models refers to the number of extracted sentences, \(k\). Best results are in bold. It is possible to observe how extracting a number of sentences similar to the reference summary length (1.4 as shown in Table 1) improves notably the performance of the systems (\(k=2\) instead of \(k=3\)). This behavior is observed especially in the NR-Ext and NR-Mix subsets, in comparison to the NR-Abs subset. This suggests that, when the reference summaries are extractive, in addition to determine the relevance of each sentence, it is also important to adjust correctly the length of the summaries. However, when the reference summaries are abstractive, the results by using \(k=2\) and \(k=3\) are very similar and clearly lower for all the systems. These bad results are due to the abstractiveness nature of this set of reference summaries, taking into account that the systems are extractive and mixed. Additionally, it is interesting to highlight that, although Lead is a robust baseline in the NR-Ext and NR-Mix subsets, it is not so good in the NR-Abs subset, where our systems obtain almost the same results. In both cases, the results obtained by our systems are better than those obtained by widely used approaches such as Pointer-Gen+Cov [31] or by Reinforcement Learning systems such as FastRL [14; 43]. Additionally, they obtain better results than TLM [44] in terms of ROUGE-2 and ROUGE-L on the full dataset, in spite of this system stands out in the abstractive subset NR-Abs. The only systems that consistently outperform the Lead systems are those based on ExConSumm [30] (both in the extractive and mixed variants), mainly due to they largely outperform the results on NR-Ext and NR-Mix subsets. Differently from our systems, these systems are able to generate variable-length summaries depending on the input text. In Table 5, several details about the convergence of our systems are shown.