# Attentional Extractive Summarization +
## 8 Evaluation
Specifically, it shows the number of samples that each system has seen until convergence, the accuracy on the development set (for each sample in the development set, two samples are built, one positive and one negative randomly sampled), and the time until the convergence. It is possible to see how the SHTE model visited a large number of samples during the training until convergence, at the same time that obtains significantly worse results in terms of accuracy. However, the time required to train these models is significantly lower, requiring up to a four times shorter duration than SHA-NN for the NewsRoom corpus. Furthermore, as Tables 2 and 3 show, the results in terms of ROUGE on both corpora are very similar for both systems. Thus, SHTE constitutes an efficient alternative to SHA-NN since, with a lower training time, obtains very similar results in terms of ROUGE. In comparison to other systems such as BanditSum [12] (76 h in a single NVIDIA Geforce Titan Xp), DQN [13] (10 days on a single NVIDIA GeForce GTX 1080) or Refresh [10] (12 h \"on a single GPU\"), both systems require a significantly lower training time for the CNN/DailyMail corpus. Furthermore, they dispense with the computation of sentence oracles previously to the training step. \begin{table}
\begin{tabular}{l l l l l l l l l l} \hline \hline  & \multicolumn{3}{c}{**NR-Ext**} & \multicolumn{3}{c}{**NR-Mix**} & \multicolumn{3}{c}{**NR-Abs**} \\ \hline System & R-1 & R-2 & R-L & R-1 & R-2 & R-L & R-1 & R-2 & R-L \\ \hline Lead-3 & 51.98 & 47.85 & 51.20 & 25.62 & 13.00 & 22.30 & 14.57 & 2.62 & 11.73 \\ SHA-NN-3 & 48.29 & 43.54 & 47.42 & 24.62 & 12.32 & 21.37 & 14.22 & 2.57 & 11.43 \\ SHTE-3 & 48.62 & 43.35 & 47.65 & 24.76 & 12.20 & 21.43 & 14.33 & 2.53 & 11.51 \\ Lead-2 & 57.87 & 53.03 & 56.83 & 28.60 & 14.33 & 24.46 & 15.68 & 2.77 & 12.35 \\ SHA-NN-2 & 54.83 & 49.25 & 53.72 & 28.03 & 13.79 & 23.85 & 15.67 & 2.74 & 12.29 \\ SHTE-2 & 53.97 & 47.87 & 52.59 & 27.78 & 13.41 & 23.56 & 15.57 & 2.67 & 12.22 \\ ECS-Ext & **69.40** & **64.30** & **68.30** & **31.90** & **16.30** & 26.90 & 17.20 & 3.10 & 13.60 \\ PGen + Cov & 39.10 & 28.00 & 36.20 & 25.50 & 11.00 & 21.10 & 14.70 & 2.30 & 11.40 \\ TLM & 53.30 & 44.20 & 50.10 & 28.10 & 12.10 & 23.00 & **18.50** & **3.90** & **14.70** \\ ECS-Comp & 68.40 & 62.90 & 67.30 & 31.70 & 16.10 & **27.00** & 17.10 & 3.10 & 14.10 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Results on the three test subsets of NewsRoom (Extractive, Mixed, and Abstractive). The suffix of Lead, SHA-NN, and SHTE models refers to the number of extracted sentences, \(k\). Best results are in bold. It is interesting to observe in Table 5 that, in spite of the significant differences in terms of accuracy during the evaluation with the development set, the results in terms of ROUGE in the evaluation of the test summaries are very similar. This clearly illustrates the mismatch discussed by [PERSON] et al. in [10], derived from the disconnection between the task definition and the training objective. This is the main drawback of the summarization systems based on optimizing the cross-entropy instead of the ROUGE measure. Due to this reason, it is interesting to search alternatives to Reinforcement Learning in order to optimize directly the evaluation measure. Finally, despite of the systems based on the attentional framework obtained similar results to Lead in both corpora (mainly due to the bias to the first article sentences in almost all the extractive samples), those systems are capable of generalizing on unseen documents where the sentences are more scattered, as shown in [PERSON] et al. [45].