# Sentence Graph Attention For Content-Aware Summarization
## 4 Evaluation
### Results  
Table 2 shows the Rouge scores (Rouge-1, Rouge-2, and Rouge-L) [41] of the models. Our models _\"our model + coverage\"_ and _\"our model + coverage + covscore\"_ obtained higher R-1 and R-2 scores than [PERSON] et al. 's models. We suspect that our models had a low R-L due to their high abstraction power since the generated summaries contained tokens that were not present in the reference one; the longest common sub-sequence was short. \begin{table}
\begin{tabular}{c c} \hline \hline
**Parameter** & **Value** \\ \hline Embedding size & 128 \\ \hline Hidden size & 256 \\ \hline \(\lambda\) & 0.9 \\ \hline Gradient clipping & 2.0 \\ \hline Dropout & 0.2 \\ \hline Optimizer & Adagrad \\ \hline Learning rate & 0.15 \\ \hline Batch size & 8 \\ \hline Beam size & 5 \\ \hline [PERSON] et al. [21]’s \(\alpha\) & 1.4 \\ \hline \# words per sentence & 50 \\ \hline \# sentences & 8 \\ \hline Decoder output length & 100 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: The hyperparameters (i.e., parameters set by the user) and their values. We noticed that the auxiliary loss did not improve the Rouge scores. Since the auxiliary loss modified the attention scores, it also impacted on the sentence-level scores and shook the sentence representations, invalidating part of the training. This was supported by the fact that the covloss incremented the loss in the validation set from 3.5 to 3.8 rather than decreasing it. In light of these results, we can conclude that the proposed graph-based sentence-level attention generally performed better than the sentence-level attention proposed by [PERSON] et al. [4]. It allowed for the generation of more abstractive summaries, as reported in Section 4.3, at the cost of sacrificing recall. In more detail, since the Rouge score only evaluates the presence of tokens in the generated summary of the reference model, it indirectly penalizes models that use synonyms and periphrasis. Indeed, our best model had very high precision (41.76 for R-1, 16.27 for R-2, and 27.16 for R-L) but very low recall (33.22 for R-1, 13.03 for R-2, and 21.50 for R-L). This is also the reason why our models had lower Rouge scores than those of [PERSON] et al. [7], whose models were more oriented toward copying words from the input document than generating novel ones. Finally, our models underperformed with respect to those in [PERSON] et al. [22] and [PERSON] et al. [20]. This was expected since their models had a more complex architecture and used more complex features. The model of [PERSON] et al. [22] is composed of two trained models: an extractive model to select the sentences that are useful for the summary and an abstractive one to digest them; our models, instead, were jointly trained to compute a score for each sentence and generate a summary. The model of [PERSON] et al. [20] first generates a sentence decoder state then generates the words of a sentence using the word-level attention and the pointer network. Compared to our models, which only combine both attentions and require fewer resources to be trained, theirs is resource-intensive because the hierarchical decoder adds a further million parameters, having a positive impact on the results. Figure 4 shows an example of the generated summaries. From the table, it can be seen that our model produced a good summary. There was only an error caused by the phrase \"_somalia's internationally recognized government that had been under pressure from al-shabaab_\" because it was attached to the previous sentence, which was semantically distinct from the latter phrase. The adoption of the coverage vector corrected the error and the use of the coverage loss slightly improved the summary. The use of the _covscore_ function modified part of the summary, including more details about the \"beliefs\" of the terrorist group. Figure 5 shows the average percentage of duplicate n-grams. The measure gives an insight into both the reasons behind the low Rouge F1 scores and the impact of the coverage