# Sentence Graph Attention For Content-Aware Summarization
## 3 Our Approach: A Model with Graph Sentence-Level Attention
### Sentence-Level Scores  
Inspired by recent works on knowledge bases [9; 10; 13; 14], our idea was to create a knowledge graph (KG) from which the neural network could select the relevant entities to insert in the summary. However, creating a KG for the summarization task was challenging: let us suppose we extracted all the entities present in the text and we created the KG from them, the model could select the entities that were not relevant for the summary, generating a text that was not related to the document's content. As mentioned in the Introduction, we decided to create a latent graph looking at the informativeness of each sentence. The salient words could be copied into the summary via  
Figure 1: Architecture of the model. The _sentence-selection_ method is described in Section 3.1.  
the pointer network [6; 7]; the sentences were then seen as containers whose information must be unveiled. The importance of a sentence to the summary was calculated by comparing it with _both_ the current decoder state _and_ the other sentences. The graph was, in particular, needed to compare the sentences to one another, specifically to understand whether some sentences were more informative than others. For instance, let us suppose that we want to calculate the score for the sentence \(d_{1}\). If we compare \(d_{1}\) with the decoder state \(d\), we obtain a score that highlights the relevance of \(d_{1}\) with respect to the generated summary. Now, let us suppose that we want to calculate the score for the sentence \(d_{2}\), which contains the same facts as \(d_{1}\) as well as some additional ones. If we compare \(d_{2}\) and \(d\), we obtain its relevance with respect to the summary but not with respect to \(d_{1}\). In other words, in this simpler setting, we cannot know whether a sentence is more informative than another one. We thus constructed a graph where the vertexes were the sentences and the edge between two sentences was weighted by their cosine similarity. For simplicity, we treated the current decoder state, i.e., what the model summarized so far, as a sentence. Following the idea of [PERSON] et al. [20], we then applied the PageRank algorithm to the graph to calculate the score of each vertex, assigning high scores to informative and relevant sentences while pushing the scores of redundant sentences down; the pointer network of the model (defined in Equation (7)) then copied the salient words coming from the high-score sentences. Figure 2 depicts an example of the graph score calculation:  
We defined the adjacency matrix \(A\) of the graph via a bi-linear product of the sentences over which we applied the cosine similarity function, similar to the work proposed by [PERSON] et al. [36]. The cosine similarity ensured that the diagonal of \(A\) only contained values equal to 1 (because we were comparing a vector with itself), as required by graph-based models (they require that the identity matrix be added to the adjacency matrix in order to normalize the values). Figure 3 shows an example of a graph and its adjacency matrix. Figure 3: The graph shows an example of a handmade graph with its adjacency matrix obtained by applying the ReLU function over it. Negative weights, i.e., negative cosine similarity scores, were set to zero thanks to the ReLU function, for example, between \(d_{1}\) and \(d_{2}\) (and between \(d_{2}\) and \(d_{4}\)). Figure 2: A graphical example of the score calculation of sentence \(\mathbf{d}_{3}\). Dotted arrows represent the relationship between the current sentence and the other representations. In detail, \(A\) is defined as follows, in which \(\mathbf{d}_{i}\) and \(\mathbf{d}_{j}\) are two sentence representations. \[A[i,j]=cosine(\mathbf{d}_{i},\mathbf{d}_{j}) \tag{10}\]  
We also applied the ReLU function to the matrix \(A\) in order to remove all edges between dissimilar sentences (i.e., those with a negative cosine score).