# Sentence Graph Attention For Content-Aware Summarization
## 3 Our Approach: A Model with Graph Sentence-Level Attention  
Our proposed model is similar to [PERSON] et al. (2019)'s model. It consists of a hierarchical bidirectional LSTM encoder and an attention-based LSTM decoder. In detail, unlike the classic encoder, the hierarchical one is composed of two encoders, each one formed by a bidirectional LSTM. In this paper, we call the first encoder the _word encoder_ and the second one the _sentence encoder_. The model works as follows: first, the word encoder generates sentence representations by reading all the \(M\) words in a sentence; then, the sentence representations are read by the sentence encoder to generate the document representation. Let \(\mathbf{x}_{1}=[x_{1}^{1},x_{1}^{2},\dots,x_{1}^{M}]\) be the tokens that compose the first sentence of the document and \(\mathbf{D}=[\mathbf{x}_{1},\dots,\mathbf{x}_{K}]\) the set of \(K\) sentences that compose the document. At each step \(i\), the word encoder is fed with the input tokens \(x_{1}^{i}\) and produces two encoded states \(\overrightarrow{\mathbf{h}}_{1}^{i}\) and \(\overleftarrow{\mathbf{h}}_{1}^{i}\), where the former is generated by reading the sentence's words from left to right and the latter by reading the sentence's words from right to left. Those two statesare distinguished through the jargon terms _forward state_ and _backward state_. Equation (1) represents the forward and backward states' construction:  
\[ \begin{split}\overrightarrow{\mathbf{h}}_{i}^{j}&= \overrightarrow{LSTM}_{W}(E(x_{i}^{j}),\overrightarrow{\mathbf{h}}_{i}^{j-1} )\\ \overleftarrow{\mathbf{h}}_{i}^{j}&= \overrightarrow{LSTM}_{W}(E(x_{i}^{j}),\overleftarrow{\mathbf{h}}_{i}^{j-1} )\end{split}  \tag{1}\]  
In (1), \(\overrightarrow{LSTM}_{W}\) represents the word-level forward LSTM and \(\overleftarrow{LSTM}_{W}\) the backward one. In addition, \(E(\cdot)\) is a function that returns the word-embed [33; 34] of an input word. Once the encoded states of a sentence are generated, the last forward state \(\overrightarrow{\mathbf{h}}_{i}^{M}\) and the last backward state \(\overleftarrow{\mathbf{h}}_{i}^{1}\) are concatenated together to generate the sentence representation \(\mathbf{h}_{i}\). Such an operation is represented in Equation (2), in which \(||\) is the concatenation operator. \[\mathbf{h}_{i}=[\overrightarrow{\mathbf{h}}_{i}^{M}||\overleftarrow{ \mathbf{h}}_{i}^{1}] \tag{2}\]  
Then, the sentence representations \([\mathbf{h}_{1},\dots,\mathbf{h}_{K}]\) are passed to the sentence encoder, which produces a forward state \(\overrightarrow{\mathbf{d}}_{i}\) and backward state \(\overleftarrow{\mathbf{d}}_{i}\). The document representation \(\mathbf{d}\) is constructed concatenating the last forward state with the last backward one:  
\[\mathbf{d}=[\overrightarrow{\mathbf{d}}_{K}||\overleftarrow{\mathbf{d}}_{1}] \tag{3}\]  
Similar to the encoder, at each step \(t\), the decoder (a single unidirectional LSTM) receives as input the embedding of the previous word from the decoder state \(\mathbf{s}_{t}\) and the context vector \(\mathbf{c}_{t}\), and uses them to emit a word in the output (the network emits a probability distribution over the vocabulary that is used to select the next word). During training, it is the embedding of the target word \(w_{t-1}\), whereas in testing it is the embedding of the previous word emitted by the decoder. In the model, the decoder state is initialized with the document representation. The context vector \(\mathbf{c}_{t}\) is calculated as in [PERSON] et al. [35]:  
\[\mathbf{c}_{t}=\sum_{k=1}^{M}\hat{a}_{k}^{t}\,\mathbf{h}_{k} \tag{4}\]  
where \(\hat{a}_{k}\) represents an attention score calculated via the attention approach and \(\mathbf{h}_{k}\) is the representation of the \(t\)-th word. Inspired by the works of [4; 20; 22], we decided to combine the word-level attention with the sentence-level scores to select only those words coming from sentences that were relevant for the summary.