# Attentional Extractive Summarization +
## 5 Proposed Systems
### Siamese Hierarchical Attention Networks  
The Siamese Hierarchical Attention Neural Network (SHA-NN) [18] is an instance of the general attentional framework when the encoders are Hierarchical Attention Networks [19] based on Bidirectional Long Short Term Memory (BLSTM) [33; 39] with attention mechanisms, i.e., \(g_{1}(.;\theta_{11})=\text{BLSTM}_{1}(. ;\theta_{1})\), \(g_{1}^{\prime}(.;\theta_{21})=\text{BLSTM}_{1}(. ;\theta_{1})\), \(g_{2}(.;\theta_{12})=\text{BLSTM}_{2}(. ;\theta_{2})\) and \(g_{2}^{\prime}(.;\theta_{22})=\text{BLSTM}_{2}(.;\theta_{2})\). The BLSTM layers are shared for documents and summaries, both at sentence level (\(\text{BLSTM}_{1}\) with dimensionality \(d_{w}\)) and at document level (\(\text{BLSTM}_{2}\) with dimensionality \(d_{s}\)). However, the attention mechanisms for both branches of the Siamese model are not shared. Regarding classifier \(c\), it is a feed-forward network. The architecture can be seen in Figure 2.  
For this approach, \(R\in\mathbb{R}^{T\times d_{w}}\) and \(R^{\prime}\in\mathbb{R}^{Q\times d_{w}}\) are computed, following Equations (2) and (4), as proposed in [37]. They are the output from the sentence level \(d_{w}\)-dimensional BLSTM\({}_{1}\) with attention, where each row \(i\) is computed as the average of the hidden vectors of the sentence \(i\) attended by \(\alpha\in\mathbb{R}^{T\times W}\) (Equation (3)) and \(\beta\in\mathbb{R}^{Q\times V}\) (Equation (5)) for document and summary respectively. This process is applied independently to each word embedding matrix that represents each sentence both for document and summary (\(R_{i}:1\leq i\leq T\) and \(R^{\prime}_{i}:1\leq i\leq Q\)). The following equations show a sentence encoder composed by \(N=1\) BLSTM network. \[R_{i}=\sum_{j=1}^{W}\text{BLSTM}_{1}(\text{E}(X_{i}))_{j}\cdot\alpha_{ij} \tag{2}\]  
\[\alpha_{ij}=\frac{e^{\tanh(W_{u}\text{BLSTM}_{1}(\text{E}(X_{i}))_{j}+b_{u})} }{\sum_{k=1}^{W}e^{\tanh(W_{u}\text{BLSTM}_{1}(\text{E}(X_{i}))_{k}+b_{u})}} \tag{3}\]  
\[R^{\prime}_{i}=\sum_{j=1}^{V}\text{BLSTM}_{1}(\text{E}(X^{\prime}_{i}))_{j} \cdot\beta_{ij} \tag{4}\]  
\[\beta_{ij}=\frac{e^{\tanh(W_{u}\text{BLSTM}_{1}(\text{E}(X^{\prime}_{i}))_{j} +b_{v})}}{\sum_{k=1}^{V}e^{\tanh(W_{u}\text{BLSTM}_{1}(\text{E}(X^{\prime}_{i} ))_{k}+b_{v})}}, \tag{5}\]  
where \(W_{u}\in\mathbb{R}^{d_{w}}\), \(W_{v}\in\mathbb{R}^{d_{w}}\), are the weights of the attention mechanism for document and summary at word level. From \(R\) and \(R^{\prime}\), \(r\in\mathbb{R}^{d_{s}}\) and \(r^{\prime}\in\mathbb{R}^{d_{s}}\) can be obtained, following Equations (6) and (8), similarly to the sentence level but using BLSTM\({}_{2}\) and the attentions \(\hat{\alpha}\in\mathbb{R}^{T}\) and \(\hat{\beta}\in\mathbb{R}^{Q}\) for document and summary respectively. The following equations show a document encoder composed by the \(\hat{N}=1\) BLSTM network. \[r=\sum_{j=1}^{T}\text{BLSTM}_{2}(R)_{j}\cdot\hat{\alpha}_{j} \tag{6}\]  
\[\hat{\alpha}_{j}=\frac{e^{\tanh(W_{u}\text{BLSTM}_{2}(R)_{j}+b_{u})}}{\sum_{ k=1}^{T}e^{\tanh(W_{u}\text{BLSTM}_{2}(R)_{k}+b_{u})}} \tag{7}\]  
\[r^{\prime}=\sum_{j=1}^{Q}\text{BLSTM}_{2}(R^{\prime})_{j}\cdot\hat{\beta}_{j} \tag{8}\]  
Figure 2: SHA-NN Architecture. \[\hat{\beta}_{j}=\frac{e^{\tanh(W_{g}\text{BLSTM}_{2}(R^{\prime})_{j}+b_{g})}}{\sum_{k =1}^{Q}e^{\tanh(W_{g}\text{BLSTM}_{2}(R^{\prime})_{k}+b_{g})}}, \tag{9}\]  
where \(W_{h}\in\mathbb{R}^{d_{s}}\), \(W_{h}\in\mathbb{R}^{d_{s}}\), are the weights of the attention mechanism for document and summary at document level. These vector representations, \(r\) and \(r^{\prime}\), capture bidirectional relationships among the sentence representations, which are obtained from the representations of their words. Then, they can be used to distinguish correct summaries for documents by forcing the attention mechanisms of the document branch to focus on the most relevant sentences.