# Attentional Extractive Summarization +
## 4 Framework Definition
Therefore, the encoders \(g(\cdot;\theta_{1})\) and \(g^{\prime}(\cdot;\theta_{2})\) are defined as a composition of two levels, \(g=g_{2}(R;\theta_{12})\) and \(g^{\prime}=g_{2}^{\prime}(R^{\prime};\theta_{22})\), where \(R=g_{1}(\cdot;\theta_{11})\) and \(R^{\prime}=g_{1}^{\prime}(\cdot;\theta_{21})\). Because both documents and summaries come from the same domain, they could be represented in the same way through the use of the same set of parameters in both cases, i.e., \(\theta_{11}=\theta_{21}\) and \(\theta_{12}=\theta_{22}\), leading to Siamese architectures. Although this is possible, the \(\theta\) parameters are not constrained to be always shared, so, for the sake of simplicity, we also refer to these architectures as Siamese networks. The parameters of the documents and summaries encoders are defined as \(\theta_{1}=[\theta_{11},\theta_{12}]\) and \(\theta_{2}=[\theta_{21},\theta_{22}]\). As stated before, the document encoder \(g(\cdot;\theta_{1})\) must be interpretable so that it must assign relevance scores both to words, in order to compute sentence representations, and to sentences, in order to compute document representations. Our approach consists in designing these encoders by means of attention mechanisms that assign scores to words and sentences. Then, document representations are computed as an average of their sentence representations, using the document level attention mechanism. At the same time, the sentence representations are computed as an average of their words, using the sentence level attention mechanism. The application of these mechanisms is diverse and they can be applied as auxiliary functions on top of the encoders [37; 38] as in [18] or as main mechanisms to compute representations [21] as in [20]. Let \(r=g(\cdot;\theta_{1})\) and \(r^{\prime}=g^{\prime}(\cdot;\theta_{2})\) be the representations of document and summary respectively, the system must be able to determine if the summary is correct for the document, by using \(r\) and \(r^{\prime}\). In order to do this, a classifier \(c(\cdot,\cdot;\theta_{3})\) whose output is a probability distribution over \(\mathbb{C}\), \(c:\mathbb{R}^{d_{g}}\times\mathbb{R}^{d_{g^{\prime}}}\to\mathbb{R}^{2}\), is applied. Therefore, the model \(f(\cdot,\cdot;\Theta)\) can be  
Figure 1: General scheme of the Attentional Extractive Summarization framework. seen as a classifier \(c(.,. ;\theta_{3})\) applied on top of the encoder outputs, both for document, \(r\), and summary, \(r^{\prime}\), i.e., \(f(.,.;\Theta)=c(r,r^{\prime};\theta_{3})\). The parameters of the model are determined by the parameters of each subpart: encoders for documents and summaries and the classifier, \(\Theta=[\theta_{1},\theta_{2},\theta_{3}]\). The objective is that the model \(f(.,. ;\Theta)\) must be able to classify correctly the largest number of pairs, both the positives (extracted directly from the corpora) and the negatives (for a given document, reference summaries from all the other documents in the corpora, sampled by following a distribution \(p\)). Therefore, the objective is determined by the minimization of Equation (1). \[ \begin{split}\mathcal{L}(\Theta)=\sum_{k=1}^{|\mathcal{D}|}\text {L}(f(X_{k},X_{k}^{\prime};\Theta),y=1)+\\ \operatorname*{\mathbb{E}}_{p(X_{j\
eq k}|X_{k})}[\text{L}(f(X_{ k},X_{j}^{\prime};\Theta),y=0)],\end{split}  \tag{1}\]  
where L is a loss function, and \(\operatorname*{\mathbb{E}}_{p(X_{j\
eq k}|X_{k})}\) denotes expectation with respect to a Bernoulli distribution with parameter \(p\). It is interesting to highlight that, once the system is trained for minimizing the training objective, the encoders \(g(. ;\theta_{1})\) and \(g^{\prime}(. ;\theta_{2})\) must compute proper representations of documents and summaries, respectively. In this way, the document representations, computed from their sentences by using the attention mechanism of \(g_{2}(. ;\theta_{12})\), are useful to distinguish correct and incorrect (document, summary) pairs.