# An Abstractive Summarization Model Based on Joint-Attention Mechanism and a Priori Knowledge
## 3 Model
### Joint-Attention Mechanism for Word-Sentence-Level Fusion  
In this paper, each word is used as input to the encoder, and after GRU training, the relationship between word-level vectors is obtained. Distinguish from the relationship between sentence-level vectors mentioned later. GRU suffers from the long-range dependency problem, i.e., when the input is too long, the later input words will forget the previously input words, which in turn causes incomplete information between words, resulting in incomplete features received by the decoder. In this paper, each sentence in the paragraph is subjected to sentence embedding processing, as follows: The vector obtained by summing and averaging the word-embedding of all the words in the sentence is used as the final sentence embedding. After this processing, there will be relationships not only between word-level vectors, but also between sentence-level vectors, and the decoder discriminates  
Figure 6: Normalization. Figure 7: Weighted summation. the word-level sentence-level vectors based on their relationship with the hidden state of the decoder. The specific formula is as follows. \[d_{j}^{w}=GRU(y_{j-1},d_{j-1}^{w},c_{j}^{w}) \tag{9}\]  
\[e_{jn}=V_{a}^{T}tanh(W_{a}d_{j-1}^{w}+U_{a}h_{n}^{{}^{\prime}}) \tag{10}\]  
\[\alpha_{jn}=\frac{exp(e_{jn})}{\sum_{k=1}^{n}exp(e_{jk})} \tag{11}\]  
\[c_{j}^{w}=\sum_{p=1}^{n}\alpha_{jn}h_{p}^{{}^{\prime}} \tag{12}\]  
\[d_{j}^{s}=GRU(y_{j-1},d_{j-1}^{s},c_{j}^{s}) \tag{13}\]  
\[e_{jm}=V_{a}^{T}tanh(W_{a}d_{j-1}^{s}+U_{a}S_{m}) \tag{14}\]  
\[\alpha_{jm}=\frac{exp(e_{jm})}{\sum_{k=1}^{n}exp(e_{jk})} \tag{15}\]  
\[c_{j}^{s}=\sum_{q=1}^{m}\alpha_{jm}S_{q} \tag{16}\]  
Equations (9)-(12) is the process of obtaining word-level context vectors \(c_{j}^{w}\), and Equations (13)-(16) is the process of obtaining sentence-level context vectors \(c_{j}^{s}\). Taking the word-level context vector \(c_{j}^{w}\) calculation process as an example: First, the decoder generates a hidden vector \(d_{j}^{w}\) based on the hidden vector of the previous neuron \(d_{j-1}^{w}\), the output of the previous neuron \(y_{j-1}\), and the context vector of the current neuron \(c_{j}^{w}\). Second, the \(j^{th}\) neuron of the decoder calculates the correlation \(e_{jn}\) between the current state and each neuron in the encoder \(h_{n}^{{}^{\prime}}\) based on the hidden vector of the previous neuron \(d_{j-1}^{w}\). Third, obtain the generated word weight score \(\alpha_{jn}\); Finally, the word-level context vector \(c_{j}^{w}\) is obtained. Similarly, the sentence-level context vector \(c_{j}^{s}\) is obtained. Where \(V_{a}\), \(U_{a}\) are the parameter vectors. \(n\) means that there are \(n\) words in the paragraph and \(m\) means that there are \(m\) sentences in the paragraph (i.e., assuming that each input text has m sentences). \(d_{j}\) is the hidden vector generated by the decoder based on the hidden vector \(d_{j-1}\) of the previous neuron, the output \(y_{j-1}\) of the previous neuron and the context vector \(c_{j}\) of the current neuron. \(d_{j}^{w}\) is the hidden vector in word-level dimension and \(d_{j}^{s}\) is the hidden vector in sentence-level dimension. \(h_{n}\) is the output of each neuron of the encoder, and \(S_{m}\) is the sentence-level representation computed from each word-level output. The jth neuron of the decoder has to calculate the correlation \(e_{jn}\) of the current state with each neuron in the encoder based on the hidden vector \(d_{j-1}\) of the previous neuron. The sentence-level relevance vector is denoted as \(e_{jm}\). If the \(n^{th}\) dimension of \(e_{jn}\) is larger, it means that the current node is more correlated with the nth neuron of the encoder. \(\alpha_{jn}\) is the generated word weight score and \(\alpha_{jm}\) is the generated sentence weight score. As a result, the word-level context vector \(c_{j}^{w}\) and the sentence-level context vector \(c_{j}^{s}\) are obtained.