# An Abstractive Summarization Model Based on Joint-Attention Mechanism and a Priori Knowledge
## 3 Model
### Encoding Side  
To solve the RNN long-range dependency problem, a deformed long short-term memory (LSTM) or gated recurrent unit (GRU) of RNN is usually used in the codec model. Compared with LSTM, GRU is easier to train and can improve the training rate to a great extent. Therefore, GRU is used as the encoder in this paper. GRU is defined as follows.  
\[u_{t}=\sigma(W_{u}[w_{t},h_{t-1}]) \tag{1}\]  
\[r_{t}=\sigma(W_{r}[w_{t},h_{t-1}]) \tag{2}\]  
\[h_{t}=\tanh(W_{h}w_{t}+U_{h}(r_{t}\odot h_{t-1})) \tag{3}\]  
\[h_{t}=(1-u_{t})\odot h_{t-1}+u_{t}\odot\dot{h}_{t} \tag{4}\]  
where \(W_{u}\), \(W_{r}\), and \(W_{h}\) are the weight matrices, \(w_{t}\) is the current input, and \(h_{t}(t-1)\) is the hidden state vector passed in from the previous time node, and this hidden state vector contains the relevant information of the previous node.  
In this paper, we use a bidirectional GRU(Bi-GRU), as shown in Figure 4, to retain both forward and backward time step information. The forward network reads the word-embedding vector from left to right to obtain the forward GRU hidden state \((\overset{\rightarrow}{h}_{1},\overset{\rightarrow}{h}_{2},...,\overset{ \rightarrow}{h}_{n})\), and the backward network reads the word-embedding vector from right to left to obtain the backward GRU hidden state \((\overset{\leftarrow}{h}_{1},\overset{\leftarrow}{h}_{2},...,\overset{ \leftarrow}{h}_{n})\). The two hidden state formulas are as follows.  
\[\overset{\rightarrow}{h}_{t}=GRU(w_{t},\overset{\rightarrow}{h}_{t-1}) \tag{5}\]  
\[\overset{\leftarrow}{h}_{t}=GRU(w_{t},\overset{\leftarrow}{h}_{t+1}) \tag{6}\]  
The initial state of Bi-GRU is set as a zero vector, i.e., \(\overset{\rightarrow}{h}_{1}=0\), \(\overset{\leftarrow}{h}_{n}=0\). The forward and backward hidden states are cascaded to obtain the final hidden state. Expressed as: \(h_{t}=[\overset{\rightarrow}{h}_{t},\overset{\leftarrow}{h}_{t}]\).  
The self-attention mechanism can determine how relevant the current word is to other words in the sentence. Less reliance on external information and better at capturing the internal relevance of data or features. Inspired by this, we calculate the correlation between the hidden vectors trained by the neural network and the source text word vectors to avoid deviating from the original meaning. In addition, it is also possible to give higher weight to keywords, making the decoder focus on important information in the decoding process. The process is as follows.  
1. First, obtain the similarity matrix, as shown in Figure 5:
2. Normalization, as shown in Figure 6:  
Figure 4: Bi-GRU.  
Figure 5: Similarity matrix.  
The final result is:  
\[H^{{}^{\prime}}=softmax(\frac{HW^{T}}{\sqrt{d_{w}}})H \tag{7}\]  
\(H\) is the matrix corresponding to the hidden vector and W is the matrix corresponding to the word vector. To prevent the gradient from disappearing, divide the inner product by the square root of the dimensions.  
The \(h^{{}^{\prime}}\) vector after assigning weights is used for the representation of sentence vectors, meaning that to provide high-quality summaries, the decoder pays attention to both the source text's keywords and the sentences in which they appear. The word-embedding averaging method by [PERSON] et al. (2018) works well for sentence embedding across different domains and summarizer architectures. We obtain the sentence embedding, i.e., \(S^{{}^{\prime}}_{m}\), by this simple method. As shown in Equation (8).  
\[S^{{}^{\prime}}_{m}=\frac{1}{n}\sum_{t=1}^{n}h^{{}^{\prime}}_{t} \tag{8}\]