# Multistage Mixed-Attention Unsupervised Keyword Extraction for Summary Generation
## 3 MSAMaukee-Summ\({}^{N}\) Model
### N-Coarse-Grained Summary Generation Stage
#### 3.1.2 Coarse-Grained Summary Generation  
During coarse-grained summary generation, the process of keyword extraction and integration is executed iteratively \(N\) times. Keywords are utilized to aid the inference process of summarization; a multistage approach is taken to extract coarse-grained level summary information from long texts. There are five main steps in the process, which are keyword extraction, self-attention accumulation calculation, cross-attention calculation, linear combination score calculation, and keyword incorporation into summary inference. Figure 2: The data filtering and segmentation. (1)  
The keyword extraction task is accomplished by using the keyword extraction module implemented in EmbedRank, which is capable of the lexical annotation of words. The words labeled NN, NNS, NNP, JJ, VB, VBD, etc., are recognized and the keywords are obtained by phrase extraction using Python's own NLTK package. For example, for a given sentence 'The process of creating experiments is minimized by the simple control of running the program code, achieved by editing most of the program's parameters into a text file', the keywords extracted from it are 'program's parameters', 'text file', 'program code', 'running', 'control','minimized', and 'process'. For the keywords initially extracted from the text, it is also necessary to use attention to calculate the importance of the keywords themselves as well as their relevance scores to the context of the sentence and the document, in order to obtain keywords that are of practical value for summary generation. (2)  
In the process of calculating self-attention accumulation for keywords, the input text segments \((D_{1},D_{2}\cdots D_{n})\) are lexically annotated and phrase marked to obtain a keyword set, \(K\). The pre-trained BERT is utilized to extract \(w_{i}^{k}\), which denotes the word self-attention weight of the keyword, \(k\), in the \(i\)-th sentence. The accumulated self-attention weight, \(a_{i}^{k}\), for the keyword, \(k\), in the sentence, \(i\), is then calculated. Finally, the self-attention of the keywords in each sentence is summed to obtain the document-level self-attention value, \(a_{k}\), for the keyword, \(k\). The self-attention accumulation calculation is shown in Figure 3.  
In Figure 3, \(a_{w}\) is obtained from the pre-trained BERT, and \(a_{w}\) denotes the value of the self-attention weights of the words in the sentence; the equation is shown below:  
\[a_{w}=\sum_{w^{\prime}\in s}a_{w^{\prime}w} \tag{1}\]  
where \(a_{w^{\prime}w}\) denotes the value of the attentional weight of word \(w\) together with all the other words, \(w^{\prime}\), in the same sentence, \(s\). Assuming that the sum of the self-attention weight values of keyword \(k\) in sentence \(i\) is \(a_{i}^{k}\), the self-attention weight values of the words in \(k\) are summed to obtain \(a_{i}^{k}\); the equation is shown below:  
\[a_{i}^{k}=\sum_{w\in k}a_{w} \tag{2}\]  
The document-level self-attention weight value, \(a_{k}\), of keyword \(k\) is the sum of the keyword weight values in all sentences; the equation is shown below:  
\[a_{k}=\sum_{i\in d}a_{i}^{k} \tag{3}\]  
where \(d\) denotes the document and sentence \(i\) denotes the \(i\)-th sentence in the document. Figure 3: The self-attention accumulation calculation. The importance of the keywords is taken into account by calculating the self-attention accumulation of the keywords. However, the evaluation of the keywords is multifaceted and needs to be considered regarding the relevance of the keywords to the context of the sentence and the document. To calculate the relevance scores of keywords to sentence context and documents, the network architecture based on the Hierarchical Attention Retrieval (HAR) model and the Bidirectional Attention model is used to design the cross-attention structure. It calculates the relevance between keywords and documents based on the context. A mixture of self-attention and cross-attention is used.