# Multistage Mixed-Attention Unsupervised Keyword Extraction for Summary Generation
## 3 MSAMaukee-Summ\({}^{N}\) Model
### 1-Fine-Grained Summary Generation Stage
TextRank [37]: A graph ranking approach is utilized by TextRank to consider the importance of each node's information. It recursively calculates the importance weight value of each node in the relationship graph from the global semantic information. It counts and ranks important node information for keyword extraction and text summarization. 3. HAT-BART [16]: Layered attention is applied by HAT-BART to Transformer. Visual Hierarchical Encoder-Decoder Attention is designed to overcome the uneven distribution of important information in long texts. Hierarchical Attention Transformer's (HAT) architecture is used to adapt to longer text inputs. 4. DDAMS [38]: Relational maps are used by DDAMS to model different discourse relationships; the interactions between discourses in a meeting are explicitly modeled. A relational graph encoder module is constructed, and graph interaction is used to model the interactions between discourses. The semantic relationships and information structure in the summaries are clarified. 5. Summ\({}^{N}\)[17]: The data samples are segmented by Summ\({}^{N}\). It generates coarse-grained summaries through multiple stages, based on which fine-grained summaries are generated. The framework can handle input text of any length by adjusting the number of stages. The results of the ROUGE metric values of the MSMAUKE-Summ\({}^{N}\) model with HMNet, TextRank, HAT-BART, DDAMS, and Summ\({}^{N}\) on the AMI, ICSI, and QMSum datasets are shown in Table 5.

For some special datasets, such as the SummScreen dataset, in this paper, the neural model Longformer + ATT and the hybrid model NN + BM25 + Neural proposed by [PERSON] et al. [35] are used as comparison models and experimented on the SummScreen dataset. In addition, for the GovReport dataset, the comparison models are divided into two main categories, one of which is the BART variant, which represents the self-attentive variant with full attention. Another category is BART HEPOS, denoting the encoder variant with head position step encoder-decoder attention. The results of the ROUGE metric values of the MSMAUKE-SummN and other models on the SummScreen and GovReport datasets are shown in Tables 6 and 7, respectively. As can be seen in Figure 5, the model is significantly improved after adding the sentence filtering layer, the mixed-attention keyword extraction module, and the self-attention keyword selection module. The improvement effect is more obvious in Figure 5b compared to (a) because the ICSI dataset is relatively large and more information can be provided for summary generation. Meanwhile, MSMAUKE-SummN outperforms models such as SummN in all categories of ROUGE because sentence filtering long texts results in high-quality data samples, which helps to improve the quality of summary generation. The importance of keywords at the sentence level and document level can be paid full attention by the mixed-attention keyword extraction module. The extracted keywords are used in the stage of coarse-grained summary generation. The inference process of summary generation is enhanced, and the information extraction ability in the coarse-grained stage is better improved. In addition, keywords with higher weights are used by the self-attentive keyword selection module in the inference process of fine-grained summary generation. It \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**ModelDataset** & \multicolumn{3}{c}{**SummScreen-FD**} & \multicolumn{3}{c}{**SummScreen-TMS**} \\
**R-1 (\%)** & **R-2 (\%)** & **R-L (\%)** & **R-1 (\%)** & **R-2 (\%)** & **R-L (\%)** \\ \hline Longformer + ATT & 25.9 & 4.2 & 23.8 & 42.9 & 11.9 & 41.6 \\ NN + BM25 + Neural & 25.3 & 3.9 & 23.1 & 38.8 & 10.2 & 36.9 \\ SummN & 32.48 & 5.85 & 27.55 & 44.64 & 11.87 & 42.53 \\
**MSMAUKE-SummN** & **34.34** & **6.86** & **28.98** & **46.38** & **12.64** & **44.74** \\ \hline \hline \end{tabular}
\end{table} 
Table 6: ROUGE metric values on the SummScreen dataset.