# Multistage Mixed-Attention Unsupervised Keyword Extraction for Summary Generation
## 3 MSAMaukee-Summ\({}^{N}\) Model
### 1-Fine-Grained Summary Generation Stage
### Evaluation Metrics

ROUGE is used as an evaluation metric in this study, which is mainly categorized into ROUGE-1 (hereinafter referred to as \(R\)-1), ROUGE-2 (hereinafter referred to as \(R\)-2), and ROUGE-L (hereinafter referred to as \(R\)-\(L\)). \(R\)-1 is usually used as the standard evaluation metric because higher scores on the \(R\)-1 metric usually imply higher scores on the \(R\)-2 and \(R\)-\(L\) metrics as well. Meanwhile, \(R\)-1 has a lower time complexity compared to other ROUGE metrics. The equation of definitions for \(R\)-1, \(R\)-2, and \(R\)-\(N\) are shown as follows:

\[R-N=\frac{\sum_{R\in\{ReferenceSummaries\}}\sum_{gram_{n}\in R}Count_{match}(gram _{n})}{\sum_{R\in\{ReferenceSummaries\}}\sum_{gram_{n}\in R}Count(gram_{n})} \tag{20}\]

where \(n\) represents the length of the \(gram_{n}\), \(Count_{match}(gram_{n})\) denotes the maximum number of \(gram_{n}\) that appear in both the candidate and reference summaries, and \(R\) denotes the reference summary. For \(R\)-\(L\), \(L\) is the longest common subsequence (\(LCS\)), and the \(LCS\) is used by \(R\)-\(L\). \(R\)-\(L\) is calculated from \(R_{lcs}\), \(P_{lcs}\), and \(P_{lcs}\). \(R_{lcs}\) and \(P_{lcs}\) denote recall and precision, respectively, while \(\Gamma_{lcs}\) denotes \(R\)-\(L\). \(R\)-\(L\) is defined by Equations (21)-(23) as follows:

\[R_{lcs}=\frac{LCS(X,Y)}{m} \tag{21}\]

\[P_{lcs}=\frac{LCS(X,Y)}{n} \tag{22}\]

\[F_{lcs}=\frac{\left(1+\beta^{2}\right)R_{lcs}P_{lcs}}{R_{lcs}+\beta^{2}P_{lcs}} \tag{23}\] \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Dataset** & **Type** & **Domain** & **Size** & \
\begin{tabular}{c} **Original Text** \\ **Length** \\ \end{tabular} &
\begin{tabular}{c} **Target Text** \\ **Length** \\ \end{tabular} & \(N\) + 1** \\ \hline AMI & Dialogue & Meetings & 137 & 6007.7 & 296.6 & 2 \\ ICSI & Dialogue & Meetings & 59 & 13,317.3 & 488.5 & 3 \\ QMSum & Dialogue & Meetings & 1808 & 9069.8 & 69.6 & 2 \\ SummScreen & Dialogue & TV shows & 26,851 & 6612.5 & 337.4 & 2 \\ GovReport & Document & Reports & 19,466 & 9409.4 & 553.4 & 3 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Statistics of the dataset. where \(LCS(X,Y)\) is the length of the longest common subsequence of \(X\) and \(Y\). \(m\) and \(n\) denote the lengths of the reference and auto-summaries, respectively. Since \(\beta\) is a very large number, \(R\)-\(L\) considers \(R_{lcs}\) almost exclusively. ### Ablation Experiments

To better validate the usefulness of the modules, the following ablation experiments are designed on the relatively long dataset GovReport. Only the sentence filtering layer PureText is added in Experiment 1. Only the mixed-attention keyword extraction module is incorporated to aid in summary inference in Experiment 2. The sentence filtering layer PureText and the mixed-attention keyword extraction module are added in Experiment 3. The mixed-attention keyword extraction module and the self-attention keyword selection module are incorporated in Experiment 4. The sentence filtering layer PureText, the mixed-attention keyword extraction module, and the self-attention keyword selection module are added in Experiment 5. The ablation experiment module selection is shown in Table 3.

The results of ablation Experiments 1, 2, 3, 4, and 5 on the three metrics \(R\)-1, \(R\)-2, and \(R\)-\(L\) are shown in Table 4.

As can be seen from Table 4, the addition of the sentence filtering layer PureText and the mixed-attention keyword extraction module in Experiments 1 and 2, respectively, resulted in an improvement in the R-1, R-2, and R-L metrics compared to Summ\({}^{N}\). The improvement was more obvious when the latter was added alone, with an improvement of 0.46%, 0.92%, and 0.81%, respectively. Sentence filtering provides a streamlined sample of data without any effective processing of keywords.