## 3 Topic-Oriented Multi-Document Summarization (TOMDS) Model
### Abstractive Stage
#### 3.3.5 Topic Attention Mechanism  
In the decoder, it is necessary to pay attention not only to the information of the summary but also to the topic information input by the user. For the information of the summary itself, we can learn from the self-attention mechanism of the decoder part of the transformer model. For the word-level and paragraph-level representation information output by the encoder and the given topic information, this article proposes a topic attention mechanism to enable the model to simultaneously focus on the multi-granularity representation information output by the encoder and the given topic information. The topic attention mechanism mainly includes the topic attention module, the global paragraph attention module, and the local paragraph attention module. The structure is shown in Figure 6.  
In the topic attention module, the contextual information of the tokens in the summary \(S\) is encoded. If we define the input token as \(x_{i}\)\(\in\)\(S\)(i = 1, 2,.., ), then the attention head \(head_{j}\)\(\in\)\(Head\)(j = 1, 2,.., 2), and then we obtain  
\[\begin{array}{l}q_{i,\ headj}=W^{q_{i,\ headj}}.x_{i}\\ k_{i,\ headj}=W^{k_{i,\ headj}}.x_{i}\\ v_{i,\ headj}=W^{v_{i,\ headj}}.x_{i}\end{array} \tag{29}\]where \(W^{y_{i,~{}head1}}\), \(W^{k_{i,~{}head2}}\), \(W^{v_{i,~{}head2}}\), \(W^{v_{i,~{}head2}}\)\(\in\)R\({}^{narm}\) are the weight matrices. \(q_{i,~{}head1}\), \(k_{i,~{}head2}\), \(v_{i,~{}head2}\)\(\in\)R\({}^{narm}\) are the sub-query, sub-key, and sub-value in different attention heads. After connecting them, we obtain, respectively,  
\[\begin{array}{l}q_{i}=concat(q_{i,~{}head1}\cdot q_{i,~{}head2},\cdots,q_{i,~ {}head2})\\ k_{i}=concat(k_{i,~{}head1},k_{i,~{}head2},\cdots,k_{i,~{}head2})\\ v_{i}=concat(v_{i,~{}head1},v_{i,~{}head2},\cdots,v_{i,~{}head2})\end{array} \tag{30}\]  
where \(q_{i}\), \(k_{i}\), \(v_{i}\)\(\in\)R\({}^{xxi}\) are the corresponding key, query, and value, which are used for subsequent attention calculations.  
Assume that there are k keywords in the given topic information \(T\), such that \(K\) = \(\{k_{1},k_{2},\ldots,k_{k}\}\). We assign weights to the importance of keywords, calculate the weighted average, and then obtain the topic vector as  
\[T=\sum\
olimits_{k\in K}w_{k}\cdot v_{k} \tag{31}\]  
where \(v_{k}\) is the word vector, and \(w_{k}\) is the weight of keyword \(k\).  
Topic information is introduced into the attention calculation process, such that  
\[TAttn=softmax\left(\frac{q_{i}(k_{i}+TW^{t})^{T}}{\sqrt{d_{head}}}\right) \tag{32}\]  
where \(W^{T}\) is a parameter matrix that linearly transforms the topic information \(T\), which is mapped onto the same dimension as \(k_{i}\) through \(W^{t}\).  
The global paragraph attention module and the local paragraph attention module allow the decoder to learn the global paragraph information and local intra-paragraph information output by the encoder, respectively. The attention calculation formulas are:  
\[GParaAttn=softmax\left(\frac{q_{i}(DGAtt\cdot W^{K})^{T}}{\sqrt{d_{head}}}\right) \tag{33}\]  
\[LParaAttn=softmax\left(\frac{q_{i}(DaAtt\cdot W^{K})^{T}}{\sqrt{d_{head}}}\right) \tag{34}\]  
Finally, they are connected in one dimension, and the final topic attention is obtained after linear transformation.  
\[TopicAttn=concat(TAttn,GParaAttn,LParaAttn)\cdot W \tag{35}\]  
Figure 6: Topic attention mechanism.  
where \(W\in R^{3d\times d}\) is a parameter matrix, which can synthesize all information from topics, global paragraphs, and inter-paragraph attention and incorporate them simultaneously into the summary generation process.