## 3 Topic-Oriented Multi-Document Summarization (TOMDS) Model
### Abstractive Stage
#### 3.3.4 Global Transformer Layer  
The global transformer layer is used to exchange information between paragraphs. Each paragraph of the input can collect information from other paragraphs and then capture global information from the entire input to encode paragraph-level vector representation. * Multi-head Pooling  
In order to obtain paragraph-level vector representation with a fixed length, we use the multi-head pooling mechanism proposed by [PERSON] et al. [22]. Unlike the common max pooling and average pooling, multi-head pooling uses weighted operations to reduce dimensionality. By calculating the weight distribution of tokens for each input paragraph, the model can flexibly encode paragraphs in different representation subspaces by focusing on different words. Similar to the transformer's multi-head attention mechanism, each input paragraph \(R_{i}^{s}\) has an attention head \(head_{i}^{t}\in Head_{i}(t=1,2,\dots,h)\). Using different dimensional projections for each pooling head separately will translate \(R_{i}^{s}\) into attention keys \(K_{i}^{t}\) and values \(V_{i}^{t}\), such that:  
\[K_{i}^{t}=R_{i}^{s}W_{K}^{t},V_{i}^{t}=R_{i}^{s}W_{V}^{t} \tag{21}\]  
where \(W_{K}^{t}\in\ R^{d\times d}\) and \(W_{V}^{t}\in R^{d\times d_{head}}\) are the weight matrices. \(d_{head}=d/n_{head}\) is the size of each head, and n is the number of tokens in \(R_{i}^{s}\). After the softmax transformation of \(K_{i}^{t}\), it is used as the weight of each token in the paragraph, and then the matrix is weighted and added along the dimension of the word, and then \(h\) pooling heads are connected to obtain a paragraph-level vector representation. \[MultiHeadPooling(K_{i},V_{i})=Concat\Big{(}head_{i}^{1},head_{i}^{2},\dots, head_{i}^{h}\Big{)}W_{iO} \tag{22}\]  
\[head_{i}^{t}=\sum_{j=1}^{m}softmax\big{(}K_{i}^{t}\big{)}V_{i}^{t} \tag{23}\]
* Discourse Graph Attention  
Although the paragraph-level representation vector obtained through the multi-head pooling mechanism considers the words' information within the paragraph, it does not consider the associated information between paragraphs. Here, we integrate the relationship matrix M between paragraphs in Section 3.3.2 into the global transformer layer to better learn the relationship between paragraphs in multiple documents. The structure of the discourse graph attention mechanism is shown in Figure 5.  
Figure 5: Discourse graph attention (DGAtt) mechanism. The input of the discourse graph attention mechanism is a multi-document representation matrix \(D=\{R_{1}^{s},\ R_{2}^{s},\ldots,R_{L^{\prime}}^{s}\}\) composed of paragraph-level vectors, where \(R_{i}^{s}\in R^{d_{head}}\). For each attention head, the paragraph-level vector \(R_{i}^{s}\) is first linearly projected into a vector with dimension size \(d_{head}\), and the subsequent steps are similar to the scaling dot-product attention in the multi-head attention mechanism. The implicit relationship matrix between paragraphs is:  
\[E_{i}=\frac{DW_{i}^{Q}.(DW_{i}^{K})^{T}}{\sqrt{d_{head}}} \tag{24}\]  
\[Weight_{i}=softmax(E_{i}+G_{i}) \tag{25}\]  
\[Head_{i}=Weight_{i}.DW_{i}^{V} \tag{26}\]  
\[DGAtt=Concat(Head_{1},Head_{2},\ldots,Head_{h})W^{O} \tag{27}\]  
\(i\in\) (1,h) represents each attention head of the discourse map attention mechanism and the linear transformation matrix \(W_{i}^{Q},W_{i}^{K},W_{i}^{V}\in R^{d\times d_{head}}\). The linear transformation matrix \(Weight_{i}\) represents the weights assigned to each paragraph in the attention mechanism of the discourse graph, indicating the attention of each paragraph to other paragraphs from different global perspectives. \(G_{i}\) is the Gaussian deviation of the relationship between paragraphs calculated according to the discourse graph, expressed as:  
\[G=-\frac{(1-M)^{2}}{2\sigma^{2}} \tag{28}\]  
In Equation (28), \(\sigma\) is the hyperparameter of the influence intensity of the discourse graph structure in the attention mechanism, and \(M\) is the relationship matrix between paragraphs.