## 3 Topic-Oriented Multi-Document Summarization (TOMDS) Model
### Abstractive Stage
#### 3.3.3 Local Transformer Layer  
The local transformer layer is used to encode the contextual information of the token within each paragraph. In order to better preserve the primary and secondary relationships of discourses, we expand the coding part of the original transformer model [38] and propose a discourse-aware attention (DaAtt) mechanism. DaAtt integrates the processed primary and secondary relationships of discourses into multi-head attention to generate rich semantic features in a complementary manner, as shown in Figure 4.  
Figure 3: Example of discourse segmentation and RST tree conversion. The original paragraph is segmented into 9 EDUs in box (**a**) and then parsed into an RST discourse tree in box (**b**). Elab, Interp, Conc, Seq, and Cond are relation labels (Elab = elaboration, Interp = interpretation, Conc = concession, Seq = sequence, and Cond = condition). The converted primary–secondary relationship-based RST discourse tree is shown in box (**c**). Nucleus nodes including 1, 3, 4, 5, 6, and 8, and satellite nodes including 2, 4, 6, and 8, are denoted by solid lines and dashed lines.  
In Section 3.3.1, we define the input token as \(\mathbf{x}_{ij}^{0}\). For the attention head \(\mathit{head}_{ik}\in\mathit{Head}_{i}(k=1,2,\dots,z)\), such that  
\[\mathit{head}_{ik}=\mathit{Attention}\left(W^{q,head_{ik}}\cdot\mathbf{x}_{ij}^{0 },W^{k,head_{ik}}\cdot\mathbf{x}_{ij}^{0},W^{v,head_{ik}}\cdot\mathbf{x}_{ij}^{0}\right) \tag{15}\]  
\[\mathit{MHAtt}_{ijk}=\mathit{concat}(\mathit{head}_{i1},\mathit{head}_{i2}, \dots,\mathit{head}_{i2})W^{v,head_{ik}} \tag{16}\]  
where \(W^{q,head_{ik}}\), \(W^{k,head_{ik}}\), \(W^{v,head_{ik}}\in\mathrm{R}^{\mathrm{nxm}}\), \(W^{v,head_{ik}}\in\mathrm{R}^{\mathrm{zmxn}}\) are the weight matrices.  
Then, the discourse-aware attention mechanism merges the primary and secondary relationship P with multi-head attention, such that  
\[\mathit{DaAt}_{ijk}=\alpha\mathrm{P}_{ijk}\ \odot\mathit{MHAtt}_{ijk}+ \mathit{MHAtt}_{ijk} \tag{17}\]  
\[\mathit{Context}_{ij}=\sum_{k}D\ a\mathit{Att}_{ijk} \tag{18}\]  
where \(\alpha\) is a hyperparameter used to balance the primary and secondary relationship of the discourse and multi-head attention information; \(\odot\) is the Hadamard product, which represents the numerical multiplication of the corresponding positions in the matrix; \(\mathit{Context}_{ij}\in\ \mathrm{R}^{\mathrm{zm}}\) represents the context vector generated by the discourse-aware attention.  
The subsequent processing is the same as the encoder part of the original transformer model, expressed as:  
\[h=\mathit{LayerNorm}\left(\mathbf{x}_{ij}^{l-1}+\mathit{Context}_{ij}\left(\mathbf{x}_ {ij}^{l-1}\right)\right) \tag{19}\]  
\[\mathbf{x}_{ij}^{l}=\mathit{LayerNorm}(h+\mathit{FFN}(h)) \tag{20}\]  
where LayerNorm represents layer normalization, and \(\mathit{FFN}\) is a two-layer feed-forward neural network with ReLU as the activation function. For the \(l\)-th transformer encoder layer, the input is \(\mathbf{x}_{ij}^{l-1}\), and the output is \(\mathbf{x}_{ij}^{l}\).  
Figure 4: Discourse-aware attention (DaAtt) mechanism.