# HGTMFS: A Hypergraph Transformer Framework for Multimodal Summarization
## 4 Hypergraph-Based Multimodal Summarization
### Multimodal Hypergraph Group Transformer
#### 4.4.1 Intra-Hyperedge Aggregation  
The intra-hyperedge information aggregation module is designed to derive representations for both claim and evidence hyperedges by effectively capturing semantic correlations across different levels of granularity. Given a hyperedge \(e_{i}\) and its associated node set \(\mathcal{V}(e_{i})=\{v_{1},v_{2},\ldots,v_{m}\}\), each node \(v\in\mathcal{V}(e_{i})\) corresponds to a textual token or an image patch, which is processed through a pretrained multimodal model to obtain representations. All nodes in \(\mathcal{V}(e_{i})\) form an embedding set \(\mathcal{S}=\{\mathbf{s}_{i}\in\mathbb{R}^{d}\}_{i=1}^{m}\), where \(\mathcal{S}\in\mathbb{R}^{m\times d}\). Inspired by the transformer [42], the multimodal model is jointly trained with the hyperedge transformation (HT) module. By introducing a learnable reference set \(\mathcal{Z}=\{\mathbf{z}_{i}\in\mathbb{R}^{d}\}_{i=1}^{p}\), elements from \(\mathcal{V}(e_{i})\) are aggregated into fixed-size representations. Each \(\mathbf{z}_{i}\) acts as a pooling unit gathering semantic information from \(\mathcal{V}(e_{i})\). The calculation of self-attention among intra-hyperedge nodes can be formulated as:  
\[ \begin{split}\mathbf{Q}^{v}=\mathbf{W}_{S}^{q}\mathbf{S},\quad \mathbf{K}^{v}=\mathbf{W}_{S}^{k}\mathbf{S},\quad\mathbf{V}^{v}=\mathbf{W}_{S}^ {v}\mathbf{S},\\ \mathbf{Z}=&\text{self\_att}(\mathbf{Q}^{v},\mathbf{ K}^{v},\mathbf{V}^{v})=\text{softmax}\Bigg{(}\frac{\mathbf{Q}^{v}\cdot(\mathbf{K}^{v}) ^{\top}}{\sqrt{d}}\Bigg{)}\mathbf{V}^{v},\end{split}  \tag{1}\]  
where \(\mathbf{W}_{S}^{q},\mathbf{W}_{S}^{k},\mathbf{W}_{S}^{v}\in\mathbb{R}^{p\times m}\) are trainable parameter matrices. Subsequently, the hyperedge \(e_{i}\) is represented as  
\[\mathbf{e}_{i}=\mathbf{W}_{e}[\mathbf{z}_{1}:\mathbf{z}_{2}:\ldots:\mathbf{z} _{p}]^{\top}, \tag{2}\]  
where [:] denotes concatenation, and \(\mathbf{W}_{e}\in\mathbb{R}^{p}\) is the trainable parameter matrix. The basic transformer attention mechanism has certain limitations, such as only considering relationships between individual tokens or patches at a single granularity level. To enhance attention across varying granularities within claims and evidence, we propose an improvement to the conventional self-attention mechanism. This enhancement introduces grouped attention to capture interactions among neighborhoods of tokens or patches. Our approach involves associating neighborhoods of tokens or patches to implement grouped attention. Specifically, we replace certain entries in the query, key, and value matrices with aggregated representations of entire groups, generating proxy representations for each group. Aggregation methods such as max pooling or convolution are used. Typically, we partition Q, K, and V into \(n\) segments, with each segment's feature denoted as \(X_{i}\), where \(i\in\{1,2,\ldots,n\}\) (with \(X\) representing Q, K, or V). The aggregation operation for each segment is represented as \(Agg_{i}(X_{i})\). The aggregated results of all segments are concatenated to form \(X^{\prime}\), yielding the group proxies \(Q^{\prime}\), \(K^{\prime}\), and \(V^{\prime}\). These group proxies are then used for attention computation to generate the final output \(Z^{\prime}\), formulated as:  
\[\mathbf{Z}^{\prime}=\text{self\_att}(\mathbf{Q}^{\prime},\mathbf{K}^{\prime}, \mathbf{V}^{\prime})=\text{softmax}\Bigg{(}\frac{\mathbf{Q}^{\prime}\cdot( \mathbf{K}^{\prime})^{\top}}{\sqrt{d}}\Bigg{)}\mathbf{V}^{\prime}. \tag{3}\]  
Given that the input to attention calculation now consists of group proxies, we model correlations among \(K\times K\) tokens, where \(K\) is the kernel size of the aggregator and may vary across segments. This strategy extends beyond individual tokens or patches, enabling more comprehensive correlation modeling. Consequently, this method captures relationships between groups of varying sizes as well as single tokens, thereby enhancing overall model performance. The complexity of the group attention is \(O(|\mathcal{V}|^{2})\).