# HGTMFS: A Hypergraph Transformer Framework for Multimodal Summarization
## 5 Experiments
### Human Evaluation  
In addition, a comprehensive human evaluation study was conducted to assess model outputs across four critical metrics on the Daily Mail and CNN datasets: Informedness (INFOR.)--measuring the extent to which the summary provides sufficient and necessary information from the input; expressiveness (EXP)--evaluating the fluency and quality of the presentation; consistency (CONSIS)--assessing the alignment of the summary with the reference; and overall evaluation (OVER)--providing a holistic assessment of the summary's quality. Fifty examples were randomly selected, and three annotators independently assigned scores on a scale from 1 (worst) to 5 (best) for each metric.  
The human evaluation results, presented in Table 4, reveal that HGTMFS consistently achieved the highest performance across all four categories. Specifically, HGTMFS outperformed the second-best model by an average of +0.215 in INFOR, +0.95 in CONSIS, +1.275 in EXP, and +0.98 in OVER. This superior performance can be attributed to HGTMFS's ability to effectively process three data modalities, thereby delivering more comprehensive and contextually relevant information, as evidenced by its high informedness score.  
Furthermore, the architecture of HGTMFS adeptly mitigates conflicts between modalities, leveraging the strengths of the pre-trained model to enhance summary quality. Notably, HGTMFS achieved an average score of 3.575 in INFOR and 3.975 in EXP, demonstrating its robustness in providing informative and well-presented summaries. However, its performance in CONSIS (average 2.72) and OVER (average 2.48) indicates slight room for improvement in ensuring alignment with the reference and overall summary quality.  
In contrast, the S2S model, despite scoring an average of 3.36 in INFOR, exhibited a significantly lower CONSIS score of 1.275, reflecting its tendency to generate summaries that were largely irrelevant to the reference. This discrepancy underscores the limitations of the S2S model in maintaining content relevance and consistency.  
\begin{table}
\begin{tabular}{l c c c c} \hline \hline  & \multicolumn{4}{c}{**Daily Mail Dataset**} \\ \hline Method & INFOR. & CONSIS. & EXP. & OVER. \\ \hline TextRank & 2.02 & 1.80 & 2.30 & 1.52 \\ PG & 2.90 & 1.18 & 1.20 & 1.12 \\ S2S & 3.40 & 1.30 & 2.72 & 1.34 \\ MSMO & 1.10 & 1.35 & 1.05 & 1.05 \\
**HGTMFS (ours)** & **3.60** & **2.74** & **4.00** & **2.50** \\ \hline \hline \multicolumn{4}{c}{**CNN Dataset**} \\ \hline Method & INFOR. & CONSIS. & EXP. & OVER. \\ \hline TextRank & 1.98 & 1.74 & 2.25 & 1.48 \\ PG & 2.84 & 1.15 & 1.15 & 1.10 \\ S2S & 3.32 & 1.25 & 2.68 & 1.28 \\ MSMO & 1.08 & 1.32 & 1.02 & 1.02 \\
**HGTMFS (ours)** & **3.55** & **2.70** & **3.95** & **2.46** \\ \hline \hline \end{tabular}
\end{table}
Table 4: Human evaluation scores on four measures: Informedness (INFOR.), Consistency (CONSIS.), Expressiveness (EXP.), and Overall Evaluation (OVER.) for Daily Mail and CNN Datasets.  
These human evaluations, particularly in the areas of informedness and expressiveness, unequivocally highlight the effectiveness of HGTMFS in integrating and processing multimodal information, thereby substantiating its superiority over existing models.