# FrameSum: Leveraging Framing Theory and Deep Learning for Enhanced News Text Summarization
## 4 Experiments
### News Text Summarization Generation Experiments
#### 4.3.3 Experimental Results and Analysis  
Our evaluation of the FrameSum model and baseline methods consists of both automatic metrics and expert assessments. We first present the results of the automatic evaluation using ROUGE metrics, followed by a detailed analysis of the manual evaluation conducted by our expert panel. This combination of quantitative and qualitative assessments provides a comprehensive view of the performance and quality of the generated summaries. 1. Automatic Evaluation Results  
Tables 4-6 present the ROUGE metrics of each model on the CNN/Daily Mail, BBCnews, and XSum datasets, respectively. The results demonstrate that our proposed frame-aware summarization model (FrameSum) outperforms all baseline methods across all metrics, with statistically significant improvements (\(p<0.01\)). Specifically, the three unsupervised methods, Lead, LexRank, and TextRank, perform relatively poorly. Lead simply extracts sentences from the beginning of the article, ignoring the semantic information of the entire text. Although LexRank and TextRank consider the semantic relationships between sentences, they fail to capture the overall thematic framework of the article. The pointer-generator alleviates the OOV problem through a copying mechanism and introduces coverage loss to avoid repetition, thereby achieving better results than the unsupervised methods. However, its performance is still significantly lower than FrameSum. T5, flat transformer, and GPT-2 represent the application of pre-trained language models in text summarization tasks. Compared to the pointer-generator, these models can better model the semantic information of long documents and generate more coherent and fluent summaries. However, due to the lack of explicit modeling of the thematic framework of news reports, the summaries generated by these models are often not concise enough and lack prominent themes. Llama3 is a pre-trained language model with a larger number of parameters, possessing stronger language understanding and generation capabilities. Experimental results show that the fine-tuned Llama3 surpasses other pre-trained models in automatic evaluation metrics but still lags behind FrameSum. This indicates that merely increasing the model size cannot completely solve the issue of topic deviation in news summarization. As a commercialized AI writing assistant, Qwen excels in the readability of summaries, often generating more accessible and easy-to-understand text. However, from the perspective of information coverage, the summaries generated by [PERSON] often omit some important details and lack alignment with the thematic framework of the original text. Compared to the aforementioned baseline models, FrameSum achieves the best performance on all metrics. On the CNN/Daily Mail dataset, FrameSum's ROUGE-1, ROUGE-2, and ROUGE-L scores are 0.64, 0.41, and 1.65 percentage points higher than the best baseline (Llama3), respectively. On the BBCnews dataset, the corresponding metric improvements are 0.41, 0.31, and 0.07 percentage points. On the XSum dataset, the corresponding metric improvements are 1.35, 4.78, and 2.44 percentage points. These results fully demonstrate that by introducing news framing knowledge and performing explicit modeling, FrameSum can better grasp the main ideas of articles and generate concise and thematically prominent summaries. Moreover, FrameSum makes targeted improvements based on pre-trained language models, inheriting their strong language expression capabilities while compensating for their deficiencies in thematic summarization. 2 Manual Evaluation Results  
As introduced in the Evaluation Metrics section, we invited a panel of three experts to manually evaluate the quality of summaries generated by different models. Figure 6 shows the average scores given by the experts for each model in three dimensions: readability, information coverage, and framework relevance.