# FACTMS: Enhancing Multimodal Factual Consistency in Multimodal Summarization
## 2 Related Work  
Our research is mainly based on previous studies on multimodal summarization and factual consistency of text summarization.  
**Text Summarization**  
Automatic text summarization involves generating a succinct and coherent summary that retains the essential information and overarching meaning of the original text [9]. Numerous methods of text summarization have been proposed [10; 11; 12; 13; 14; 15]. [PERSON] et al. [13] presented a denoising autoencoder called BART designed for pretraining sequence-to-sequence models. This approach proves particularly effective when subsequently fine-tuned for text generation tasks. [PERSON] et al. [15] introduced a pioneering domain feature miner, crafted by conceptualizing the feature mining problem as a clustering issue. Three recently formulated empirical observations are incorporated to enhance the effectiveness of summarizing textual data. [PERSON] et al. [11] proposed sentence topic embeddings and sentence content embeddings for capturing long-range semantic dependencies and structural content information in texts. [PERSON] et al. [10] introduced the DEPSYM++ simplifier, designed to perform four distinct types of simplification on sentences within the input text. These simplifications are tailored to address the presence of appositive clauses, relative clauses, conjoined clauses, and passive voice. Those methods strive in every possible way to maximally utilize textual data to generate summaries.  
**Multimodal Summarization**  
Multimodal summarization aims to generate the crucial part of the multimodal input. In recent years, the generation of text summarization from multimodal data, including text, vision, and audio has been extensively studied [16; 17; 18]. A substantial body of research has concentrated on integrating visual data to enhance the quality of text summarization [1; 2; 3; 19; 20; 21; 22]. [PERSON] and [PERSON] [1] proposed a multimodal attention mechanism to generate text summaries by combining visual and text information. [PERSON] et al. [2] introduced a multimodal sentence summarization task that creates a condensed text summary from a long sentence and its corresponding image. [PERSON] et al. [3] used a multimodal selective gate network to encourage the encoder to focus on the important information of the input text. These studies are mainly devoted to improving the word overlap between the generated summary and the reference summary (i.e., ROUGE), which ignores factual inconsistency ([PERSON] et al. [19]).  
**Factual Consistency of Text Summarization**  
Nearly 30% of the summaries generated by abstractive text summary models have faithfulness problems such as distortion and fabrication [4]. With the development of abstractive text summarization, the problem of factual consistency has received increasing attention in recent years. [PERSON] et al. [4] proposed a dual attention decoder, which integrates source sentences and factual descriptions to improve the factual consistency of text summarization. [PERSON] et al. [6] treated subjects and objects as nodes to construct a graph transformer [23], introducing factual consistency information to enhance the model's semantic understanding of entity interaction with a multi-choice cloze reward. [PERSON] et al. [7] used a graph attention network [24] to learn a feature representation of the entity relation in the source article and employ a factual accuracy model designed to refine the generated summary. By regarding the factual inconsistency problem as natural language inference and fact checking, [PERSON] et al. [5] proposed the weakly supervised factual consistency metric FactCC, which is widely used to study factual consistency of text summarization. The current studies on factual consistency mainly focus on text-only summarization, which is not directly adaptable to the multimodal consistency in MS.