FKA-Owl: Advancing Multimodal Fake News Detection through Knowledge-Augmented LVLMs

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-06 07:01:19

Abstract: The massive generation of multimodal fake news involving both text and images exhibits substantial distribution discrepancies, prompting the need for generalized detectors. However, the insulated nature of training restricts the capability of classical detectors to obtain open-world facts. While Large Vision-Language Models (LVLMs) have encoded rich world knowledge, they are not inherently tailored for combating fake news and struggle to comprehend local forgery details. In this paper, we propose FKA-Owl, a novel framework that leverages forgery-specific knowledge to augment LVLMs, enabling them to reason about manipulations effectively. The augmented forgery-specific knowledge includes semantic correlation between text and images, and artifact trace in image manipulation. To inject these two kinds of knowledge into the LVLM, we design two specialized modules to establish their representations, respectively. The encoded knowledge embeddings are then incorporated into LVLMs. Extensive experiments on the public benchmark demonstrate that FKA-Owl achieves superior cross-domain performance compared to previous methods. Code will be made publicly available.

Main paper: 
 introduction: 
 The wide spread of fake news has become an important social issue, posing threats to politics [11] , finance [10] , and public health [36] . Misusing advanced generative models to create misinformation further exacerbates these issues, manifested in the rise of both text fake news [50] and also visual deepfakes [60] . Furthermore, multimodal forgery media through convergence disseminates more expansive information with greater impact to mislead readers. Detecting such fake news poses a unique challenge due to the existence of manipulations in both image and text modalities. Faced with this challenge, current multimodal fake news detection (MFND) methods [42, 47, 57] primarily focus on incorporating cross-modal features. While some progress has been made, the acquisition of broad world information remains challenging due to the confinement of training to given domains (i.e., closed systems). However, open-world fake news exhibits substantial distribution discrepancies [63] , termed domain shift [38, 64] , which is manifested in abundant forgery methods and diverse real-world context. The existence of domain shift increases the difficulty of understanding and characterizing open-world fake news in MFND tasks.To address this problem, we propose to leverage Large Vision-Language Models (LVLMs) [31, 62] which possess rich world knowledge for fake news detectors. This knowledge encompasses a wide array of world facts [4] about public celebrities, social events etc, which enables a comprehensive understanding of agnostic fake news. However, despite their proficiency in recognizing common instances, the performance of applying off-the-shelf LVLMs to the MFND task is not always satisfying. On the one hand, since LVLMs are not inherently tailored for MFND tasks, they are still challenging to understand and discover the subtle cross-modal differences [41] . For example, in detecting manipulated image-text pair in Fig. 1-(a) , the model must discern the sentiment tendencies of both image and text, which are typically not present in LVLMs' training data. On the other hand, LVLMs lack sensitivity to localized spatial details [58] . As shown in Fig. 1-(a) , when swapping Trump's face with another person, there exists intrinsic image discrepancies between edited regions and pristine backgrounds, which is hard to perceive with existing LVLMs. Therefore, it is important to augment LVLMs with external information, referred to as forgery-specific knowledge, which is absent from model parameters yet useful for manipulation reasoning.We identified two types of forgery-specific knowledge essential for manipulation reasoning [61] : semantic correlation and artifact trace. Firstly, manipulated media often disrupt the natural coherence between different modalities, resulting in semantic discrepancies [47] . Secondly, alterations in images usually produce distinctive artifact traces, such as irregular blending boundaries and inconsistencies in color sources, among others [51] . As a result, it would be beneficial to incorporate these two types of knowledge into the training and inference of large vision-language models for multimodal fake news detection.In this paper, we present a novel framework, namely FKA-Owl which augments LVLMs with forgery-specific knowledge to enhance cross-domain performance for multimodal fake news detection. This framework leverages the rich world knowledge of LVLMs, supplementing it with domain-specific knowledge crucial for identifying multimodal fake news. As shown in Fig. 1-(c ), to achieve that, we establish representations of the aforementioned two kinds of forgery-specific knowledge with two specialized modules. The cross-modal reasoning module applies dual cross-attention mechanisms to integrate visual and textual information from frozen encoders, aiming to identify semantic inconsistencies. Meanwhile, the visual-artifact localization module focuses on detecting precise visual artifacts at multiple levels of detail, using sparse bounding boxes and detailed mask regions to trace artifacts. Subsequently, the encoded knowledge representation embeddings are mapped to the language space of LVLMs with projectors. We devise MFND instruction-following data for fine-tuning and employ both candidate answer heuristics and soft prompts to unleash the extensive knowledge of language models.Our contributions are summarized as follows:• We pioneer leveraging rich world knowledge from large vision-language models (LVLMs) and enhancing them with forgery-specific knowledge, to tackle the domain shift issue in multimodal fake news detection. Our proposed method, FKA-Owl, serves as a general detector to bridge the gap. • FKA-Owl augments LVLMs with forgery-specific knowledge for manipulation reasoning. We propose two lightweight modules: the cross-modal reasoning module and the visual-artifact localization module to extract semantic correlations and artifact traces, respectively. • The extensive experiments demonstrate the effectiveness of our proposed method under multiple cross-domain settings. 
related work 2.1 fake news detection: 
 Fake news detection works can be categorized into unimodal (image-based and text-based) and multimodal methods. Imagebased method [5] proposes to exploit edited traces to verify the truth of visual content. One group of CNN-based methods focus on the spatial domain to capture artifact traces, such as blending [25, 51] , multiple instance learning [26] patch consistency [59] , reconstruction [18, 27] , and local mining [9] . The other works transformed images into the frequency domain by applying DCT [43] , combining phase spectrum [32] , and extracting high-frequency noises [33] .Text-based methods primarily delve into various aspects. Ghanem et al. [13] proposes to incorporate topic and affective information extracted from text. Some social context-based methods leverage user feedback [35] , news environment [49] , propaganda techniques [17] and temporal patterns [16] . Recently, Nan et al. [38] and Zhu et al. [64] all discover the domain shift issue caused by the word frequency and emotion etc, and propose domain gate and domain memory bank to enrich domain information, respectively.In contrast to unimodal methods, multimodal methods adhere to incorporate cross-modal features to extract semantic representations [19] . Sabir et al. [46] and Wang et al. [55] both propose to combine with the external knowledge base to provide complementary semantics information. Qi et al. [40] proposes to extract visual entities to understand the high-level semantics of news. Coattention network [56] and contextual attention network [42] are both designed to better fuse textual and visual features. Moreover, Ying et al. [57] proposes improved Multi-gate Mixture-of-Expert networks (iMMoE) to refine and fuse features extracted from multiple views. Ambiguity learning [6] and causal reasoning [7] are separately introduced to address the issue of modal disagreement decisions and spurious correlation in data bias. A recent work [47] presents HAMMER, a powerful model that combines contrastive learning and cross-modal aggregation. However, all these methods are typically trained and deployed within closed systems, overlooking the potential benefits of accessing world knowledge.Large language models (LLMs) such as GPT-3 [3] , LLaMA [53] and Vicuna [8] , have showcased remarkable performance on various linguistic tasks. More recently, researchers are exploring extending the capability of LLMs to perceive visual signals. LLaVA [31] and Mini-GPT4 [62] first facilitate image-text feature alignment followed by visual instruction tuning. Visual instruction tuning entails additional training of pre-trained models using curated instructionformatted datasets to enhance models' generalization to unseen tasks. PandaGPT [52] employs a simple linear layer as a bridge between ImageBind [14] and the Vicuna model, allowing for the multimodal input. The success of LVLMs in the general domain has led to the growth of communities such as medical [24] , video understanding [21] and image editing [12] . In this work, we leverage the world knowledge inherent in LVLMs for a better understanding of open-world fake news.Utilizing external knowledge to augment language models (LMs) has emerged as a promising solution in knowledge-intensive tasks [22] . One line of works is retrieval-augmented LMs which retrieve relevant passages and incorporate them into LMs. Borgeaud et al. [2] proposes a chunked cross-attention module to incorporate the retrieved text as explicit memory. A lightweight retrievalaugmented dual fine-tuning [29] is introduced to retrofit any LLM. Asai et al. [1] introduces self-reflection on retrieved passages to  
proposed method: 
 In this section, we present a unified framework named FKA-Owl which incorporates forgery-specific knowledge into LVLMs for manipulation reasoning. We first introduce the overall framework architecture. Then, we elaborate on our proposed multi-level crossmodal reasoning module, dual-branch visual-artifact localization module, and forgery-aware vision-language model. Finally, we detail the loss function for training.The comprehensive architecture of FKA-Owl is illustrated in Fig. 2 . FKA-Owl consists of an image encoder (ImageBind [14] ), a crossmodal reasoning module, a visual-artifact localization module, and a large language model (Vicuna [8] ). Given a manipulated image-text pair (𝐼,𝑇 ), we design two lightweight modules to extract semantic correlations and artifact traces as forgery-specific knowledge representations, respectively. Specifically, the cross-modal reasoning module utilizes dual-branch cross-attention mechanisms to guide cross-modal interactions, facilitating the encoding of semantic embeddings. Concurrently, the visual-artifact localization module gathers local spatial information to establish artifact embeddings through supervised localization. Then, the forgery-aware visionlanguage model leverages both the forgery-specific knowledge and inherent world knowledge for deep manipulation reasoning.The input image and text are encoded first by the frozen pre-trained encoders, denoted as 𝐸 𝑣 for the image and 𝐸 𝑡 for the text. Both 𝐸 𝑣 and 𝐸 𝑡 originate from the ImageBind, which is aligned with the Vicuna in the off-the-shelf LVLM [52] . To obtain both low-level elements and high-level semantic cues, we partition the image encoder into multiple layers with layer index 𝑙, enabling the integration of intermediate patch-level features. As shown in Fig. 2-(a) , the image features 𝑓 𝑣 and text features 𝑓 𝑡 are separately represented as:EQUATIONBoth features contain information of object instances within a single modality but lack prior insight into objects referenced by the other modality. This absence of complementary information between modalities may hinder cross-modal semantic reasoning. To this end, we devise the dual-branch cross-attention to guide the interaction between visual and textual features, enabling the extraction of semantic correlations. Attention function [54] is performed on normalized query (𝑄), key (𝐾), and value (𝑉 ) features as:EQUATIONIn dual-branch cross-attention, each modal feature (e.g., image) is allowed to serve as queries 𝑄, while keys 𝐾 and values 𝑉 can be taken from the other modality (e.g., text): Based on the cross-modal interaction described above, the [CLS] tokens 𝑢 cls 𝑣 and 𝑢 cls 𝑡 can naturally focus on the inter-modal semantic correlations. We concatenate these two [CLS] tokens {𝑢 cls 𝑣 , 𝑢 cls 𝑡 } as a joint representation of semantic embedding. Then we use a learnable linear layer to project this generated knowledge into the word embedding space of LVLMs, facilitating profound semantic reasoning aided by world factual knowledge.EQUATIONEQUATIONIn addition to extracting semantic correlations between visual and textual features, mining artifact traces is also crucial for distinguishing fake news. Unlike [CLS] token, visual patch tokens with position encoding [54] contain richer local spatial information. Given visual patch tokens 𝑢 pat 𝑣 , we propose a visual-artifact localization module to encode them into artifact embeddings, guided by grounding annotations.As depicted in Fig. 2-(b) , the upper branch which comprises a text encoder and a pixel decoder, is designed to utilize language-driven contrastive learning for pixel-wise localization. Dense features with good language alignment can provide complementary benefits for fine-grained segmentation [15, 20] . Specifically, two class prompts are initially encoded by the text encoder to obtain corresponding features 𝐹 𝑖 𝑝 (𝑖 = 1, 2) ∈ R 2×𝐶 𝑡𝑒𝑥𝑡 , representing "natural" and "unnatural" states. Detailed contrastive class prompts are presented in the Supplementary Materials. To restore local spatial details, the pixel decoder with consecutive deconvolution layers converts lowresolution features 𝑢 pat 𝑣 ∈ R ℎ𝑤×𝐶 𝑖𝑚𝑔 into high-resolution features 𝐹 ℎ ∈ R 𝐻 ×𝑊 ×𝐶 𝑖𝑚𝑔 . Since patch-level features are not aligned with the textual space, we project both textual features 𝐹 𝑝 and visual features 𝐹 ℎ into a shared representation space. Subsequently, the projected features F𝑝 ∈ R 2×𝐶 and Fℎ ∈ R 𝐻 ×𝑊 ×𝐶 are employed to compute similarity scores 𝑤 ∈ R 𝐻 ×𝑊 ×2 :EQUATIONBy applying the scores 𝑤 spatially, we can establish the manipulated segmentation map 𝑀 𝑠 ∈ R 𝐻 ×𝑊 ×2 to achieve per-pixel prediction:EQUATIONTo enrich the representation of artifact traces with multiple levels of details, the lower branch employs multi-head attention for patchlevel localization. We utilize a learnable token 𝑞 tok ∈ R 1×𝐶 𝑖𝑚𝑔 serving as a query, while visual features 𝑢 pat 𝑣 act as both the key and value. Through the multi-head attention mechanism, local information related to artifacts is aggregated within 𝑢 agg ∈ R 1×𝐶 𝑖𝑚𝑔 under the supervision of bounding box grounding:EQUATIONTo leverage the artifact knowledge contained in the two-branch features (i.e., the manipulated segmentation map 𝑀 𝑠 and the aggregated token 𝑢 agg ), we separately devise multiple convolution layers and a simple linear layer as projectors. In this manner, both 𝑀 𝑠 and 𝑢 agg are converted into continuous artifact embeddings aligned with the final vision-language model.With the extraction of two types of forgery-specific knowledge, merging this external knowledge into LVLMs becomes imperative. Both the obtained semantic embeddings and artifact embeddings have been refined to lie in the embedding space for compatibility with language models. Moreover, the introduction of MFND instruction data along with answer heuristics and soft prompts can further activate the capacity of LVLMs.Due to the lack of instruction-follow data in the MFND task, we carefully design an instruction template following the conversational format of the Vicuna model [8] , as shown below:###Human: <Img><ImageFeature></Img><ForgeryFeature>[Human Prompt] ###Assistant:In this prompt, <ImageFeature> represents the visual tokens produced by the image encoder and <ForgeryFeature> is the combination of semantic and artifact embeddings. The human prompt adopts the multiple choice question answering format, as shown in the dialog box of Fig. 2-(c ).To unleash the potential knowledge of LVLMs in solving MFND tasks, we devise two prompt strategies to serve as more informative inputs in Fig. 2-(c ). On the one hand, we utilize candidate answer heuristics [45, 48] to present both the question and answer options to LVLMs and make them predict the symbol (e.g., "A") associated with the selected answer. This approach enables the language models to explicitly compare different candidate answers showcasing more accurate responses. On the other hand, we implement soft prompt tuning to introduce learnable continuous vectors while freezing the language models. These vectors combined with semantic embeddings facilitate the extraction of additional semantic information. Meanwhile, this approach reduces the burden of LVLMs to learn forgery-aware alignment, thereby mitigating the catastrophic forgetting problem.Two groups of loss functions are employed in our training procedure: cross-entropy loss, and two-branch localization losses.In the training of language models, crossentropy loss is employed to measure the disparity between the text sequence predicted by the models and the target text sequence. The formula is computed according to:EQUATIONwhere 𝑛 denotes the token count, 𝑦 𝑖 is the true label for token 𝑖 and 𝑝 𝑖 is the corresponding predicted probability.Loss. Two-branch localization losses are designed for the precise encoding of artifact traces guided by pixel-wise and patch-level localization, respectively. Pixelwise localization introduces focal loss [28] and dice loss [34] to enable supervision for the manipulated segmentation map 𝑀 𝑠 :L pixel = L focal + L dice .Patch-level localization involves regressing the final bounding box with the aggregated token 𝑢 agg and computing the regression losses with the ground-truth box by introducing L1 loss and GIoU loss [44] : L patch = L L 1 +L giou . More details about localization losses are provided in the Supplementary Materials. Finally, the overall loss function is defined as:EQUATION 
experiments: 
 In this section, we first introduce the overall experimental setup and then provide comprehensive experimental results to demonstrate the superiority of our proposed method. [47] and Fakeddit dataset 2 [37] . DGM 4 . DGM 4 dataset is the recently released large-scale multimodal manipulation dataset which comprises 230K image-text paired samples with over 77K pristine pairs and 152K manipulated pairs. In the DGM 4 dataset, image manipulation involves face swapping and facial emotion editing while text manipulation includes sentence replacement and textual sentiment editing. The construction of the DGM 4 dataset is based on the VisualNews dataset [30] , which is collected from multiple news agencies. Different agencies cover diverse regional perspectives, thematic focus, and language styles (see Supplementary Materials for the analysis of word clouds), resulting in substantial distribution discrepancies. To simulate the open-world domain-shift scenarios, we partition the DGM 4 dataset into four subsets based on news sources: BBC, The Guardian, USA TODAY (USA), and The Washington Post (Wash.). The statistics of four subsets are listed in Table 1 .Fakeddit. Fakeddit dataset is curated from multiple subreddits of the Reddit platform where data varies in its content, ranging from political news stories to simple everyday posts. We follow the official dataset partition to only use multimodal samples and use the 2-way categorization for this dataset. Furthermore, we preprocess the data by removing excessively short tweets based on their text length. Short texts often lack sufficient information for semantic 1 https://github.com/rshaojimmy/MultiModal-DeepFake 2 https://github.com/entitize/Fakeddit inconsistency detection. For our task, we only use the test set data for the cross-dataset evaluation.We treat the multimodal fake news detection problem as a binary classification task. Following previous works [39, 47] , we apply the Area Under the Receiver Operating Characteristic Curve (AUC), Equal Error Rate (EER), the Accuracy Score (ACC) as our evaluation metrics.4.1.3 Baselines. The proposed FKA-Owl is compared with the following strong baseline models: 1) PandaGPT [52] : The off-theshelf PandaGPT model effectively aligns visual features with the text space of LLMs, enabling it to perform complex multimodal tasks in a zero-shot manner. 2) PandaGPT+SPT: This model integrates PandaGPT with soft prompt tuning [23] by using the predefined question prompt along with learnable continuous vectors to fine-tune the LVLM during the instruction tuning phase. 3) HAM-MER [47] : HAMMER employs two unimodal encoders to encode image and text embeddings with contrastive learning for alignment, and then summarize multimodal information through the multimodal aggregator.We utilize the visual encoder and text encoder sourced from ImageBind-Huge [14] as the backbone for our image and text feature extractors. Moreover, we employ Vicuna-7B [8] as the inferential LLM, connected through a linear layer. The model is initialized from the instruction-tuned checkpoint provided by PandaGPT. All training images are resized to 224 × 224 and subjected to random horizontal flipping, as well as random perturbation techniques such as JPEG compression and Gaussian Blur following [47] . The multi-level cross-modal reasoning module extracts intermediate patch features from the 8th, 16th, 24th, and 32nd layers of the image encoder. We set the learning rate as 1.5e-5 with a batch size of 16 and a maximum of 10 epochs when trained on the BBC subset. Linear warm-up and the one-cycle cosine learning schedule are adopted. All experiments are conducted on four NVIDIA GeForce 3090 GPUs with PyTorch. More details about hyperparameter settings are provided in the Supplementary Materials.We evaluate the cross-domain performance of our FKA-Owl with baselines in single-domain, multiple-domain, and cross-dataset settings respectively. 2 presents the performance of our method and other baseline models in the challenging scenario where a single domain is available. We randomly select one subset of the DGM 4 dataset as the source domain for training and the remaining subsets as the target domains for testing. From the results, we make the following observations:• The performance of the existing MFND method drops significantly when tested on the unknown subsets, which verifies the existence of domain shift caused by the deviation in propagation contents. • FKA-Owl yields substantial improvement on recent LVLMs, PandaGPT, and PandaGPT with soft prompt tuning, in both intra-domain and cross-domain testing. Such huge improvement demonstrates the effectiveness of forgery-specific knowledge augmentation in our framework. • Compared with the state-of-the-art method, HAMMER, our approach shows more remarkable improvement in cross-domain testing. For instance, for models trained on the BBC subset, FKA-Owl achieves a 7.7% increase in AUC when testing on the Washington Post subset. This may be credited to the effective utilization of inherent world knowledge from LVLMs in mitigating distribution discrepancies. The combination of forgery-specific knowledge and world knowledge facilitates profound manipulation reasoning in FKA-Owl.Multiple-domain Setting. The inclusion of domestic news such as the BBC and The Guardian from British, as well as USA TODAY and The Washington Post from America, increases dataset diversity in practical scenarios. We select two subsets from identical countries for training and the remaining two for testing. The results are summarized in Table 3 . Our FKA-Owl exhibits significant superiority over both PandaGPT using soft prompt tuning and HAMMER by a large margin. This reveals the effectiveness of our framework in instance-wise domain generalization guided by world knowledge derived from LVLMs, even when jointly learning multiple source domains. To comprehensively represent the contextual diversity inherent in multimodal fake news detection tasks, We select one subset from DGM 4 for training and evaluate the performance on the Fakeddit dataset. As shown in Table 4 , the performance of all methods notably decreases when tested on the Fakeddit dataset, which implies that the difference in the distribution of different datasets does exist. Furthermore, we can observe that our method outperforms the state-of-the-art model HAMMER when trained on different subsets, confirming the generalizability of our approach.We perform several ablation experiments to explore the necessity of the proposed component modules, prompt strategies, and LVLMs knowledge respectively, and analysis of potential module choices.In the following experiments, all ablation results are evaluated on the remaining three subsets when trained on the BBC subset. The Effect of the Prompt Strategies. The prompt strategies designed in the alignment process and the corresponding results for each case are tabulated in Table 6 . First, when removing continuous prompt vectors (w/o SPT), the performance drops a little bit. In addition, after removing the candidate answer options (w/o CAH) in the human instruction prompt, the average performance decreases from 77.62% to 73.69%. In particular, the third row of Table 6 represents that none of both strategies is employed in our proposed framework. Our method substantially outperforms this variant model, implying both two strategies enable the introduction of implicit information to fully activate the capacity of LVLMs.The Effect of the LVLMs Knowledge. To analyze the impact of world knowledge derived from LVLMs, we compare our method with the common practice of processing fused embeddings in a supervised classification manner [47] . This variant model (w/o World Knowledge) replaces the Vicuna model in FKA-Owl with a binary classifier to predict true/fake labels. As shown in Fig. 3 , harnessing the inherent knowledge in LVLMs improves an average performance by 9.92 points over the variant model. Furthermore, for target domains from distinct countries exhibiting huge distribution differences, FKA-Owl yields more significant improvements in 13.26% and 12.15%. This could be attributed to the fact that world knowledge from LVLMs can effectively guide the representation of agnostic instances.Choice. We replace the artifact extractor module in our framework for other design choices. This variant model (w/ Pre-trained Artifact Detector) replaces the visual-artifact localization module in FKA-Owl with the off-theshelf pre-trained detector [47] to obtain the artifact embeddings. The pre-trained detector employs the VIT backbone with supervised training by grounding annotations. As depicted in Fig. 3 , our visual-artifact localization module brings a significant improvement over the variant model. This could be attributed to the fact that Our designed module leverages the intrinsic visual encoder in LVLMs to extract artifact traces, thereby alleviating the burden of aligning forgery-specific knowledge with LVLMs. 
conclusion: 
 Our work presents FKA-Owl, a novel framework that leverages rich world knowledge from LVLMs and enhances them with forgeryspecific knowledge, to tackle the domain shift issue in multimodal fake news detection. Two types of critical forgery-specific knowledge are augmented in FKA-Owl: semantic correlation between text and images and artifact trace in image manipulation. To inject this knowledge into the LVLM, we first propose two lightweight specialized modules to learn their representations respectively. Then, we transform the generated knowledge into refined embeddings for alignment with language space. The candidate answer heuristics and soft prompts are introduced as supplementary inputs to unleash the extensive knowledge of LVLMs. Extensive experiments verify that FKA-Owl shows superior cross-domain performance compared to the state-of-the-art methods.  
 
 Appendix:  


Decision: All the reviewers suggest to accept this submission, giving final rating as weak accept.

Reviews: review: paper_summary: paper_summary: This paper presents a novel framework called FKA-Owl to address the challenge of detecting multimodal fake news that combines both text and images. The proposed method leverages Large Vision-Language Models (LVLMs) and enhances them with forgery-specific knowledge to better detect manipulations. FKA-Owl incorporates two types of forgery-specific knowledge: semantic correlation between text and images and artifact traces in image manipulation. Two specialized modules are designed to extract these features and inject them into LVLMs. The framework demonstrates superior cross-domain performance in detecting fake news compared to existing methods.
strength_weakness: strength_weakness: 1）By integrating forgery-specific knowledge, FKA-Owl significantly improves the ability of LVLMs to detect subtle manipulations in multimodal content that traditional models might miss.
2）The introduction of specialized modules for encoding semantic correlations and artifact traces is particularly innovative, enhancing the model's capability to interpret complex multimodal data.
limitations: limitations: Add case study to demonstrate the role of Forgery specific knowledge in detecting fake news.

FKA-Owl represents a significant advancement in the field of multimodal fake news detection. By combining the strengths of LVLMs with forgery-specific knowledge, the framework addresses critical challenges related to domain shift and cross-modal manipulations. The comprehensive evaluation and ablation studies provide strong evidence of its effectiveness.
paper_summary: paper_summary: The paper leverage Large Vision-Language Models (LVLMs) to tackle the domain shift issue in multimodal fake news detection. Besides, the paper utilizes multi-modal semantic discrepancies and image artifact tracing to augment LVLMs. The experimental results indicate that the proposed method achieves good cross-domain detection performance.
strength_weakness: strength_weakness: 1. The explore of LVLMs for fake news detection is interesting and worthy of investigation. 
2. To address the limitations of directly applying LVLMs, this article introduces the forgery-specific knowledge to enhance LVLMs. And ablation experiments demonstrate the effectiveness of the proposed method.
limitations: limitations: 1. The proposed method is based on two assumptions, 1) the inconsistency between the text and image, and 2) a local part of the image is tempered. It is not clear what will happen when the senmentic of the image and the text are consistent or the image is generated by AI techniques.
2. The techniqucal contribution seems limited, where most of the components are borrowed from the existing techniques.
3. It is not clear how the instruction template would affect the performance.
paper_summary: paper_summary: The paper aims at leveraging forgery-specific knowledge to augment LVLMs for combating fake news.
strength_weakness: strength_weakness: The proposed method focuses on cross-modal semantic correlation and artifact traces of image manipulation. Authors conduct some experiments for validation. The proposed method achieves good cross-domain performance.
limitations: limitations: - The author propose to leverage artifact-related knowledge to augment LVLMs for fake news detection. However, the chosen validation datasets are not real-world fake news datasets, which contain fake news that are spreading or have spread. The DGM4 dataset construct its false samples by intentionally applying forgery methods. Fakeddit collects false samples from subreddits include “photoshopbattles submission”, “fakealbumcovers”, “photoshopbattles”, and so on. Both datasets contain deliberately forged samples, which makes them suitable for multi-modal manipulation detection, but not appropriate for testing fake news detection ability.

- The cross-domain ability of FKA-Owl may stem from the common forgery methods shared by the dataset but not by the cross-domain reasoning ability. What is the cross-domain performance without visual-artifact localization?