MMDFND: Multi-modal Multi-Domain Fake News Detection

Authors: yu  tong
weihai  lu
zhe  zhao
song  lai
tong  shi
  mmdfnd
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 14:13:46

Abstract: Recently, automatic multi-domain fake news detection has attracted widespread attention. Many methods achieve domain adaptation by modeling domain category gate networks and domain-invariant features. However, existing multi-domain fake news detection faces three main challenges: (1) Inter-domain modal semantic deviation, where similar texts and images carry different meanings across various domains. (2) Inter-domain modal dependency deviation, where the dependence on different modalities varies across domains. (3) Inter-domain knowledge dependency deviation, where the reliance on cross-domain knowledge and domain-specific knowledge differs across domains. To address these issues, we propose a Multi-modal Multi-Domain Fake News Detection Model (MMDFND). MMDFND incorporates domain embeddings and attention mechanisms into a progressive hierarchical extraction network to achieve domain-adaptive domain-related knowledge extraction. Furthermore, MMDFND utilizes Stepwise Pivot Transformer networks and adaptive instance normalization to effectively utilize information from different modalities and domains. We validate the effectiveness of MMDFND through comprehensive comparative experiments on two real-world datasets and conduct ablation experiments to verify the effectiveness of each module, achieving state-of-the-art results on both datasets. The source code is available at https://github.com/yutchina/MMDFND.

Main paper: 
 introduction: 
 With the rapid advancement of communication technologies, individuals now have the ease of posting everyday information on social media and online platforms. However, certain users deliberately fabricate and disseminate false information to mislead the public, incite emotions, or achieve specific political or economic objectives, thereby causing severe consequences for societal stability. Moreover, distinguishing between true and false information often proves challenging for people, leading to the rapid spread of fake news. Manually verifying facts is not only costly but also timeconsuming, hence, automatic Fake News Detection (FND) have garnered widespread attention [8, 45, 46] .Existing methods for fake news detection often focus solely on single-domain news, such as health or politics. However, news on social media in the real world frequently originates from multiple domains. This presents two challenges for single-domain fake detection methods: (1) When dealing with smaller, single-domain news datasets, the effectiveness of the model is significantly impacted due to the scarcity of training data; (2) The accuracy of single-domain fake news detection methods greatly diminishes on cross-domain news datasets. To address these challenges, some multi-domain fake news detection (MFND) methods have been proposed. Some previous works have employed a hard-sharing mechanism to learn domain-specific knowledge and cross-domain knowledge for multidomain fake news detection [3, 31] . In addition, some methods based on a soft-sharing mechanism utilize domain-class gated networks to adjust the weights of multi-perspective information in text semantics, thereby achieving domain-adaptive multi-domain fake news detection [24, 47] . However, there are three problems with current multi-domain fake news detection methods: (1) Inter-domain modal semantic deviation: Texts and image descriptions from different domains exhibit varying degrees of semantic differences [2, 4, 10, 12, 20, 25, 33] , which may lead to drastically different interpretations of the same news across different domains. For example, Figures 1(b ) and 1(c) illustrate two news articles about "viruses, " where in the "science" domain, "virus" typically refers to pathogens causing everyday diseases, while in the "technology" domain, it might refer to computer viruses. Additionally, Figures 1(d ) and 1(e) display images from different news articles in two domains, where in the "science" domain, they often depict biological viruses, while in the "technology" domain, they may represent schematics of computer viruses. Even similar images may have different semantics in different domains. However, the existing MFND methods are all unimodal and have not taken into account the semantic bias between domains of multimodal information such as images and text.(2) Inter-domain modal dependency deviation: Different domains vary in their reliance on modal information for decisionmaking, necessitating the consideration of multiple modalities [22, 35] . Some domains may prioritize the authenticity of text, while others may focus more on the authenticity of images, and certain scenarios may require the simultaneous consideration of both text and images for decision-making. For instance, the same artificially manipulated image may be classified as fake news in the "political" domain but as real news in the "entertainment" domain. Thus, different domains exhibit differences in the reliance on textual features, visual features, and multimodal features of the same news. However, existing MFND methods have not accounted for this dependency deviation during the modeling process, potentially leading to difficulties in achieving optimal performance.(3) Inter-domain knowledge dependency deviation: Different domains exhibit varying degrees of reliance on cross-domain knowledge and domain-specific knowledge for decision-making. For example, domains with a higher density of specialized terminology may rely more on domain-specific knowledge, while others may depend more on cross-domain knowledge. However, existing MFND methods inadequately utilize domain-relevant knowledge by simply concatenating cross-domain knowledge and domain-specific knowledge, resulting in poor cross-domain predictive performance.To address these challenges, we propose a multi-modal multidomain fake news detection model called MMDFND. Initially, we extract news information from three distinct modal views. Specifically, we encode images using pre-trained CLIP [30] and MAE [13] encoders, while employing pre-trained CLIP text encoder and BERT [9] encoder for textual content. By fusing features from CLIP encodings through cross-modal similarity, we obtain multi-modal representations. To alleviate semantic biases across domains, we introduce the Domain Progressive Layered Extraction (DPLE) network for extracting cross-domain and domain-specific knowledge from different modalities. Subsequently, we fuse domain-relevant knowledge from different modal views using the Stepwise Pivot Transformer network to mitigate domain-modal dependency biases. Finally, we reweightedly aggregate domain-specific and cross-domain knowledge through Adaptive Instance Normalization. Through this approach, MMDFND effectively mitigates biases in domain knowledge dependencies, facilitating the effective utilization of domain-relevant knowledge across different modal views, thus enhancing the performance of multi-domain fake news detection. Our contributions are as follows:• We propose MMDFND, a multi-modal multi-domain fake news detection model that comprehensively models information from different modalities across domains.  
related works 2.1 unimodal fake news detection: 
 Existing unimodal fake news detection technologies primarily depend on either single image information or single textual information. In terms of textual information analysis, [28] detects fake news by capturing linguistic information in the text of articles and generating new user responses. [1] utilizes TM's conjunction clauses to capture lexical and semantic properties of true and false news texts, achieving the detection of fake news.In addition to detecting fake news based on content features, text sentiment [43] , writing style [26] , and discourse-level structure [15] are also widely used in the detection of fake news. In the analysis of image information, MVNN [27] identifies fake news by integrating complex patterns of fake news images in the frequency domain with visual features in the pixel domain. However, these unimodal methods neglect crossmodal features and fail to utilize modal correlation information in the original data, such as the consistency and relevance between different modalities.Recently, some methods have utilized cross-modal discriminative patterns to detect fake news and have achieved outstanding performance [21, 40] . To learn a shared representation of multimodal information, [16] proposes a multimodal variational autoencoder, which can reconstruct a multimodal representation from the learned probabilistic latent models. CAFE [6] computes the Kullback-Leibler (KL) divergence between different unimodal features to measure cross-modal consistency. Before the final classification, the crossmodal consistency is used to adjust the weights of both unimodal and multimodal features. [37] guides the training of unimodal networks through the relevance of cross-modal features, then fuses the trained unimodal features for the detection of fake news. In this paper, we fuse features from different modal views using the Stepwise Pivot Transformer and adjust the fused weights adaptively, thereby effectively leveraging information from various modal views features.In the real world, news data often comes from diverse domains. The goal of multi-domain learning is to model data from multiple domains simultaneously, enhancing the performance of individual domains and thereby improving overall performance. Several approaches [7, 11, 19, 38] based on multi-domain learning have achieved remarkable results in the detection of fake news. MD-FEND [24] innovatively applies multi-domain learning to the detection of fake news. Specifically, it integrates representations extracted by multiple experts by feeding domain information into a gating network. To achieve multi-domain fake news detection, [47] extracts semantic, sentiment, and stylistic representations from textual information. It then adaptively aggregates the information of these three representations using a domain memory bank. However, these methods merely adjust the weights of different views or expert representations in text semantics based on domain embedding representations, failing to effectively learn and utilize domain-invariant and domain-specific information. EDDFN [31] learns features from domain-specific and cross-domain embeddings, then concatenates domain-relevant knowledge for fake news identification. However, this approach fails to consider the varying dependencies different domains have on cross-domain knowledge and domain-specific knowledge. Furthermore, all the above methods are single-modal approaches, which struggle to leverage the visual modality information present in news articles. In comparison, we extract cross-domain knowledge and domain-specific knowledge using domain-shared and domain-specific experts. We then aggregate domain-relevant representations of text, visual, and fused modalities through the pivot transformer, and reweight and aggregate cross-domain knowledge and domain-specific knowledge, effectively utilizing domain knowledge from multiple modal views. 
problem statement: 
 Each input of multimodal news is represented as N = [I, T] ∈ D, where I, T, and D respectively represent the image, text, and dataset. The news in the dataset is categorized into 𝑘 classes, each assigned a domain label 𝑑 ∈ {𝐷𝑜𝑚𝑎𝑖𝑛 1 , . . . , 𝐷𝑜𝑚𝑎𝑖𝑛 𝑘 }. Within the dataset, the quantity of news varies significantly across domains, with some domains featuring a large number of articles, while others have relatively few. Given a news piece N that incorporates both text and image information, and a domain label 𝑑, the objective of multi-modal multi-domain fake news detection is to determine the authenticity of the news piece. 
methodology: 
 In this section, we introduce our model, MMDFND, which achieves multi-domain fake news detection by extracting and fusing crossdomain and domain-specific knowledge from multiple modal views, and then reweighting these pieces of knowledge. The structure of the model is illustrated in Figure 2 . Given an image-text pair, we first extract features from both single-modal and multi-modal views ( §4.1). Subsequently, our approach comprises three crucial components: a module for extracting cross-domain and domain-specific knowledge ( §4.2), a module for fusing multi-view representations ( §4.3), and a module for reweighting the cross-domain and domainspecific knowledge ( §4.4).Employing pretrained models, we encode the image I and text T into single-modal embedding. We then merge the aligned single-modal embedding, obtained through the CLIP [30] encoder, to derive a multimodal feature representation. 4.1.3 Multimodal View Feature. Multimodal features reflect the correlations between two modes, while also encompassing a broader spectrum of semantic characteristics, enabling a more comprehensive analysis of the authenticity of news [42] . Therefore, we extract multimodal representations, considering it as a unique view. However, the image and text features extracted separately by MAE and BERT exhibit a significant cross-modal semantic gap, making their direct fusion difficult. To address this challenge, we employ the CLIP model, trained on a large-scale dataset of image-text pairs. By utilizing the CLIP model's encoder to encode both images and text, we obtain aligned image and text embeddings. These embeddings are then concatenated and fed into an MLP network to generate the multimodal representation. To address the ambiguity issues introduced by the classifier, we utilize the CLIP cosine similarity as a cross-modal similarity measure to weight the multimodal features.The resulting multimodal feature is denoted as 𝑒 𝑚 : EQUATIONEQUATIONIn this stage, our objective is to extract and learn both cross-domain knowledge and domain-specific knowledge from the representations obtained in the initial stage. To achieve this, we propose a Domain Progressive Layered Extraction (DPLE) model that enables the separate learning of cross-domain knowledge and domain-specific knowledge within the image, text, and multimodal representations.As illustrated in Figure 3 , the DPLE model comprises crossdomain experts, domain-specific experts, a gating network, domain embeddings, attention mechanism, and a classifier. The original PLE model assigns task-specific expert networks for each task. As depicted in Eq. ( 3), the PLE model extracts task-shared information and task-specific information from the data by sharing expert networks and task-specific expert networks, respectively, to model multi-task relationships. Specifically, the input 𝑥 is equally sent to both the shared expert network and the task-specific expert network. Subsequently, the gating network employs a SoftMax function to perform an adaptive weighted aggregation of both the shared and the task-specific expert networks, thus yielding the final representations for the different tasks. The variable 𝑘 denotes the number of distinct tasks.EQUATIONwhere 𝐺 𝑘 (𝑥) ∈ 𝑅 (𝑚 𝑠 +𝑚 𝑘 ) ×𝑑 represents the output of the gate network specific to the task 𝑘, and 𝑑 is the dimension of input representation. 𝑆 𝑘 (𝑥) represents the selection matrix consisting of chosen vectors, which includes 𝑚 𝑘 task-specific experts for task 𝑘 and 𝑚 𝑠 shared experts, 𝑚 𝑘 and 𝑚 𝑠 are two hyperparameters: 𝑆 𝑘 (𝑥) = 𝐸 𝑇 (𝑘,In MMDFND, we treat fake news detection in different domains as distinct tasks. In this paper, we take two domains as examples. We leverage DPLE to unearth both cross-domain and domain-specific knowledge. We enhance the PLE network from two perspectives. Firstly, we compute the importance scores for each token using an MLP network with shared weights based on an attention mechanism, and perform aggregation according to these scores. Subsequently, we feed the aggregated low-dimensional representation into a gating network as part of its input. Secondly, acknowledging that domain experts excel more in specific fields than cross-domain experts, we aim to select appropriate experts via the gating network. Therefore, we incorporate a learnable feature 𝑒 𝑑 , namely domain embedding, as another input to the gating network, thereby guiding the selection process of the gating network.We revise Eq. (3) as Eq. (5) for DPLE:E , A A m E , 1 A E , 2 S E , S S m E , 1 S E , 2 B E , B B m E , 1 B E  domainAEQUATIONwhere 𝑡 represents the number of tokens, and MLP 𝑘 denotes the attention network used for domain 𝑘. 𝑘 ∈ [1, 2] .The classifier consists of an MLP network and a Softmax output layer. By inputting the results of the two gating networks into their corresponding classifiers, the predicted values for the two domains can be obtained:EQUATIONThe purpose of fake news detection is to determine the veracity of news articles. We use ŷ𝑘 to represent the predicted label in domain 𝑘 and 𝑦 𝑘 to represent the true value in domain 𝑘. We employ the Binary Cross-Entropy (BCE) loss as the loss function in domain 𝑘:EQUATIONBy calculating the weighted sum of the losses across different domains, we obtain the joint loss for multi-domain learning:EQUATION)Where 𝐾 is the number of domains. Constrained by the loss function of the DPLE network, we utilize the cross-domain expert network and domain-specific expert networks to extract cross-domain knowledge of visual modality, denoted as 𝑒 𝑐 𝑖 , and domain-specific knowledge for two domains, represented as [𝑒 𝑑1 𝑖 , 𝑒 𝑑2 𝑖 ], from 𝑒 𝑖 . From 𝑒 𝑡 , we extract cross-domain knowledge of textual modality, denoted as 𝑒 𝑐 𝑡 , and domain-specific knowledge for the same two domains, represented as [𝑒 𝑑1 𝑡 , 𝑒 𝑑2 𝑡 ]. From 𝑒 𝑚 , we extract cross-domain knowledge of multimodal, denoted as 𝑒 𝑐 𝑚 , and domain-specific knowledge for the same two domains, represented as [𝑒 𝑑1 𝑚 , 𝑒 𝑑2 𝑚 ]. To effectively utilize domain-specific knowledge and domain-shared knowledge from different modalities, domain-shared knowledge and domainspecific knowledge from the same domain will be fused in subsequent steps through a pivotal Transformer.In the knowledge fusion stage, our goal is to integrate domainspecific knowledge and cross-domain knowledge from different views. To achieve this, we draw upon ideas from MMSBR [44] and propose the Stepwise Pivot Transformer. where 𝑆 𝑐 𝑡 , 𝑆 𝑐 𝑖 and 𝑆 𝑐 𝑚 represent the cross-domain knowledge feature sequences of text, image, and multimodal views, respectively, and 𝑅 denotes the length of the sequence.A standard Transformer layer primarily consists of three components: Multi-Head Attention (MHA), Layer Normalization (LN), and Feed-Forward Networks (FFN). Therefore, the Transformer layer as 𝐹 𝑙+1 = Trans(𝐹 𝑙 ) can be represented as follows, where 𝐹 𝑙 is the input sequence, and 𝐹 𝑙+1 represents the output sequence:EQUATIONEQUATIONIn the initial Transformer layer, we create a pivot P = [𝑝 1 , . . . , 𝑝 𝑀 ] to gradually fuse information from different views. Taking the fusion of cross-domain knowledge from various views as an example, the Stepwise Pivot Transformer integrates information from visual (𝑆 𝑐 𝑖 ), multimodal (𝑆 𝑐 𝑚 ) and textual (𝑆 𝑐 𝑡 ) sequences through the following process:EQUATIONEQUATION[𝑆 𝑐,𝑙+1 𝑚 , P 𝑙 𝑚 ] = Trans([𝑆 𝑐,𝑙 𝑚 , P 𝑙 ])EQUATIONEQUATIONEQUATIONwhere 𝑆 𝑐,0 𝑖 = 𝑆 𝑖 , 𝑆 𝑐,0 𝑚 = 𝑆 𝑚 , 𝑆 𝑐,0 𝑡 = 𝑆 𝑡 and P 0 = P = (𝑆 𝑐,0 𝑡 + 𝑆 𝑐,0 𝑖 + 𝑆 𝑐,0 𝑚 )/3, In the above process, the pivot integrates information from different views. For instance, as illustrated by Eq. ( 16), the pivot, which has merged textual information, provides insights from the textual view to the image view, thereby fusing information from both text and image views through the Transformer layer. To fully integrate information from different views, we stacked the Stepwise Pivot Transformer 𝑁 times. After the final fusion, the information encapsulated within the pivot P 𝑁 is extracted through an MLP network, denoted as 𝑓 𝑐 :EQUATIONThrough Stepwise Pivot Transformer, we can also obtain domainspecific information 𝑓 𝑑𝑘 for domain 𝑘 after multi-view fusion. 𝑘 ∈ [1, 2] . These domain related knowledge will be re weighted and aggregated in subsequent stages.After the process of knowledge fusion, we have obtained domainspecific knowledge and cross-domain knowledge that incorporate multi-view information. To effectively utilize this knowledge, in this stage, we adaptively reweight and aggregate domain-specific knowledge and cross-domain knowledge.We leverage Adaptive Instance Normalization (AdaIN) method for reweighting. By training the mean and variance of the original features through an MLP, and then using the trained mean and variance to guide the adjustment of the normalized feature representation, we can selectively enhance the knowledge that is more important to the current domain while reducing the impact of knowledge that contributes less. Taking the generation of the reweight feature 𝜔 𝑐 based on 𝑓 𝑐 as an example, the specific formula is as follows:EQUATION)EQUATIONSimilarly, we generate the reweighted representation for the domainspecific knowledge of domain 𝑘, denoted as𝜔 𝑑𝑘 . 𝑘 ∈ [1, 2].The reweighted representations [𝜔 𝑑𝑘 ,𝜔 𝑐 ] are concatenated and fed into the DPLE, resulting in a feature 𝑓 𝑘 𝑎𝑔𝑔 that is enriched with both domain-specific and cross-domain knowledge:EQUATION𝑓 𝑘 𝑎𝑔𝑔 is subsequently fed into the final classifier, yielding the ultimate prediction of news veracity, denoted as ŷ𝑘 . Given that fake news detection constitutes a binary classification challenge, we employ a Binary Cross-Entropy (BCE) loss for the final classifier, represented as L 𝑘 𝐹𝑖𝑛𝑎𝑙 . Moreover, according to Eq. ( 8), the losses for the text view, visual view, and fusion view branches within the DPLE model are represented as L 𝑡 , L 𝑖 , and L 𝑚 , respectively. The total loss for the DPLE model classification is the sum of L 𝑡 , L 𝑖 , and L 𝑚 , indicated as L 𝐷𝑃𝐿𝐸 . Hence, the overall loss for domain 𝑘 in the MMDFND model is presented as follows:EQUATIONThe DPLE loss is invariant across domains. Therefore, the total loss of the MMDFND model is computed as the sum of the losses from the final classifiers across different domains, augmented by the DPLE loss. The loss function for the MMDFND model is articulated as follows:EQUATIONWhere 𝐾 denotes the number of domains, while 𝛼 represents hyperparameter. 
experiments: 
 In this section, we empirically evaluate MMDFND through two benchmark tests covering news data from different domains, aiming to address the following research questions:• RQ1:Can our proposed MMDFND outperform the latest baselines in multi-domain fake news detection and multimodal fake news detection? • RQ2:What is the impact of each component in MMDFND?• RQ3: Can MMDFND improve the performance of fake news detection in data-scarce domains?5.1.1 Datasets. Our model is evaluated on two real-world datasets: Weibo [36] and Weibo-21 [24] . For the Weibo dataset, we adhere to the same data partitioning method as the benchmark [36, 41] . The training set consists of 3,749 real news items and 3,783 fake news items, while the testing set comprises 1,000 fake news items and 996 real news items. To meet the requirements of multi-domain datasets, we divided the Weibo dataset into nine domains, namely, finance, health, military, science, politics, international, education, entertainment, and society. Weibo-21, a new multi-domain and multi-modal dataset, contains a total of 4,640 real news items and 4,487 fake news items; it is divided into training and testing data following the partitioning approach of benchmark [47] . Besides, to ensure the quality of the entire dataset, we follow the same steps as in the work [14, 39] to remove duplicated and low-quality images.To conduct a comprehensive evaluation of our proposed model, we compare it with both multi-domain (MD) fake news detection methods and multimodal (MM) fake news detection approaches. MD1: MMoE [23] , which is a multi-domain model that shares a mixture of experts (MoE) across various domains.MD2: MoSE [29] , which employs Long Short-Term Memory networks (LSTM) as the experts within the MMoE framework.MD3: EDDFN [31] , which retains domain-specific and domainshared knowledge extracted from text and propagation information for multi-domain fake news detection.MD4: MDFEND [24] , which is a multi-domain fake news detection model that utilizes a domain gate to perform weighted aggregation of MoE experts.MD5: M 3 DFEND [47] , which replaces the experts in MDFEND with text semantic, sentiment, and style extractors, while substituting domain gating with domain adapters.MM1: EANN [36] , which is a model grounded in Generative Adversarial Networks (GAN) capable of learning event-invariant representations.MM2: SpotFake [32] , which utilizes VGG and BERT to extract image and text features, respectively, for fake news detection.MM3: CAFE [6] , which utilizes cross-modal ambiguity to adaptively aggregate unimodal features and cross-modal correlations.MM4: CMC [37] , which introduces a novel distillation method to extract cross-modal correlations during training.MM5: BMR [41] , which aggregates multi-view features and cross-modal consistency through weighted integration. In the text encoding section, we set the maximum input length for text at 197 characters. The pretrained BERT [9] and CLIP models are utilized to encode the text data respectively. For the visual encoding portion, the input images are resized to 224x224 pixels, and encoded separately using pretrained MAE [13] and CLIP models. We utilize TextCNN [5] as the expert for the textual view of DPLE, CNN [18] as the expert for the visual view of DPLE, and MLP as the expert for the multimodal view of DPLE. Our evaluation metrics encompass Accuracy, Precision, Recall, and the F1 Score. The experiments are conducted with a batch size of 64, employing the Adam optimizer [17] with an initial learning rate of 0.0001. The models are trained over 50 epochs with early stopping implemented to prevent overfitting. In the DPLE loss equation (Eq. 8), equal weight is assigned across different domains, whereas in the MMDFND model loss equation (Eq. 25), the parameters 𝛼 is set to 0.15. We set the number of both domainspecific experts 𝑚 𝑘 and shared experts 𝑚 𝑠 to 6.To assess our proposed method from both domain-specific and overall perspectives, we conduct comparisons between MMDFND and various cutting-edge multi-domain and multimodal fake news detection methodologies on the Weibo and Weibo21 datasets, respectively. As indicated in Table 1 , MMDFND notably surpasses other multi-domain strategies in F1 scores across several domains and overall. Demonstrated in Table 2 , MMDFND significantly outperforms alternative multimodal approaches in terms of accuracy and F1 scores across each dataset, substantiating the efficacy of our proposed model. Specifically, MMDFND achieves unprecedented accuracy rates of 93.4% on the Weibo dataset and 93.9% on the Weibo21 dataset, achieving a new state-of-the-art on Weibo and Weibo21 datasets.The results lead to the following conclusions:In multi-domain approaches, MDFEND and M 3 FEND outperform MMoE, MoSE, and EDDFN across the majority of domains. MMoE and MoSE employ a shared expert base with multiple independent heads for different domains. EDDFN learns domain-specific and cross-domain knowledge separately. These three methodologies all adopt hard sharing mechanisms to acquire cross-domain In multimodal approaches, EANN's neglect of semantic differences leads to poor fusion and the worst performance. SpotFake excels in rumor classification using pre-trained models for separate text and image features but performs weakly in non-rumor classification. CAFE improves multimodal fusion by aligning semantic spaces through auxiliary tasks, achieving better performance. CMC and BMR show strong overall performance by recognizing crossmodal correlations but fall short on cross-domain datasets due to their lack of cross-domain and domain-specific knowledge. The superiority of MMDFND over other state-of-the-art methods in multimodal and multi-domain datasets can be attributed primarily to the following reasons: (1) MMDFND combines hard and soft sharing mechanisms, using hard sharing to extract cross-domain and domain-specific knowledge, and soft sharing to aggregate and weigh this knowledge, enhancing predictive performance across multiple domains. ( 2) From an information-theoretic perspective, MMDFND uses cross-domain knowledge to enhance data-scarce domains, improving the predictive performance of these single domains and overall performance. (3) MMDFND aligns semantic spaces across modalities using CLIP's encoder, enhancing multimodal fusion. (4) MMDFND utilizes the Stepwise Pivot Transformer to integrate information from different modal views, enabling fake news detection from multiple angles.To evaluate the effectiveness of each component in the MMDFND framework, we perform comparative analyses by omitting each component individually. The experimental setups are as follows:(1) w/o MVF: The multi-view knowledge fusion module is omitted, and fusion is performed by concatenating features from different views. (2) w/o ITA: The CLIP encoder, which aligns the graphical and textual modalities, is removed. Encoding of different modalities is performed exclusively using the BERT and MAE models. Table 3 presents the results of ablation studies. We observe that the original MMDFND outperforms all variants, demonstrating the efficacy of each component. Moreover, our findings lead to the following conclusions:• MMDFND performs poorly without cross-domain knowledge, highlighting its crucial role in enhancing multi-domain learning.• The performance of MMDFND with the original PLE model drops significantly, proving our enhancements improve crossdomain and domain-specific knowledge extraction.To verify if our model improves fake news detection in data-scarce domains, we used t-SNE [34] to visualize features before each domain classifier (Figure 4 ). The results show that MMDFND excels even in domains with limited data, such as international, finance, and science. This is because MMDFND transfers knowledge from data-rich domains (like health, entertainment, and social) to enhance detection in data-scarce areas. Additionally, MMDFND adaptively reweights these knowledge sources, combining cross-domain and domain-specific insights to boost overall performance.  
 
 Appendix:  


Decision: This paper focuses on multi-domain multi-modal fake news detection, which exhibits a more complex scenario. To address the inter-domain and intra-domain challenges, a multi-modality and multi-domain fake news detection method MMDFND is proposed. All reviews provide positive final ratings and believe that this submission is above the acceptance threshold with minor revisions. I recommend the acceptance of this paper, but the authors are strongly required to revise this paper carefully according to the reviews. Specially, the claim about "the first XXX" should raise a very serious caution.

Reviews: review: paper_summary: paper_summary: In this study, a multi-modality and multi-domain fake news detection method MMDFND is proposed to incorporate domain embeddings and attention mechanisms into a progressive hierarchical extraction network to achieve domain-adaptive domain-related knowledge extraction, and it utilizes Stepwise Pivot Transformer networks and adaptive instance normalization to effectively utilize information from different modalities and domains. The experimental results have demonstrated the effectiveness of the proposed method.

However, the manuscript has suffered from the following problems.
1. Equation 2 may have a typo error, where there are two f-CLIP-I.
2. There are two L (Equation 8 and Equation 25), one is domain classfication loss, the other is fake news classfication loss. So, what is the final loss? Does final loss contain both losses? what's the relationships among LDomain, LDPLE and Li, Lt and Lm.
3. MOE  and CLIP is used to enhance the model's performance. Other works like [1-2] also use CLIP to help for debunking fake news at cross-modal setting. But the manuscript did not mention  and compare these methods. So, it is not clear, what makes the method achieve the better performance.
[1] Multimodal Fake News Detection via CLIP-Guided Learning. Arxiv 2022.
[2] Multi-modal Fake News Detection on Social Media via Multi-grained Information Fusion. ICMR 23.
strength_weakness: strength_weakness: A  multi-modality and multi-domain fake news detection method is proposed. From the pespective of cross-modal and cross-domain, it is new.
limitations: limitations: CLIP itself has the ablity of cross-modal and cross-domain generalization. MOE has the ability of permance boosting. It is not clear, what make the method proposed supass the other methods.
paper_summary: paper_summary: MMDFND sets a new benchmark in the field of fake news detection by effectively combining multi-modal and multi-domain information through advanced mechanisms like domain embeddings, attention, Stepwise Pivot Transformer networks, and adaptive instance normalization. The extensive experimental validation underscores its robustness and superior performance.
strength_weakness: strength_weakness: 1. The authors consider that existing methods  faces three main challenges: (1) Inter-domain modal semantic deviation, (2) Inter-domain modal dependency deviation, and (3) Inter-domain knowledge dependency deviation.
2. The authors propose MMDFND, a multi-modal multi-domain fake news detection model that comprehensively models information from different modalities across domains. MMDFND stands as the first model to address cross-domain fake news detection leveraging multi-modal domain-relevant information.
3. The proposed network achieves state-of-the-art performance on two public real-world datasets.
limitations: limitations: 1. The author's claim that this is the first to jointly model multi-modality and multi-domain in the fake news detection scenario is inaccurate[1,2].
2. The proposed network achieves state-of-the-art performance. But one thing that seems missing but potentially important is the efficiency (e.g., number of parameters in the model, number of examples that can be processed per second and time-sensitivity) of the system.
3. The related work section lacks an introduction to recent studies from the past one or two years. For example, the section on Unimodal Fake News Detection is missing research from 2022 and 2023, and the Multimodal Fake News Detection section is missing research from 2023.
4. The selection of parameters α, m_k, and m_s needs to be supported by experimental results, so the author can add related parameter analysis.
5. Please check if formulas 14 to 19 are written incorrectly.


[1] Silva A, Luo L, Karunasekera S, et al. Embracing domain differences in fake news: Cross-domain fake news detection using multi-modal data[C]//Proceedings of the AAAI conference on artificial intelligence. 2021, 35(1): 557-565.
[2] Zhang T, Wang D, Chen H, et al. BDANN: BERT-based domain adaptation neural network for multi-modal fake news detection[C]//2020 international joint conference on neural networks (IJCNN). IEEE, 2020: 1-8.
paper_summary: paper_summary: The paper is about fake news detection, which is an interesting topic. The authors propose a Multi-modal Multi-Domain Fake News Detection Model (MMDFND), the first to jointly model multi-modality and multi-domain in the fake news detection scenario. However, there are several concerns in the current version of the paper that addressing them will increase the quality of this paper.
strength_weakness: strength_weakness: [1] Novel ideas and research questions.

[2] Reasonable writing logic.

[3] sufficient experimental results.
limitations: limitations: 1 Multi-modal scenarios should be relatively common in fake news detection. The author claims that this is the first work. Is the main focus on multi-domain? In this regard, the author should provide further clarification.

2 Figure 2 can be further improved. At present, a clear model structure can be seen, but there needs to be a clear logic/process line.

3 Is it not detailed enough to use the same hyperparameter to adjust the weights of different loss functions? Have you considered using multiple hyperparameters to correspond to different loss functions?

4 Using graph technology to model multi-modal data seems to be an optional path, which is also common in fake news detection. It is recommended that the authors increase the discussion on this part. [1-3]
[1] Improving fake news detection by using an entity-enhanced framework to fuse diverse multimodal clues. 2021.
[2] TMac: Temporal multi-modal graph learning for acoustic event classification. ACM MM 2023.
[3] Multi-modal graph contrastive learning for micro-video recommendation. ACM MM 2022.
paper_summary: paper_summary: The paper titled "MMDFND: Multi-modal Multi-Domain Fake News Detection" addresses the challenges in automatic multi-domain fake news detection. The authors propose a novel model, MMDFND, which is the first to jointly model multi-modality and multi-domain in the fake news detection scenario.
strength_weakness: strength_weakness: 1. The use of domain embeddings and attention mechanisms in a progressive hierarchical extraction network seems innovative.
2. The authors conduct comprehensive comparative experiments on two real-world datasets, demonstrating the model's effectiveness and achieving state-of-the-art results.
limitations: limitations: 1.	Ambiguity in Terminology: The use of "another" in line 144 is ambiguous and could be clarified.
2.	Unused Parameters: Some parameters are introduced but not used in the text, formulas, or figures, such as “The hyperparameter 𝑛” in line 346.
3.	Inconsistent Abbreviations: The abbreviation for "Feed-forward Networks" is inconsistent, with "(FN)" used in line 512 and "(FFN)" in line 528.
4.	Inconsistent Formula and Figure Order: The order of formulas (14)-(19) does not match the sequence shown in Figure 1 for the "Stepwise Pivot Transformer Module".
paper_summary: paper_summary: The paper presents a novel approach to detecting fake news across multiple domains and modalities, titled "MMDFND: Multi-modal Multi-Domain Fake News Detection." It addresses key challenges in fake news detection that arise due to variations in semantics, modal dependencies, and knowledge dependencies across different domains. The proposed model, MMDFND, integrates multi-modality (text and images) and multi-domain (various content domains) considerations into a single framework.

Key features of MMDFND include the use of domain embeddings and attention mechanisms to adaptively extract domain-specific knowledge, and the incorporation of Stepwise Pivot Transformer networks and adaptive instance normalization to handle information from various modalities effectively.

The effectiveness of the MMDFND model is validated through comprehensive experiments on two real-world datasets, demonstrating its ability to outperform existing methods in accuracy and adaptability across different domains. The paper also includes detailed ablation studies to verify the importance of each component in the model.
strength_weakness: strength_weakness: 1.The paper introduces the MMDFND model, an innovative approach that uniquely combines multi-domain and multi-modal data, enabling it to tackle the complex and varied nature of fake news effectively across different content types and formats.

2. The model is robustly validated with extensive experiments on real-world datasets and detailed ablation studies. This comprehensive testing demonstrates the model’s effectiveness and the critical role of each component, enhancing the paper's credibility.

3. MMDFND achieves state-of-the-art performance, outperforming existing methods in fake news detection. This superior performance across diverse domains and modalities illustrates the model's potential to significantly impact the fight against misinformation.
limitations: limitations: 1. The manuscript introduces the Stepwise Pivot Transformer, which gradually integrates different types of information (textual, visual, multimodal) through a series of Transformer layers, detailing the process extensively. However, there may still be issues of information loss or insufficiency during each step of the information updating process. Information from different views may present conflicts or inconsistencies, posing a challenge on how to effectively address these conflicts.

2. Regarding the hyper parameter α in the formula, please provide some details about its selection and tuning. For example, how was the initial value of α determined? Was cross-validation performed to adjust the value of α? Such information helps readers understand the process and decisions involved in model training.

3. In the experimental setup, some details may require further explanation, such as why specific input lengths were chosen, why certain optimizers and learning rates were used, why specific parameters were set in the model's loss function, and so on.
paper_summary: paper_summary: This paper proposed MMDFND to solve the problem of multi-modal multi-domain fake news detection, which encodes images and text via different pre-trained models to get single-modal features and fused multi-modal features. Domain Progressive Layered Extraction modules are further exploited to extract cross-domain and domain-specific features for different model views. They exploit Stepwise Pivot Transformers to fuse multi-modal knowledge for cross-domain and domain-specific branches for further aggregation. Experimental results on Weibo datasets demonstrate the effectiveness of MMDFND.
strength_weakness: strength_weakness: The research question is important, and this paper proposed an effective method to solve multi-modal multi-domain fake news detection problem.
limitations: limitations: (1) As far as I know, **Knowledge augmented transformer for adversarial multidomain multiclassification multimodal fake news detection,
Neurocomputing** is an earlier research that focuses on multi-modal multi-domain fake news detection but missed in your related work and compared methods.

(2) I think the factuality of one image is independent of the domain, so can you provide some examples to prove **the same artificially manipulated image may be classified as fake news in the "political" domain but as real news in the "entertainment" domain.** mentioned in lines 106-108?

(3) I think the formula (5) is incorrect because $x$ does not represent a token but the input to one MLP module is a token of $x$ according to Figure 3. 

(4) In Section 4.5, this paper describes that the aggregated feature $f^k_{agg}$ is acquired by concatenating $\omega_{dk}$ and $\omega_{c}$ and feeding into the DPLE, which is not consistent with the Formula (23). 
Moreover, I am confused about the reason that you also use the DPLE in the aggregation stage, in which domain-specific and cross-domain knowledge is prepared well. How about just fusing them via concatenating or simple weighting？