# MVACLNet: A Multimodal Virtual Augmentation Contrastive Learning Network for Rumor Detection
###### Abstract  
In today's digital era, rumors spreading on social media threaten societal stability and individuals' daily lives, especially multimodal rumors. Hence, there is an urgent need for effective multimodal rumor detection methods. However, existing approaches often overlook the insufficient diversity of multimodal samples in feature space and hidden similarities and differences among multimodal samples. To address such challenges, we propose MVACLNet, a Multimodal Virtual Augmentation Contrastive Learning Network. In MVACLNet, we first design a Hierarchical Textual Feature Extraction (HTFE) module to extract comprehensive textual features from multiple perspectives. Then, we fuse the textual and visual features using a modified cross-attention mechanism, which operates from different perspectives at the feature value level, to obtain authentic multimodal feature representations. Following this, we devise a Virtual Augmentation Contrastive Learning (VACL) module as an auxiliary training module. It leverages ground-truth labels and extra-generated virtual multimodal feature representations to enhance contrastive learning, thus helping capture more crucial similarities and differences among multimodal samples. Meanwhile, it performs a Kullback-Leibler (KL) divergence constraint between predicted probability distributions of the virtual multimodal feature representations and their corresponding virtual labels to help extract more content-invariant multimodal features. Finally, the authentic multimodal feature representations are input into a rumor classifier for detection. Experiments on two real-world datasets demonstrate the effectiveness and superiority of MVACLNet on multimodal rumor detection. rumor detection; multimodal learning; data augmentation; contrastive learning; social media analysis 202417999105091991991991991991991991991991991991991991991991991991919919919199199199191991991991991991991919919919199199199199199199191991991919919199199199191991919919191991919199191991919919199191919191919199191919191991919191919919191919919199191919191919191 With the development of multimedia technology, rumors spreading on social media have gradually changed from textual forms to multimodal forms with images. Multimodal rumors with images can often create an immersive reading experience to attract readers and give the illusion of high reliability to mislead readers, making it easier to rapidly spread the rumor. However, traditional rumor detection methods [3; 4; 5; 6; 7; 8; 9; 10; 11; 12; 13; 14; 15; 16] primarily utilize features extracted from text to detect rumors, which limits their ability to detect such multimodal rumors. Furthermore, previous studies [17; 18; 19] have demonstrated that image content in posts effectively serves as supplementary information for rumor detection, thereby improving detection performance. For these reasons, an increasing number of researchers [19; 20; 21; 22; 23; 24; 25; 26; 27; 28; 29; 30] have turned their attention to multimodal rumor detection to mitigate the adverse impact of multimodal rumors. In the real world, aside from rumors unintentionally created and spread by regular users, there are also malicious rumors that are carefully crafted and deliberately spread by rumor creators. Whether through content tampering or relationship network manipulation, the intention of these rumor creators is to make rumors appear indistinguishable from non-rumors in feature space, thus confusing detection models. Moreover, when humans assess the veracity of a post, they generally attempt to find similarities and differences between the post and other confirmed posts to help identify rumors. These phenomena make it vital to consider enhancing the diversity of multimodal samples in feature space and capturing hidden similarities and differences among multimodal samples to assist rumor detection models in adapting to a broader range of complex data distributions and understanding underlying data structures and patterns more effectively for further improvement of their detection performance.