# MVACLNet: A Multimodal Virtual Augmentation Contrastive Learning Network for Rumor Detection
###### Abstract
Unfortunately, most existing multimodal models tend to overlook these crucial considerations. Based on the above considerations and to explore a more effective method for multimodal rumor detection, we propose MVACLNet, a Multimodal Virtual Augmentation Contrastive Learning Network. In MVACLNet, we first design a Hierarchical Textual Feature Extraction (HTFE) module to extract textual features from multiple perspectives, namely local, global continuous, and global non-continuous, to achieve a more comprehensive utilization of text data. Then, we enhance the fusion of textual and visual features using a modified cross-attention mechanism, which operates from textual-visual and visual-textual perspectives at the feature value level, to obtain richer and more precise authentic multimodal feature representations. Following this, we devise a Virtual Augmentation Contrastive Learning (VACL) module as an auxiliary training module. This module serves three purposes. One is to generate extra virtual multimodal feature representations and corresponding virtual labels using an interpolation-based data augmentation strategy, regarded as augmented samples, to enhance the diversity of multimodal samples in feature space, thus strengthening the feature learning of the model. The second is to introduce contrastive learning and additionally leverage ground-truth labels and virtual multimodal feature representations to enhance it, thereby helping the model capture more crucial similarities and differences among multimodal samples. The third is to perform the Kullback-Leibler (KL) divergence constraint between predicted probability distributions of the virtual multimodal feature representations and their corresponding virtual labels to help the model capture more content-invariant multimodal features. In particular, the introduced ground-truth labels can provide effective supervisory signals for contrastive learning to prevent the model from misclassifying posts with noise. Finally, a Rumor Classification module is developed to perform rumor prediction on the authentic multimodal feature representations. Overall, with the help of VACL, MVACLNet can learn more robust and generalized multimodal feature representations, thereby improving its detection performance. In summary, the contributions of our paper are as follows:  
* We propose MVACLNet, a Multimodal Virtual Augmentation Contrastive Learning Network, which achieves more effective multimodal rumor detection. It consists of five modules: a Hierarchical Textual Feature Extraction module, a Visual FeatureExtraction module, a Multimodal Feature Fusion module, a Virtual Augmentation Contrastive Learning module, and a Rumor Classification module. Each designed module has a different role, and all the modules contribute to the improvement of detection performance. * We design a Hierarchical Textual Feature Extraction (HTFE) module to extract textual features from multiple perspectives in order to make comprehensive use of text data. * We utilize a modified cross-attention mechanism, which operates from different perspectives at the feature value level, to obtain richer and more precise multimodal feature representations. * We devise a Virtual Augmentation Contrastive Learning (VACL) module as an auxiliary training module to improve detection performance, which can help the model learn more robust and generalized multimodal feature representations by enhancing the diversity of multimodal samples in feature space to enhance feature learning, capturing more crucial similarities and differences among multimodal samples, and extracting more content-invariant multimodal features. * Experiments on two real-world datasets demonstrate the effectiveness and superiority of MVACLNet in multimodal rumor detection.