# MVACLNet: A Multimodal Virtual Augmentation Contrastive Learning Network for Rumor Detection
## 2 Related Work
### Multimodal Rumor Detection Methods  
Existing multimodal rumor detection methods simultaneously utilize features extracted from both text and images to detect rumors. Some works added auxiliary tasks to the multimodal rumor detection task to improve detection performance [20; 21; 22]. Specifically, [PERSON] et al. [20] designed an event adversarial neural network (EANN), which introduces an event discrimination task to capture event-invariant features by an event discriminator with the adversarial method. [PERSON] et al. [21] proposed a multimodal variational autoencoder (MVAE), which introduces a multimodal reconstruction loss to learn shared feature representation between textual and visual modalities by the variational autoencoder. [PERSON] et al. [22] devised a similarity-aware detection method called SAFE. They defined an extra detection loss based on the relevance score between modalities to help identify rumors, where the score is calculated through a modified cosine similarity algorithm. Moreover, with the tremendous success and advancement of pre-trained models, more and more methods extract multimodal features through pre-trained models for learning deeper and more complex semantics [23; 24; 26; 27; 28; 29; 30]. For instance, [PERSON] et al. [23; 24] first used different pre-trained models to obtain both textual and visual feature representations. After that, using pre-trained models to extract multimodal features became the norm. In addition, considering that only employing a simple concatenation to fuse textual and visual features would discard associations between different modalities, many approaches explored more effective strategies to sufficiently utilize the relationships between modalities [25; 26; 27; 28; 29; 30]. For example, [PERSON] et al. [25] employed LSTM to obtain a joint representation of text and social context. Then, they designed a neuron-level attention mechanism to capture correlations between it and visual feature representations. Based on the better fusion of multimodal features by using the crossmodal weight-sharing layer and attention mechanism, [PERSON] et al. [28] additionally considered visual tampering features and the semantic consistency of multimodal features. They defined an extra detection loss based on the cosine similarity between textual and visual semantic features to help detect rumors. [PERSON] et al. [29] first defined an auxiliary correlation learning task to help achieve crossmodal feature alignment. Then, they adaptively aggregated unimodal and crossmodal fusion features based on an ambiguity score between modalities learned by estimating the KL divergence between distributions of textual and visual features. Moreover, [PERSON] et al. [30] additionally considered multilevel modal features to obtain finer-grained representations. They used different encoding layers of BERT to capture different levels of textual semantic features and aggregated them in layers. Then, they used attention-based intramodal and intermodal fusion blocks to capture corresponding correlations, thus obtaining higher-order fusion features.  
Although these multimodal rumor detection methods achieve relatively excellent performance, they mostly overlook the insufficient diversity of multimodal samples in feature space and hidden similarities and differences among multimodal samples, which limits the further improvement of their detection performance. Additionally, in recent years, contrastive learning has achieved remarkable performance improvements in various fields [32; 33; 34; 35; 36; 37]. Meanwhile, the idea of data augmentation through interpolation has been proven to be effective [38]. With such considerations and inspired by these studies, we designed the Virtual Augmentation Contrastive Learning (VACL) module as an auxiliary training module to improve detection performance.