Multimodal Fusion with Dual-Attention Based on Textual Double-Embedding Networks for Rumor Detection  
[PERSON]  
1 Xinjiang Multilingual Information Technology Laboratory, Xinjiang Multilingual Information Technology Research Center, College of Information Science and Engineering, Xinjiang University, Urumqi 830017, China 1  
[PERSON]  
2 School of Software, Xinjiang University, Urumqi 830017, China 2  
[PERSON]  
2 School of Software, Xinjiang University, Urumqi 830017, China 2  
[PERSON]  
2 School of Software, Xinjiang University, Urumqi 830017, China 2  
[PERSON]  
1 Xinjiang Multilingual Information Technology Laboratory, Xinjiang Multilingual Information Technology Research Center, College of Information Science and Engineering, Xinjiang University, Urumqi 830017, China 1  
Footnote 1 on textual content, social information, or communication structure content, some of the works use feature-based structure detection methods with textual and user information as the main features. In contrast, other works consider factors such as transmission time, transmission structure, and language features.Among the rumor detection methods using visual features combined with textual features, [PERSON] et al. [2] extracted text and social context features using a recurrent neural network (RNN) [3] and extracted image features using a pretrained VGG-16 network, after which the text and image features were stitched together and put into the classifier for detection. [PERSON] et al. [4] added an event identifier to understand common features among events, which helped to detect unanticipated new events. [PERSON] et al. [5] proposed a multimodal variational self-encoder to discover the correlation between modalities. [PERSON] et al. [6] designed a similarity-aware rumor detection method that combined similarity features with the connection of text features and visual features to identify fake posts. [PERSON] et al. [7] carried out rumor detection by purposefully extracting key feature information related to the detection target mode from a mode. [PERSON] et al. [8] used an adaptive aggregation of single-mode features and cross-mode correlation to improve the accuracy of rumor detection by analyzing cross-mode ambiguity learning problems. In summary, many methods have been proposed to detect rumors. Although previous research has made great progress, two problems remain:  
* Challenge 1: How to effectively utilize the internal dependencies of text and the dependencies of the local and global features of images to assist in the construction of multimodal rumor features? * Challenge 2: How to make full use of the dense interaction information within and between modalities to increase the granularity of understanding and improve rumor detection performance? To deal with the above challenges, a multimodal fusion neural network (dual-attention based on textual double embedding, TDEDA) based on textual double embedding is used for information interaction between text and visual objects. The model has three parts: modal feature extraction, modal feature fusion, and rumor detector. The multimodal feature extraction part includes a text feature extraction and visual feature extraction: for the text feature extraction, we use a pretrained BERT (Bidirectional Encoder Representations from Transformers) [9] to extract word features and sentence features; for the visual feature extraction, we use a pretrained ResNet50 [10] to obtain visual object features. The multimodal feature fusion part includes a text self-attention module and a visual self-attention module, and a text-visual coattention module. Finally, the fused textual and visual features are fed into the rumor detector for the multimodal rumor detection. The three main contributions of this paper are the following:  
* A multimodal fusion neural network (TDEDA) is proposed for rumor detection.