## 3 Multimodal Fusion Network Based on Attention Mechanism
### Multimodal Fusion Module
#### 3.2.3 Feature Fusion Module for Text-Visual Coattention  
Given an original tweet and an attached image, the words in the tweet are actually associated with the areas of the image that are usually associated with each other. Most existing multimodal rumor detection methods use only visual attention mechanisms, and a few methods use joint-attention mechanisms to capture these multimodal correlations between images and tweets. How to effectively integrate the information between different modes and capture the more extensive information interaction between text and image modes is still a problem that needs further study. In this section, we introduce a new mechanism that combines visual attention and textual attention for reasoning. We call it visual-textual joint attention. First, we cover the two basic attention blocks, as shown in Figure 4.  
These two blocks were inspired by the attention of the dot product adopted in the transformer model. The input to the scaled dot product attention consisted of the queries and keys of dimension \(d_{key}\) and the values of dimension \(d_{value}\). To simplify the process, we set \(d_{key}\) and \(d_{value}\) to have the same dimension d. We computed the dot product of queries  
Figure 4: Self-attention blocks and Coattention blocks. with all keys, divided each key by \(\sqrt{d}\), and applied the softmax function to get the attention weights of the values. At the same time, the representation of the participation features was improved by introducing a multihead attention, which consisted of h parallel \"heads\". Each head corresponded to an independent dot product attention function. Among the multiple self-attention (MSA) blocks [30], the coattention block [32] was a variant of its standard form, and this variant differed from previous approaches in terms of its modal fusion and the probabilistic computation of attention to obtain global dependencies for each position in the sequence. Thus, for multimodal rumors, we could obtain a richer representation of the features between the modalities. As shown in Figure 4a, the MSA had two components: a fully connected feed-forward network and a multiheaded self-attention function, respectively. Two components used a residual connection before performing a layer normalization. The MSA first computed (\(d\times 1\)) dimensional queries, keys, and values before inputting them to the matrices \(Q\), \(K\), and \(V\). Where the attention distribution of \(V\) was determined by the similarity of the dot product between \(Q\) and \(K\), and the attention function with \(m\) heads then contained \(m\) parallel self-attention functions. Then, for the \(i\)-th head, the input was obtained by transforming \(Q\), \(K\), and \(V\) as the following equation:  
\[Q_{i} =QW_{i}^{Q} \tag{7}\] \[K_{i} =KW_{i}^{K}\] \[V_{i} =VW_{i}^{V}\]  
where \(W_{i}^{Q}\), \(W_{i}^{K}\), \(W_{i}^{V}\in\mathbb{R}^{1\times d_{h}}\) are the projection matrices of the \(i\)th head, and \(d_{h}=d/m\) is the dimensionality of the output features of each head. The procedure for calculating the multiheaded self-attention function was as follows:  
\[MA(Q,K,V)=hW^{O} \tag{8}\] \[h=h_{1}\oplus h_{2}\oplus\cdots\oplus h_{m}\] \[h_{i}=A(Q_{i},K_{i},V_{i})=softmax(\frac{Q_{i}K_{i}^{T}}{\sqrt{d_ {h}}})V_{i}\]  
where \(W^{O}\in\mathbb{R}^{md_{h}\times 1}\), \(\oplus\) represents the cascade of vectors. The fully connected feed-forward network included, in addition to the intermediate ReLU activation function, two linear transformations. For an inner dimension of \(d_{ff}\), the input and output dimensions were \(d\times 1\). \[FFN(x)=max(0,xW_{1})W_{2} \tag{9}\]  
In Figure 4b, a common attention block (denoted hereafter as \"Co-Attn\") is shown, where the query comes from one modality and the key and value from another modality. In particular, the residual terms after the attention sublayer (multihead) were used by the query matrix, and the rest of the architecture was the same as that of MSA.