# FMFN: Fine-Grained Multimodal Fusion Networks for Fake News Detection
## 3 Model
### Feature Fusion  
Transformer is originally used for machine translation tasks. For a task to translate from English to French, the transformer draws dependencies between English sentences and French sentences thanks to the scaled dot-product attention. We apply the scaled dot-product attention to multimodal fusion so as to capture dependencies between textual features and visual features. In addition, the scaled dot-product attention also can be used to capture global information between these visual features since we extract multiple visual features instead of a visual representation.  
Motivated by the above observations, scaled dot-product attention (See Figure 2) is used for fine-grained fusion of textual features and visual features. The scaled dot-product attention block is defined as \(Scaled DotProductAttn(Queries,\ Keys,\ Values)\), where \(Queries\), \(Keys\) and \(Values\) are mapped into three representations \(Q\), \(K\), and \(V\) with three linear layers, then the scaled dot-product attention is computed on \(Q\), \(K\), and \(V\).  
We first enhance the visual features and the textual features using scaled dot-product attention blocks, which can capture global information. For visual features, it enables these  
Figure 2: The scaled dot-product attention.  
features to be further correlated, although global features are obtained by deep CNNs. The process is as follows.  
\[R_{V}^{1}=ScaledDotProductAttn(R_{V},\ R_{V},\ R_{V}) \tag{1}\]  
\[R_{V}^{2}=ScaledDotProductAttn\Big{(}R_{V}^{1},\ R_{V}^{1},\ R_{V}^{1}\Big{)} \tag{2}\]  
\[R_{V}^{M}=ScaledDotProductAttn\Big{(}R_{V}^{M-1},\ R_{V}^{M-1},\ R_{V}^{M-1}\Big{)} \tag{3}\]  
where \(M\) is the number of the scaled dot-product attention blocks and \(R_{V}^{M}=\left[v_{1}^{M},\ v_{2}^{M},\ \ldots,\ v_{m}^{M}\right]\) represents a number of enhanced visual features. Several scaled dot-product attention blocks (The number of the blocks is \(N\)) are also applied to the textual features \(R_{T}\) to obtain \(R_{T}^{N}=\left[t_{1}^{N},\ t_{2}^{N},\ \ldots,\ t_{n}^{N}\right]\) in the same way.  
Then, two scaled dot-product attention blocks are utilized to refine the enhanced visual features \(R_{V}^{M}\) and the enhanced textual features \(R_{T}^{N}\), respectively. The process to refine the visual features \(R_{V}^{M}\) is as follows.  
\[R_{V}^{\prime}=ScaledDotProductAttn\Big{(}R_{T}^{N},\ R_{V}^{M},\ R_{V}^{M}\Big{)} \tag{4}\]  
The \(R_{V}^{\prime}=\left[v_{1}^{\prime},\ v_{2}^{\prime},\ \ldots,\ v_{m}^{\prime}\right]\) are the refined visual features representing the fine-grained fusion with the textual features \(R_{T}^{N}\). Note that the queries come from the enhanced textual features, and the keys and the values come from the enhanced visual features. Therefore, it can capture the dependencies between visual features and textual features. The \(R_{T}^{\prime}\) is also obtained by computing the scaled dot-product attention, where queries come from the enhanced visual features, and the keys and the values come from the enhanced textual features.  
Finally, the refined features \(R_{V}^{\prime}\) and \(R_{T}^{\prime}\) are transformed to two vectors \(v\) and \(t\) by the averaging. The process of averaging the refined features \(R_{V}^{\prime}\) to produce the vector \(v\) is as follows.  
\[v=\frac{v_{1}^{\prime}\oplus v_{2}^{\prime}\oplus\ldots\oplus v_{m}^{\prime}} {m} \tag{5}\]  
where \(\oplus\) denotes element-wise sum. The two vectors \(v\) and \(t\) are concatenated into a vector \(r\) as the joint representation, which not only considers the correlations between different visual features but also reflects the dependencies between textual features and visual features.