# Global Local Fusion Neural Network for Multimodal Sentiment Analysis
## 3 Proposed Model
### Local Fusion Learning
#### 3.3.2 Fine-Grained Fusion Module  
Although the cross-modal module builds the correlation between image objects and words, it is difficult to perfectly align words and image regions because the relationship between images and sentences is complex in social media. Therefore, a fine-grained fusion module is applied to control the coarse fusion feature adaptively, aiming to eliminate the noises generated in the cross-modal module and explore deeper interaction between image region and words. To the extent of matched parts fusion features and suppressing the mismatched pairs of fusion features, in the beginning, a gate is designed to compute the matching value of each fusion feature. Then, the gate weights control how much information is useful to the fusion cross-modal features. The gate weight will be high when the word and image content match well. On the contrary, when the image content and word are unpaired, the weight is low, and the gate weights in the visual aspect are shown as follows:  
\[\mathrm{G}_{wo}=sigmoid\bigg{(}sum\Big{(}R_{o}\odot\overline{A_{ow}}^{T} \Big{)}\bigg{)}, \tag{22}\]  
\[\mathrm{G}_{ws}=sigmoid\bigg{(}sum\Big{(}R_{s}\odot\overline{A_{sw}}^{T} \Big{)}\bigg{)}, \tag{23}\]where \(\odot\) denotes the element-wise product, \(G_{uv}\in\mathbb{R}^{1\times k}\) is the text-object region matching weights in the visual aspect, and \(G_{uv}\in\mathbb{R}^{1\times 1}\) is the text-scene matching weights in the visual aspect. We can obtain the fine-grained fusion features by the gate weights and fused modality-based fusion features. To preserve the modality-based information which is not intensively fused, we further integrate the fine-grained visual fusion feature with original features to obtain the object wise fine-grained text-referred fusion O\({}_{wo}\) and scene-wise fine-grained text-referred fusion O\({}_{ws}\).  
\[\text{O}_{wo}=MLP\Big{(}\text{G}_{wo}\odot concat\Big{(}R_{o},\overline{A_{wo}}^ {T}\Big{)}\Big{)}+R_{o}, \tag{24}\]  
\[\text{O}_{ws}=MLP\Big{(}\text{G}_{ws}\odot concat\Big{(}R_{s},\overline{A_{sw} }^{T}\Big{)}\Big{)}+R_{s}. \tag{25}\]  
_MLP_ is a two-layer perceptron operation.  
Symmetrically, we can calculate the weight map, and textual-wise fine-grained object-referred features O\({}_{ow}\) and textual-wise fine-grained scene-referred features O\({}_{sw}\):  
\[\text{G}_{ww}=\text{sigmoid}(sum(\overline{A_{o}}^{T}\odot R_{w})), \tag{26}\]  
\[\text{G}_{sw}=\text{sigmoid}\Big{(}sum\Big{(}\overline{A_{s}}^{T}\odot R_{w} \Big{)}\Big{)}, \tag{27}\]  
\[\text{O}_{ow}=MLP\Big{(}\text{G}_{ow}\odot concat\Big{(}R_{w},\overline{A_{o }}^{T}\Big{)}\Big{)}+R_{w}, \tag{28}\]  
\[\text{O}_{sw}=MLP\Big{(}\text{G}_{sw}\odot concat\Big{(}R_{w},\overline{A_{s} }^{T}\Big{)}\Big{)}+R_{w}, \tag{29}\]  
where G\({}_{ow}\in\mathbb{R}^{1\times c},\ G_{sw}\in\mathbb{R}^{1\times c}\). We finally aggregate different fine-grained fusion features in spatial dimension based on the modality and obtain visual-wise local fusion features \(F_{IL}\) and textual-wise local fusion features F\({}_{TL}\):  
\[F_{IL}=concat(O_{wo},\ O_{ws}), \tag{30}\]  
\[F_{TL}=concat(O_{ow},\ O_{sw}). \tag{31}\]