# Global Local Fusion Neural Network for Multimodal Sentiment Analysis
## 1 Introduction  
The boom in mobile internet and smartphone access has made social networking an integral part of our daily lives. More and more people share their views and feelings through Twitter, Reddit, Weibo, and other social platforms, generating a large amount of social data. With the popularity of camera and video technology, data on these platforms have developed from a single text form to a combination of different media forms such as text, audio, and images. Massive multimodal data are formed in the form of a combination of text and images. Multimodal sentiment analysis aims to identify users' sentiment polarities, as well as their attitudes towards topics or events, from different forms of data. As the core field of social media analysis, sentiment analysis has received not only extensive attention from academia [1, 2] but also has broad commercial application prospects, such as personalized advertising [3], opinion mining [4], and decision making [5], etc. Compared with single-mode data, multimodal data contains more information and can easily express the users' real feelings. However, multimodal sentiment analysis is a challenging task. Different modal data have the possibility of containing different sentiment information, and the underlying features of these modalities have different dimensions and attributes. The traditional sentimental methods for multimodal sentiment analysis are based on handcrafted features [6] in each modality, and the performance highly relies on the quality of feature selection. Since the handcrafted features are usually designed with limited human knowledge, it is difficult to comprehensively describe the highly abstract features of sentiment to retain redundant emotional information and affect modelperformance. With the development of machine learning and deep learning, researchers have proposed new sentiment analysis methods for unimodal and multimodal data, which have achieved promising results. Compared with handcraft features, convolutional neural networks [7; 8], recurrent neural networks, and their variants [9] have been widely used for feature extraction, which can extract high-level features of each modality efficiently. In addition, benefitting from the introduction of large-scale datasets [10; 11; 12], many studies have begun using pre-trained models to analyze multimodal data better. Furthermore, deep-learning-based fusion methods have been proposed to explore the correlation among multimodal data [13; 14; 15]. The research about multimodal sentiment analysis is generally separated into two categories based on the dataset: video and image-text content. Some researchers leverage the effective methods [16; 17; 18; 19] to understand the emotion of user-generated videos [20; 21], which contain linguistic, visual, and acoustic information. Since the multimodal information parsed from the video is highly correlated, most approaches are centered around developing sophisticated fusion mechanisms to reduce the modality gaps between heterogeneous modalities. Other studies focus on the image-text data in social media. Visual and textual information have beneficial complementary properties in conveying information. This paper focuses on the text-image data from social media. Semantics in multimedia research remains challenging because of the complex relationship between text and images [22; 23; 24; 25]. Following a general approach [24; 25], the relationship between image and text is roughly divided into two parts: (1) Texts in multimedia mainly focus on the image content; the visual and textual information can be related based on the depicted or mentioned content. (2) Texts describe the feelings or evaluation of users; the image is semantically related to its corresponding text. Intuitively, when the image content and the text have overlapping parts, people tend to focus on the objects or regions mentioned by the text in the image, thus forming visual and textual context information.