Query:  multimodal Sentiment Analysis

title: Multimodal Sentiment Analysis of #MeToo Tweets using Focal Loss (Grand Challenge)
abstract: The #MeToo trend has led to people talking about personal experiences of harassment more openly. This work attempts to aggregate such experiences of sexual abuse to facilitate a better understanding of social media constructs and to bring about social change  We propose an approach to multimodal sentiment analysis using deep neural networks combining visual analysis and natural language processing. Our goal is different than the standard sentiment analysis goal of predicting whether a sentence expresses positive or negative sentiment; instead we try to detect the stand of a person on the topic and deduce the emotions conveyed. We have made use of a Multimodal Bi Transformer (MMBT) model  which combines both image and text features to produce an optimal prediction of a tweet's stand and sentiments on the #MeToo campaign.

title: Multi Tensor Fusion Network with Hybrid Attention for Multimodal Sentiment Analysis
abstract: Multimodal sentiment analysis is a highly sought after topic in natural language processing. In this paper, a multi tensor fusion network with hybrid attention architecture for multimodal sentiment analysis is proposed. Firstly, Bi LSTM is applied to encode contextual representation in different modalities. Following this, modalities features are extracted and interacted with by the hybrid attention mechanism. Finally, multi tensor fusion approach is used to further enhance the effectiveness of fusing interaction features in different modalities. The proposed approach outperforms the existing advanced approaches on two benchmarks through a series of regression experiments for sentiment intensity prediction, as it improves F1 score by 3.4% and 2.1% points respectively. Our architecture would be open sourced on Github1 for researchers to use.

title: Deep Learning approach for text, image, and GIF multimodal sentiment analysis
abstract: Recently, social media users are increasingly using visual media like GIFs, videos and images as powerful means of expressing their emotions and sentiments. We created a multimodal sentiment analysis tool for different forms of tweets in Python to compute the sentiment score not only base on the tweet's text, but also to consider GIFs and images to gain better accuracy for the tweet's overall sentiment score. For image sentiment, we use fine tuned CNN, for texts we use VADER, and for the GIFs we apply both image sentiment and facial expression analysis in every frame of the file. In our work, we demonstrate that by utilizing both textual and image features we can achieve better results compared to other models that only rely on either images or text features. The final sentiment score for the incoming tweets will be calculated by aggregating the output scores obtained from each of our text, image, and GIF modules.

title: Various syncretic co attention network for multimodal sentiment analysis
abstract: The multimedia contents shared on social network reveal public sentimental attitudes toward specific events. Therefore, it is necessary to conduct sentiment analysis automatically on abundant multimedia data posted by the public for real world applications. However, approaches to single modal sentiment analysis neglect the internal connections between textual and visual contents, and current multimodal methods fail to exploit the multilevel semantic relations of heterogeneous features. In this article, the various syncretic co attention network is proposed to excavate the intricate multilevel corresponding relations between multimodal data, and combine the unique information of each modality for integrated complementary sentiment classification. Specifically, a multilevel co attention module is constructed to explore localized correspondences between each image region and each text word, and holistic correspondences between global visual information and context based textual semantics. Then, all the single modal features can be fused from different levels, respectively. Except for fused multimodal features, our proposed VSCN also considers unique information of each modality simultaneously and integrates them into an end to end framework for sentiment analysis. The superior results of experiments on three constructed real world datasets and a benchmark dataset of Visual Sentiment Ontology (VSO) prove the effectiveness of our proposed VSCN. Especially qualitative analyses are given for deep explaining of our method.

title: SFNN: Semantic Features Fusion Neural Network for Multimodal Sentiment Analysis
abstract: Detecting sentiment in online reviews is a key task, and effective analysis of sentiment in online reviews is the foundation of applications such as user preference modeling, consumer behavior monitoring, and public opinion analysis. In previous studies, the sentiment analysis task mainly relied on text content and ignored the effective modeling of visual information in comments. This paper proposes a neural network SFNN based on semantic feature fusion. The model first uses convolutional neural networks and attention mechanism to obtain the effective emotional feature expressions of the image, and then maps the emotional feature expressions to the semantic feature level. Then, the semantic features of the visual modal is combined with the semantic features of the text modal, and finally the emotional polarity of the comment is effectively analyzed by combining the emotional features of the physical level of the image. Feature fusion based on semantic level can reduce the difference of heterogeneous data. Experimental results show that our model could achieve better performance than the existing methods in the benchmark dataset.

title: Towards Multimodal Sentiment Analysis Inspired by the Quantum Theoretical Framework
abstract: In this paper, we report on a quantum theoretical frame work that constructs a multimodal representation for the input video sequence and infers the users' sentiments regarding a textual query based topic. Furthermore, we report on the experimental results of the use of our quantum theoretical framework also in comparison with the state of the art methods. Finally, we describe the design and the implementation of a prototype which incorporates the proposed methods.

title: Multimodal Sentiment Analysis using Hierarchical Fusion with Context Modeling
abstract: Multimodal sentiment analysis is a very actively growing field of research. A promising area of opportunity in this field is to improve the multimodal fusion mechanism. We present a novel feature fusion strategy that proceeds in a hierarchical fashion, first fusing the modalities two in two and only then fusing all three modalities. On multimodal sentiment analysis of individual utterances, our strategy outperforms conventional concatenation of features by 1% which amounts to 5% reduction in error rate. On utterance level multimodal sentiment analysis of multi utterance video clips, for which current state of the art techniques incorporate contextual information from other utterances of the same clip, our hierarchical fusion gives up to 2.4% (almost 10% error rate reduction) over currently used concatenation. The implementation of our method is publicly available in the form of open source code.

title: VistaNet: Visual Aspect Attention Network for Multimodal Sentiment Analysis
abstract: Detecting the sentiment expressed by a document is a key task for many applications, e.g. modeling user preferences, monitoring consumer behaviors, assessing product quality. Traditionally, the sentiment analysis task primarily relies on textual content. Fueled by the rise of mobile phones that are often the only cameras on hand, documents on the Web (e.g. reviews, blog posts, tweets) are increasingly multimodal in nature, with photos in addition to textual content. A question arises whether the visual component could be useful for sentiment analysis as well. In this work, we propose Visual Aspect Attention Network or VistaNet, leveraging both textual and visual components. We observe that in many cases, with respect to sentiment detection, images play a supporting role to text, highlighting the salient aspects of an entity, rather than expressing sentiments independently of the text. Therefore, instead of using visual information as features, VistaNet relies on visual information as alignment for pointing out the important sentences of a document using attention. Experiments on restaurant reviews showcase the effectiveness of visual aspect attention, vis a vis visual features or textual attention.

title: Multi level context extraction and attention based contextual inter modal fusion for multimodal sentiment analysis and emotion classification
abstract: The recent advancements in the Internet technology and its associated services, led the users to post a large amount of multimodal data into social media Web sites, online shopping portals, video repositories, etc. The availability of the huge amount of multimodal content, multimodal sentiment classification, and affective computing has become the most researched topic. The extraction of context among the neighboring utterances and considering the importance of inter modal utterances before multimodal fusion are the most important research issues in this field. This article presents a novel approach to extract the context at multiple levels and to understand the importance of inter modal utterances in sentiment and emotion classification. Experiments are conducted on two publically accepted datasets such as CMU MOSI for sentiment analysis and IEMOCAP for emotion classification. By incorporating the utterance level contextual information and importance of inter modal utterances, the proposed model outperforms the standard baselines by over 3% in classification accuracy.

title: A survey of multimodal sentiment analysis
abstract: Sentiment analysis aims to automatically uncover the underlying attitude that we hold towards an entity. The aggregation of these sentiment over a population represents opinion polling and has numerous applications. Current text based sentiment analysis rely on the construction of dictionaries and machine learning models that learn sentiment from large text corpora. Sentiment analysis from text is currently widely used for customer satisfaction assessment and brand perception analysis, among others. With the proliferation of social media, multimodal sentiment analysis is set to bring new opportunities with the arrival of complementary data streams for improving and going beyond text based sentiment analysis. Since sentiment can be detected through affective traces it leaves, such as facial and vocal displays, multimodal sentiment analysis offers promising avenues for analyzing facial and vocal expressions in addition to the transcript or textual content. These approaches leverage emotion recognition and context inference to determine the underlying polarity and scope of an individual's sentiment. In this survey, we define sentiment and the problem of multimodal sentiment analysis and review recent developments in multimodal sentiment analysis in different domains, including spoken reviews, images, video blogs, human machine and human human interactions. Challenges and opportunities of this emerging field are also discussed leading to our thesis that multimodal sentiment analysis holds a significant untapped potential.