# Examining Attention Mechanisms in Deep Learning Models for Sentiment Analysis
###### Abstract  
Attention-based methods for deep neural networks constitute a technique that has attracted increased interest in recent years. Attention mechanisms can focus on important parts of a sequence and, as a result, enhance the performance of neural networks in a variety of tasks, including sentiment analysis, emotion recognition, machine translation and speech recognition. In this work, we study attention-based models built on recurrent neural networks (RNNs) and examine their performance in various contexts of sentiment analysis. Self-attention, global-attention and hierarchical-attention methods are examined under various deep neural models, training methods and hyperparameters. Even though attention mechanisms are a powerful recent concept in the field of deep learning, their exact effectiveness in sentiment analysis is yet to be thoroughly assessed. A comparative analysis is performed in a text sentiment classification task where baseline models are compared with and without the use of attention for every experiment. The experimental study additionally examines the proposed models' ability in recognizing opinions and emotions in movie reviews. The results indicate that attention-based models lead to great improvements in the performance of deep neural models showcasing up to a 3.5% improvement in their accuracy. attention mechanism; deep neural networks; global-attention; hierarchical-attention; self-attention; sentiment analysis +
Footnote †: journal:  
Attention-based models have been used in various topics, including machine translation, speech recognition, and sentiment classification [23; 24]. The field of Sentiment analysis, in particular, has been met with increased interest by the research community in recent years due to the rise of social media and the Internet. The vast amount of opinion-heavy user-generated content available to us, whether that is a product review or an opinion on an event, shows that effective sentiment analysis is needed. In particular, sentiment analysis is the computational study of people's opinions, emotions and attitudes towards entities, such as products, services and events [25]; it is highly sought after by businesses and service providers. However, automatic knowledge extraction about people's opinions and emotional states can be a very challenging task [26]. As a result, achieving performance gains, even on a small scale is of vital importance in natural language processing and sentiment analysis [27]. Overall, in this work we present various attention-based models and study their performance under different training methods and topologies for sentiment analysis. Specifically, we present attention-based models based on recurrent neural networks (RNNs) and examine their functionality in various contexts and data. A comparative analysis is performed in a text classification task where for every experiment, baseline neural models are compared with and without the use of an attention mechanism. The experimental study additionally examines the performance of the proposed methods in recognizing opinions and emotions in textual data, utilizing widely used benchmark datasets. The results indicate that attention-based models lead to si gains in performance of up to 3.5% in terms of accuracy. It is worth noting that attention-based models can weigh the importance of certain words, an ability that is quite limited in conventional neural network models. The main focus and contribution of this work is to present and implement self-attention, global-attention and hierarchical-attention-based deep neural networks, that can recognize sentiment in text and then thoroughly examine their performance in various contexts and under different baseline models, training hyperparameters and architectures. The rest of this paper is structured as follows: Section 2 reviews the related work on attention-based methods and sentiment analysis. Sections 3 and 4 introduce the theory and the mathematical background behind the aforementioned field.