# Examining Attention Mechanisms in Deep Learning Models for Sentiment Analysis
## 2 Related Works  
Attention-based methods have found many applications in various natural language processing tasks, including sentiment analysis, opinion mining as well as in recognition of emotions in text. [PERSON] et al. [23] present an encoder-decoder approach with an attention mechanism that is integrated into the decoder by repeatedly reading the representation of a source sentence, which remains fixed after being generated by the encoder. Thus, the model can search for parts that are relevant to predicting a target word and, as a result, achieved state-of-the-art performance. [PERSON] et al. [28] propose a middle ground between two popular deep neural network architectures, recurrent and recursive neural networks. The approach named neural tree indexer (NTI) is a syntactic parsing-independent tree-structured model that learns a premise and a hypothesis representation and then combines them with an attention mechanism, achieving a new state-of-the-art accuracy on inference and classification tasks. [PERSON] et al. [29] propose a model named hierarchical attention network (HAN) for a document classification task that mirrors the hierarchical structure of documents. Furthermore, since words and sentences vary in terms of informativeness, it has two levels of attention  
Figure 1: Illustration of word weighting of past memory using an attention mechanism [22]. mechanisms applied at the word and sentence level. As a result, it can differentiate its attention to more and less important content when constructing the document representation while also using context to discover when a sequence is relevant. The encoder used is a GRU, and the proposed approach outperforms previous methods on six datasets. [PERSON] et al. [30] expand upon the previous model and propose a hierarchical iterative attention model for an aspect-based sentiment analysis task. The aspect-based task is formulated as a machine comprehension problem. The model achieves greater performance than the hierarchical attention network baseline. [PERSON] et al. [31] propose a method for identifying keywords discriminating positive and negative sentences by using a weakly supervised learning method based on a CNN. The CNN model is trained on sentence matrices, and after the training, a word attention mechanism is employed that identifies high-contributing words with a class activation map (CAM), using the weights from the fully connected layer at the end of the CNN. The main advantage is that it produces both sentence-level polarity scores and word-level polarity scores by only using weak labels, i.e., the polarity of the sentence from the dataset. [PERSON] et al. [32] propose a model for extracting sentence embedding by introducing self-attention. In this new sentence embedding technique, each embedding corresponds to a 2-D matrix, with each row of the matrix attending on a different part of the sentence. First, the sentence is run through an RNN. Then multiple attention values are learned for each RNN state, and then each attention vector focuses on different parts of the sentence because a penalization term is added. It achieves high accuracy on both sentiment classification and textual entailment tasks. [PERSON] et al. [33] propose an LSTM model, which, unlike existing methods, incorporates global user preference and product characteristics on a sentiment classification task. The model is a hierarchical LSTM, and it generates sentence and document representations. Then, an attention mechanism is used on the user and product information, thus taking into consideration information at both the word level and the semantic level. [PERSON] et al. [24] propose an attention-based LSTM method with target embedding for an aspect-level sentiment classification task. The attention mechanism forces the neural model to attend to the important part of a sentence when different aspects are taken as input.