# Examining Attention Mechanisms in Deep Learning Models for Sentiment Analysis
## 2 Related Works
The authors show two ways to consider aspect information during attention: (1) to concatenate the aspect vector into the sentence hidden representations for computing attention weights (2) to additionally append the aspect vector into the input word vectors. ATAE-LSTM achieves state-of-the-art performance on a SemEval dataset. [PERSON] et al. [34] propose a BiLSTM with an attention mechanism to focus on the information output from the hidden layers, combined with a convolutional layer that extracts higher-level phrase representations. [PERSON] et al. [35] extend the traditional LSTM-based approach for sentiment analysis by using a lexicon. Overall, the lexicon enhances the ability of word embeddings to represent words, while an attention mechanism is also introduced, which utilizes the global information of the entire text instead of having a specific target. [PERSON] et al. [36] propose a deep memory network for document-level sentiment classification. The proposed model can capture both the user and product information at the same time. A memory network consists of several inference components combined with a large long-term memory while also utilizing the memory as a knowledge base. The architecture of the model consists of two parts. First, an LSTM is used to represent each document, then a deep memory network consisting of multiple layers (hops) is used to predict the ratings for each document, and each layer is a content-based attention model. [PERSON] et al. [37] propose an adversarial memory network (AMN) for cross-domain sentiment analysis. In domain adaptation tasks, the term \"pivot\" is used, which operates similarly to discriminative learning. The end-to-end AMN can automatically capture the pivots using an attention mechanism. The proposed approach is evaluated against various techniques, achieving a 4.36% better accuracy. [PERSON] et al. [38] propose an end-to-end memory network for an aspect-level sentiment classification task. It can capture the importance of each context word when inferring the sentiment polarity of an aspect. An attention mechanism is employed with external memory to capture the importance of each context word concerning the given target aspect. Multiple computational layers are applied to the input text representation, each of which is a neural attention model with external memory, while the entire model can be trained end-to-end with gradient descent. [PERSON] et al. [39] found that the keywords which express the sentiment for the aspect word to a maximum degree are always close to the aspect word itself. To this end, two parameters are designed for both sides according to their respective importance, which is also to be learned through training. Directly inputting the word vector into the model and generating the representations of the sentence vector will contain information that is not related to the aspect word. Therefore, they use the location weight to emphasize the related sentiment information towards aspect words in the process of generating sentence vectors. Given the fact that location information is beneficial to obtain a better result, input word embedding vectors are weighted by location weights. Since they expect the weights on words far away from the aspect word to fall faster, which helps to prevent the interference of information that is not related to aspects, they use the Laplacian probability density function to obtain the location weights. The authors report quite interesting results and indicate an accuracy of 0.8035 in the SemEval 2015 task 4 dataset. [PERSON] et al. [34] propose a deep neural network architecture for a text classification task. The proposed model is named attention-based bidirectional long short-term memory with convolution layer (AC-BiLSTM). In particular, the first layer operates as a phrase representation based on word embeddings, followed by a BiLSTM that accesses context representations alongside an attention mechanism. AC-BiLSTM is evaluated on sentiment analysis datasets against state-of-the-art methods, outperforming them.