# Examining Attention Mechanisms in Deep Learning Models for Sentiment Analysis
## 4 Attention-Based Methods  
Although RNNs have been a great success, they have certain limitations [44]. Among them, they are not efficient at memorizing long or distant sequences [45]. RNNs process tokens sequentially, maintaining a state vector representing the data seen after every token. The information from a token can propagate arbitrarily down the sequence by continuously encoding information. However, the main issue that arises here is that due in part to the vanishing gradient problem [44], the model's state at the end of a long sentence often does not contain information about early tokens and as a result, the mechanism does not work as intended. The aforementioned problem was addressed by the introduction of attention mechanisms. Attention mechanisms let a model directly look at and draw from the state of an earlier point in the sentence. The attention layer can access all previous states and weigh them according to some learned measure of relevancy to the current token, providing sharper information about far-away relevant tokens. A clear example of the utility of attention is in neural machine translation [23]. In an English-to-French translation system, the first word of the French output most probably depends heavily on the beginning of the English input. Though, in a classic LSTM model, to produce the first word of the French output, the model is only given the state vector of the last English word. In theory, this vector can encode information about the whole English sentence, but as mentioned before, this information is often not preserved in practice. If an attention mechanism is introduced, the model can instead learn to attend to the states of early English tokens when producing the beginning of the French output, giving it a much better concept of what it is translating. Let \(H\in\mathbb{R}\ ^{d\times N}\) be a matrix consisting of hidden vectors \([h_{1},\ldots,h_{N}]\) that the LSTM produced, where \(d\) is the size of hidden layers. Furthermore, \(\upsilon_{a}\) represents the embedding. The attention mechanism will produce an attention weight vector \(\alpha\) and a weighted hidden representation \(r\):  
\[M=tanh\bigg{(}\left[ \begin{array}{c}W_{h}H\\ W_{\upsilon}\upsilon_{a}\otimes e_{N}\end{array} \right]\bigg{)} \tag{11}\]  
\[a=softmax\Big{(}W^{T}M\Big{)} \tag{12}\]  
\[r=Ha^{T} \tag{13}\]  
where \(W\) are the weights to be learned during training. The attention mechanism allows the model to capture the most important parts of a document. Using a hyperbolic tangent on the representation would lead to a more complex feature representation \(h^{*}\in\mathbb{R}\ ^{d}\) that would be more fitting for other tasks. Finally, we can get the sentiment distribution of each text by adding a softmax layer in each final text representation \(r\):  
\[y=softmax(w\cdot r+b) \tag{14}\]  
This type of attention is also known as global (soft) attention [23]. As we can see in Figure 2, at each time step \(t\), a global attentional model infers a variable-length alignment weight vector \(\alpha_{t}\) based on the current target state \(h_{t}\) and all source states \(\overline{h}_{s}\)[46]. A global context vector \(c_{t}\) is then computed as the weighted average, according to \(\alpha_{t}\), over all the source state. The idea is to derive a context vector based on all hidden states of the encoder RNN. Hence, it is said that this type of attention attends to the entire input state space. An alternative attention mechanism is self-attention [22]. The main goal is to overcome the drawback of RNNs by allowing the attention mechanism to focus on segments of the sentence, where the relevance of the segment is determined by the contribution to the task. Different positions of the same hidden state space derived from the input sequence are related to different positions, based on the argument that multiple components together form the overall semantics of a sequence. This approach brings together differently positioned information through multiple hops' attention.