# Examining Attention Mechanisms in Deep Learning Models for Sentiment Analysis
## 3 Deep Recurrent Neural Networks  
As a base for attention-based deep neural networks, we utilize recurrent neural networks (RNNs). A recurrent neural network is an extension of the conventional feed-forward neural network. Long short-term memory (LSTM) models, which were introduced by [PERSON] et al. [2], are also based on the RNN architecture. They overcome RNN's shortcomings when it comes to gradients and also improve the learning ability for long-time sequence data. The differences are that instead of having a single neural network layer, there are four layers interacting in a specific way. The LSTM unit is composed of a cell, an input gate, an output gate, and a forget gate. The memory cell that is introduced can preserve state over long periods of time, while the three gates regulate the flow of information into and out of the cell:  
\[X=[h_{t-1},x_{t}] \tag{1}\]  
\[f_{t}=\sigma\Big{(}W_{f}\cdot X+b_{f}\Big{)} \tag{2}\]  
\[i_{t}=\sigma(W_{i}\cdot X+b_{i}) \tag{3}\]  
\[o_{t}=\sigma(W_{o}\cdot X+b_{o}) \tag{4}\]  
\[c_{t}=f_{t}\odot c_{t-1}+i_{t}\odot tanh(W_{c}\cdot X+b_{c}) \tag{5}\]\[h_{t}=o_{t}\odot tanh(c_{t}) \tag{6}\]  
where \(W_{t},W_{f},W_{o}\in\mathbb{R}^{d\times 2d}\) are the weighted matrices, \(b_{i},b_{f},b_{o}\in\mathbb{R}^{d}\) are the biases of LSTM to be learned during training, \(\sigma\) is the sigmoid function, \(x_{t}\) includes the inputs representing the word embedding vectors, and \(h_{t}\) is the vector of the hidden layer. Overall \(o_{t}\) decides the output, using a sigmoid function on the \"output gate\". Gated recurrent units (GRUs) are a variation of LSTMs introduced by [PERSON] et al. [3]. They include a gating mechanism and combine the \"forget\" and \"input\" gates into a signal update gate as well as performing some other changes, resulting in a model simpler than LSTMs:  
\[z_{t}=\sigma(W_{z}\cdot X+b_{z}) \tag{7}\]  
\[r_{t}=\sigma(W_{r}\cdot X+b_{r}) \tag{8}\]  
\[\hat{h}_{t}=tanh(W_{h}\cdot(r_{t}\odot X)) \tag{9}\]  
\[h_{t}=(1-z_{t})\odot h_{t-1}+z_{t}\odot\hat{h}_{t} \tag{10}\]  
where \(z_{t}\) is the update gate vector, \(r_{t}\) is the reset gate vector and \(h_{t}\) is the candidate activation vector. The main difference between a recurrent neural network and a deep recurrent neural network is that in the latter, we stack multiple layers of individual recurrent networks. The idea for this implementation came from the problem that, while RNNs have depth, this notion of depth is unlikely to involve a hierarchical interpretation of the data. By applying the same computation recursively to compute the contribution of children to their parents and the same computation to produce an output response, every node/phrase is being represented in the same space. This technique is different than conventionally stacking deep learners, where an important benefit of depth is the hierarchy among hidden representations: every hidden layer conceptually lies in a different representation space and potentially is a more abstract representation of the input than the previous layer [19]. In the context of this work, we utilize two different representations for the input text, which are the one-hot encoding and the word2 vec embeddings, and also examine their impact on attention-based methods. Word embeddings try to capture the characteristics of the neighbors of the target and, as a result, capture the similarity between words. These vector representations of words map a word into a common space preserving the word meaning, their relationships and the semantic information. The word2 vec method [41] that was utilized in this work implements both the continuous bag-of-words (CBOW) and the skip-gram models to learn word embeddings. As mentioned above, word2 vec is a fixed dimensional real-valued vector representation that transforms an atomic word into a positional representation of the word relative to other words in the dataset. We also examine the impact of the dropout regularization technique [42].