# Lexicon-Enhanced Attention Network Based on Text Representation for Sentiment Classification
## 3 Methodology
### Deep Neural Network Module Based on Multi-head Attention Mechanism  
Although the word embedding module based on lexicon-enhanced attention mechanism can solve the problem of word-level semantic confusion to some extent with the help of the interactive learning of attention mechanism, the generated text representation is insufficient for text sentiment analysis. Hence, we introduce a multi-head attention mechanism into the deep neural network to capture the important contextual information from different representation subspaces at different positions. And the network structure is mainly composed of the bidirectional LSTM network and the multi-head attention mechanism as shown in Figure 1.  
We use a bidirectional version of LSTM network to process the output sequence \(X=[x_{1},x_{2},\cdots,x_{n}]\) generated by the sentiment-aware word embedding module, which can capture information from both the past context and the future context. The forward \(\overrightarrow{LSTM}\) layer reads the sentence \(X\) from \(x_{1}\) to \(x_{n}\) and the backward \(\overleftarrow{LSTM}\) layer processes the sentence in reverse sequence order. The final hidden state \(h_{t}\) is the composition of the forward hidden state \(\overrightarrow{h_{t}}\) and the backward hidden state \(\overleftarrow{h_{t}}\) as follows:  
\[\overleftarrow{h_{t}}\ =\ \overleftarrow{LSTM}\left(x_{t},\ \overleftarrow{h_{t-1}}\right) \tag{10}\]  
\[\overrightarrow{h_{t}}\ =\ \overrightarrow{LSTM}\left(x_{t},\ \overrightarrow{h_{t-1}}\right) \tag{11}\]  
\[h_{t}\ =\ \left[\overleftarrow{h_{t}}\oplus\overrightarrow{h_{t}}\right] \tag{12}\]  
where \(\oplus\) is the element-wise sum operation. The final output of the bidirectional LSTM network is \(H=[h_{1},h_{2},\cdots,h_{n}]\) where \(n\) denotes the length of the original input sequence. Taking the forward \(\overleftarrow{LSTM}\) as an example, the detailed processing of each \(\overrightarrow{LSTM}\) unit is described as follows:  
For each word token \(x_{t}\) in the output sequence of the sentiment-aware word embedding module, the forward \(\overrightarrow{LSTM}\) network can generate a corresponding hidden state vector \(\overrightarrow{h_{t}}\) by an adaptive gating mechanism that takes control of the data flow from the past to the future. This adaptive gating mechanism consists of a forget gate \(f_{t}\) which controls how long certain information are stored in the current unit, an input gate \(i_{t}\) which controls the input degree of new information into the current unit, and an output gate \(o_{t}\) which controls how much the information held in the current unit influences the output activation of the block. Mathematically, the hidden state vector \(\overrightarrow{h_{t}}\) is calculated as follows:  
\[f_{t} = \sigma\Big{(}W_{f}[h_{t-1},x_{t}]+b_{f}\Big{)} \tag{13}\] \[i_{t} = \sigma(W_{i}[h_{t-1},x_{t}]+b_{i})\] (14) \[o_{t} = \sigma(W_{o}[h_{t-1},x_{t}]+b_{o})\] (15) \[\tilde{c_{t}} = tanh(W_{c}[h_{t-1},x_{t}]+b_{c})\] (16) \[c_{t} = i_{t}\bigodot\tilde{c_{t}}+f_{t}\bigodot c_{t-1}\] (17) \[\overrightarrow{h_{t}} = o_{t}\bigodot tanh(c_{t}) \tag{18}\]  
where \(\sigma\) is the sigmoid function, \(\bigodot\) denotes the element-wise multiplication, \(W_{f}\), \(W_{i}\), \(W_{o}\), and \(W_{c}\) are the learnable weight matrixes for their corresponding gated units and the intermediate cell, respectively, while \(b_{f}\), \(b_{i}\), \(b_{o}\), and \(b_{c}\) are their corresponding bias vectors. Following the same procedure of the forward \(\overrightarrow{LSTM}\) network, the backward \(\overrightarrow{LSTM}\) network can generate a corresponding hidden state vector \(\overleftarrow{h_{t}}\). The detailed structure of the basic LSTM unit can be seen from \[\overleftarrow{h_{t}}\ =\ \overleftarrow{LSTM}\left(x_{t},\ \overleftarrow{h_{t-1}}\right) \tag{10}\]0  
Then, we employ the multi-head attention mechanism to encode hidden states of the context sequence from different representation subspaces at different positions.