# Lexicon-Enhanced Attention Network Based on Text Representation for Sentiment Classification
## 3 Methodology
### Word Embedding Module Based on Lexicon-enhanced Attention Mechanism  
Above all, we do some preprocessing on the original input to improve the quality of the raw text, mainly including deleting the stopwords and segmentation. First, we remove the stopwords (e.g., 'of', 'in', and 'from') from the raw input sequence. Second, we split each sequence text into words by using spaces and punctuation (e.g., ',',' and '?'). Then, we input the processed data into the first layer of our LAN model. The first layer of our LAN model is the word embedding layer. Let \(L\in R^{|V|\times d}\) be an embedding lookup table initialized with the pre-trained GloVe embedding [28], where \(|V|\) is the size of the word vocabulary and \(d\) is the dimension of word vectors. Given the input sentence \(S~{}=~{}[s_{1},s_{2},\cdots,s_{n}]\) where \(n\) is the number of words in the sentence, we can retrieve the word vectors from \(L\) to get the context matrix \(W^{C}~{}=~{}[w_{1},w_{2},\cdots,w_{n}]\in R^{n\times d}\) where \(w_{i}\) is the word vector of the \(i^{th}\) word with \(d\) dimension. In this way, we can encode each raw word in the sentence into a real-value vector in the high-dimensional space. As we all know, the word embedding module is the basis of the model in NLP task, which plays an important role in the ability of text representation. Therefore, in order to further improve the quality of the word embedding module, we construct the novel lexicon-enhanced attention mechanism to help select the crucial sentiment-resource-relevant context words for aggregating sentiment-informative text representation. Specifically, we extract positive-sentiment words and negative-sentiment words respectively from the sentiment lexicon as sentiment resource words. And the sentiment lexicon is made up of SentiWordNet [40] and the extended version of Affective Norms of English Words [41]. Then, our attention mechanism leverages these two types of sentiment resource words extracted from the sentiment lexicon as an attention source attending to the context words to explicitly learn the sentiment-aware word embedding module. In the following, we take the positive-sentiment resource words as an example to elaborate the lexicon-enhanced attention mechanism in detail. First, similar to the general matrix of word vectors, the positive-sentiment resource sequence can retrieve the word vectors from \(L\) and form the matrix \(W^{P}~{}=~{}\left[w^{P}_{1},w^{P}_{2},\cdots,w^{P}_{m}\right]\in R^{m\times d}\) where \(m\) is the number of positive-sentiment resource words in the sentiment lexicon. Second, in order to exploit the word-level relationship between the positive-sentiment resource words and the context words, we define the dot product operation among the positive-sentiment resource words and the context words as a correlation matrix. Formally, the specific formulation is described as follows:  
\[M^{P}~{}=~{}W^{C}\cdot\left(W^{P}\right)^{T} \tag{1}\]  
Figure 1: The architecture of the proposed lexicon-enhanced attention network (LAN) model. where \(\cdot\) stands for the dot product operation and \(M^{P}\in R^{n\times m}\) is the correlation matrix to measure the relevance degree between the positive-sentiment resource words and the context words. Third, based on the correlation matrix, we can calculate the sentiment-word-relevant context word representation matrix \(X^{C}_{P}\) with the dot product operation between the correlation matrix \(M^{P}\) and the positive-sentiment resource words \(W^{P}\). Mathematically, the detailed formula is presented as follows:  
\[X^{C}_{P}\ =\ \left(M^{P}\cdot W^{P}\right)^{T} \tag{2}\]  
where \(X^{C}_{P}\in R^{d\times m}\) is the context word representation matrix related to the positive-sentiment resource words.