$\rho$-hot Lexicon Embedding-based Two-level LSTM for Sentiment Analysis

Abstract: Sentiment analysis is a key component in various text mining applications. Numerous sentiment classification techniques, including conventional and deep learning-based methods, have been proposed in the literature. In most existing methods, a high-quality training set is assumed to be given. Nevertheless, constructing a high-quality training set that consists of highly accurate labels is challenging in real applications. This difficulty stems from the fact that text samples usually contain complex sentiment representations, and their annotation is subjective. We address this challenge in this study by leveraging a new labeling strategy and utilizing a two-level long short-term memory network to construct a sentiment classifier. Lexical cues are useful for sentiment analysis, and they have been utilized in conventional studies. For example, polar and privative words play important roles in sentiment analysis. A new encoding strategy, that is, $\rho$-hot encoding, is proposed to alleviate the drawbacks of one-hot encoding and thus effectively incorporate useful lexical cues. We compile three Chinese data sets on the basis of our label strategy and proposed methodology. Experiments on the three data sets demonstrate that the proposed method outperforms state-of-the-art algorithms.

Text is important in many artificial intelligence applications. Among various text mining techniques, sentiment analysis is a key component in applications such as public opinion monitoring and comparative analysis. Sentiment analysis can be divided into three problems according to input texts, namely, sentence, paragraph, and document levels. This study focuses on sentence and paragraph levels.
Text sentiment analysis is usually considered a text classification problem. Almost all existing text classification techniques are applied to text sentiment analysis BIBREF0 . Typical techniques include bag-of-words (BOW)-based BIBREF1 , deep learning-based BIBREF2 , and lexicon-based (or rule-based) methods BIBREF3 .
Although many achievements have been made and sentiment analysis has been successfully used in various commercial applications, its accuracy can be further improved. The construction of a high-accuracy sentiment classification model usually entails the challenging compilation of training sets with numerous samples and sufficiently accurate labels. The reason behind this difficulty is two-fold. First, sentiment is somewhat subjective, and a sample may receive different labels from different users. Second, some texts contain complex sentiment representations, and a single label is difficult to provide. We conduct a statistical analysis of public Chinese sentiment text sets in GitHub. The results show that the average label error is larger than 10%. This error value reflects the degree of difficulty of sentiment labeling.
Privative and interrogative sentences are difficult to classify when deep learning-based methods are applied. Although lexicon-based methods can deal with particular types of privative sentences, their generalization capability is poor.
We address the above issues with a new methodology. First, we introduce a two-stage labeling strategy for sentiment texts. In the first stage, annotators are invited to label a large number of short texts with relatively pure sentiment orientations. Each sample is labeled by only one annotator. In the second stage, a relatively small number of text samples with mixed sentiment orientations are annotated, and each sample is labeled by multiple annotators. Second, we propose a two-level long short-term memory (LSTM) BIBREF4 network to achieve two-level feature representation and classify the sentiment orientations of a text sample to utilize two labeled data sets. Lastly, in the proposed two-level LSTM network, lexicon embedding is leveraged to incorporate linguistic features used in lexicon-based methods.
Three Chinese sentiment data sets are compiled to investigate the performance of the proposed methodology. The experimental results demonstrate the effectiveness of the proposed methods. Our work is new in the following aspects.
The rest of this paper is organized as follows. Section 2 briefly reviews related work. Section 3 describes our methodology. Section 4 reports the experimental results, and Section 5 concludes the study.
Sentiment analysis aims to predict the sentiment polarity of an input text sample. Sentiment polarity can be divided into negative, neutral, and positive in many applications.
Existing sentiment classification methods can be roughly divided into two categories, namely, lexicon-based and machine learning-based methods BIBREF5 . Lexicon-based methods BIBREF6 construct polar and privative word dictionaries. A set of rules for polar and privative words is compiled to judge the sentiment orientation of a text document. This method cannot effectively predict implicit orientations. Machine learning-based methods BIBREF7 utilize a standard binary or multi-category classification approach. Different feature extraction algorithms, including BOW BIBREF8 and part of speech (POS) BIBREF7 , are used. Word embedding and deep neural networks have recently been applied to sentiment analysis, and promising results have been obtained BIBREF9 BIBREF10 .
Lexicon-based methods are actually in implemented in an unsupervised manner. They infer the sentiment categories of input texts on the basis of polar and privative words. The primary advantage of these methods is that they do not require labeled training data. The key of lexicon-based methods is the lexical resource construction, which maps words into a category (positive, negative, neutral, or privative). Senti-WordNet BIBREF11 is a lexical resource for English text sentiment classification. For Chinese texts, Senti-HowNet is usually used.
Fig. 1 characterizes a typical lexicon-based sentiment classification approach. The approach iteratively checks each word in an input sentence from left to right. The weight score of each word is calculated according to the procedure shown in Fig. 1. The final sentiment score is the average score of the words with weight scores. The scores of positive, neutral, and negative sentiments are denoted as “+1",“0", and “-1", respectively. According to the lexicon-based algorithm shown in Fig. 1, the sentiment score of “it is not bad" is 0.25, and the sentiment score of “it is good" is 1. However, the score of “it is not so bad" is -0.75, and this score is definitely wrong. Therefore, machine learning (including feature learning) methodologies have become mainstream in sentiment analysis.
Deep learning (including word embedding BIBREF12 ) has been applied to almost all text-related applications, such as translation BIBREF13 , quality assurance BIBREF14 , recommendation BIBREF15 , and categorization BIBREF16 . Popular deep neural networks are divided into convolutional neural networks (CNNs) BIBREF17 and recurrent neural network (RNNs) BIBREF18 BIBREF19 . Both are utilized in sentiment classification BIBREF20 . Kim investigated the use of CNN in sentence sentiment classification and achieved promising results BIBREF2 . LSTM BIBREF21 , a classical type of RNN, is the most popular network used for sentiment classification. A binary-directional LSTM BIBREF22 with an attention mechanism is demonstrated to be effective in sentiment analysis.
Deep learning-based methods rarely utilize the useful resources adopted in lexicon-based methods. Qiao et al. BIBREF23 incorporated lexicon-based cues into the training of an LSTM-based model. Their proposed method relies on a new loss function that considers the relationships between polar or certain types of words (e.g., privative) and those words next to them in input texts. Our study also combines lexical cues into LSTM. Nevertheless, unlike Qiao et al.'s study that implicitly used lexical cues, the present work explicitly uses lexical cues in the LSTM network. Shin et al. BIBREF24 combined the lexicon embeddings of polar words with word embeddings for sentiment classification. The difference between our approach an the method proposed by Shin et al. the is discussed in Section 3.3.5.
Numerous studies on aspect-level sentiment analysis exist BIBREF25 . This problem is different from the sentiment classification investigated in this study.
This section first introduces our two-stage labeling procedure. A two-level LSTM is then proposed. Lexicon embedding is finally leveraged to incorporate lexical cues.
As stated earlier, sentiment is subjective, and texts usually contain mixed sentiment orientations. Therefore, texts¡¯ sentiment orientations are difficult to label. In our study, three sentiment labels, namely, positive, neutral, and negative, are used. The following sentences are taken as examples.
The service is poor. The taste is good, but the rest is not so bad.
The quality of the phone is good, but the appearance is just so-so.
In user annotation, the labels of these two sentences depend on users. If a user is concerned about service, then the label of S1 may be “negative". By contrast, for another user who does not care about service, the label may be “positive". Similarly, a user may label S2 as “positive" if he cares about quality. Another user may label it as “negative" if the conjunction “but" attracts the user¡¯s attention more. Another user may label it as “neutral" if they are concerned about quality and appearance.
The underlying reason is that sentiment is more subjective than semantics. In related research on subjective categorization, such as visual aesthetics, each sample is usually repeatedly annotated by multiple annotators, and the average label is taken as the final label of the sample. This labeling strategy can also be applied to text sentiment annotation. However, we argue that this strategy is unsuitable for a (relatively) large number of samples. The reason lies in the following two aspects.
Multiple annotators for a large number of data sets require a large budget.
In our practice, annotators claim that their judgment criteria on sentiment become fused on texts with mixed sentiment orientations (e.g., S1 and S2) over time during labeling, and they become bored accordingly.
A two-stage labeling strategy is adopted in this study. In the first stage, each sentence/paragraph is divided into several clauses according to punctuation. The sentiment of each partitioned clause is relatively easy to annotate; therefore, each clause is labeled by only one user. In the second stage, a relatively small-sized sentence/paragraph set is labeled, and each sentence is labeled by all involved annotators. We still take the two sentences, S1 and S2, as examples. S1 and S2 are split into clauses, as shown below.
S1:
S1.1: The service is poor
S1.2: The taste is good
S1.3: but the rest is not so bad.
S2:
S2.1: The quality of the phone is good
S2.2: but the appearance is just so-so.
Each of the above clauses is labeled by only one annotator. The annotation in the first stage is easy to perform; thus, the number of clauses can be larger than the number of sentences used in the second labeling stage.
Given two training data sets (denoted by T1 and T2), a new learning model should be utilized. LSTM is a widely used deep neural network in deep learning-based text classification.
LSTM is a typical RNN model for short-term memory, which can last for a long period of time. An LSTM is applicable to classify, process, and predict time series information with given time lags of unknown size. A common LSTM block is composed of a cell, an input gate, an output gate, and a forget gate. The forward computation of an LSTM block at time INLINEFORM0 or position INLINEFORM1 is as follows BIBREF21 : DISPLAYFORM0 
where INLINEFORM0 is the input vector at time INLINEFORM1 (or position INLINEFORM2 ); INLINEFORM3 and INLINEFORM4 are the input vectors of the input unit and input gate, respectively; INLINEFORM5 and INLINEFORM6 are the output and hidden vectors at time INLINEFORM7 , respectively; INLINEFORM8 is the output of the forget gate at time INLINEFORM9 ; INLINEFORM10 is the internal state of the memory cell in an LSTM block at time INLINEFORM11 ; and INLINEFORM12 is the sigmoid active function.
When LSTM is used to classify an input sentence, the hidden vectors of each input vector are summed to form a dense vector that can be considered the feature representation of the input sentence, i.e., DISPLAYFORM0 
In many applications, a bi-directional LSTM (bi-LSTM) structure is usually used, as shown in Fig. 2(a). In bi-LSTM, forward and backward information are considered for information at time INLINEFORM0 ; hence, the context is modeled. Bi-LSTM is thus significantly reasonable for text processing tasks. In our two-level LSTM, bi-LSTM is used in each level.
The output hidden state at time INLINEFORM0 of a bi-LSTM block can be described as follows: DISPLAYFORM0 
where INLINEFORM0 , INLINEFORM1 , and INLINEFORM2 are the corresponding vectors at time INLINEFORM3 in the forward LSTM block; and INLINEFORM4 , INLINEFORM5 , and INLINEFORM6 are the corresponding vectors at time INLINEFORM7 in the backward LSTM block. INLINEFORM8 . When attention is used, the dense feature vector INLINEFORM9 of an input sentence is calculated as follows: DISPLAYFORM0 
where INLINEFORM0 is the vector that consists of attention weights. The bi-LSTM with attention is shown in Fig. 2(b).
Our proposed network consists of two levels of LSTM network. In the first level, a bi-LSTM is used and learned on the basis of the first training set T1. This level is a conventional sentiment classification process. The input of this level is a clause, and the input INLINEFORM0 is the embedding of the basic unit of the input texts. The network is shown in Fig. 3(a).
In the second level, a bi-LSTM is also used and learned on the basis of the second training set T2. The input of this level is a sentence or a paragraph. The input INLINEFORM0 consists of two parts. The first part is the feature vector of the INLINEFORM1 -th clause. The feature vector is generated by the first-level network. In other words, the dense feature shown in Fig. 3(a) ( INLINEFORM2 ) is used. The second part is the sentiment score (not predicted label) output by the first-level network. The sentence S1 (The service is poor. The taste is good, but the rest is not so bad.) used in Subsection 3.1 is taken as an illustrative example. S1 contains three clauses. Therefore, the input vector of S1 can be represented by INLINEFORM3 
where DISPLAYFORM0 
where INLINEFORM0 is the output score of the INLINEFORM1 th clause by the first-level LSTM and INLINEFORM2 is the feature representation of the INLINEFORM3 th clause by the first LSTM. The network of the whole two-level network is shown in Fig. 3(b).
The proposed lexicon embedding is based on INLINEFORM0 -hot encoding. Therefore, INLINEFORM1 -hot encoding is first described.
For categorical data, one-hot encoding is the most widely used encoding strategy when different categories are independent. For example, if one-hot encoding is used to represent three categories, namely, positive, neutral, and negative, the encoding vectors for the three categories are INLINEFORM0 , INLINEFORM1 , and INLINEFORM2 , respectively.
In this work, many lexical cues are categorical data, and different categories are independent. These lexical cues can directly be represented by one-hot encoding. The encoded vectors for lexical cues are then concatenated with other vectors, such as character/word embedding. However, one-hot encoding presents two main limitations when the encoded vector is concatenated with other vectors.
The value difference between the elements of one-hot encoded vectors and those of other encoded vectors (e.g., word embedding vectors) may be large. Fig. 4 shows the histogram of the values of the elements of the word embedding vectors. The magnitude of most elements are smaller than 1.
The lengths of one-hot encoded vectors are usually shorter than those of other encoded vectors. Consequently, the proportion of one-hot encoded part is small in the concatenated vectors.
The above two limitations affect the final sentiment analysis performance. To this end, we propose a new encoding strategy. DISPLAYFORM0 
where INLINEFORM0 is the INLINEFORM1 -hot encoded vector, INLINEFORM2 is the proportion parameter, INLINEFORM3 is the one-hot encoded vector, and INLINEFORM4 is an INLINEFORM5 -dimensional vector. If INLINEFORM6 and INLINEFORM7 are equal to 1, then INLINEFORM8 -hot encoding is reduced to one-hot encoding. The parameter INLINEFORM9 is applied to increase the length of the final encoded vector.
Most lexicon-based sentiment methods rely on four types of words, namely, positive, negative, neutral, and privative. These words are useful cues for predicting the sentiment labels of input texts. The incorporation of these words should also be useful. A previous study has shown that a typical document comprises approximately 8% of such sentences BIBREF26 . Sentiments expressed in a conditional sentence can be difficult to determine due to the semantic condition. The sentiment polarities of interrogative sentences are also difficult to classify according to our empirical study.
Five types of words, namely, positive (Pos), negative (Neg), privative (Pri), suppositive (Sup), and interrogative (Int), are represented by the proposed encoding method. The rest words, which do not belong to any of the above five types, are named “others (Oth)" instead of “neutral" because some words, such as “the", are unrelated to “sentiment". The value of INLINEFORM0 in Eq. (6) is set as 10. The encoded vectors are as follows. INLINEFORM1 
In the proposed INLINEFORM0 -hot embedding, the parameter INLINEFORM1 can be learned during training. The representation of the third clause (“but the rest is not so bad") of S1 in Subsection 3.1 is taken as an illustrative example. The new embedding of each word in this clause is as follows. DISPLAYFORM0 
Certain types (e.g., positive, negative, and privative) of words should play more important roles than other words do in texts; therefore, their embeddings are also used in the attention layer. A new LSTM based on our lexicon embedding is proposed, as shown in Fig. 5. The attention layer and final dense vector of the network in Fig. 3(a) are calculated as follows. DISPLAYFORM0 
where INLINEFORM0 is the attention weight for the INLINEFORM1 -th input, lt is the lexicon embedding for key lexical words for the INLINEFORM2 -th input, and INLINEFORM3 is the final dense vector. Eq. (2) is used in the first-level LSTM.
POS is usually used as a key cue in sentiment analysis BIBREF27 . To this end, we use additional lexicon embedding. The new lexicon embedding includes several major types of POS, namely, interrogative, exclamatory, and others. This new lexicon embedding is also applied to the attention layer. The motivation lies in that certain types of POS should play important roles in sentiment.
The proposed INLINEFORM0 -hot embedding is still applied to POS types in this study. According to our initial case studies, eight POS types are considered. They are noun, adjective, verb, pronoun, adverb, preposition, accessory, and others. The eight POS types are represented by the proposed INLINEFORM1 -hot encoding. We let INLINEFORM2 in Eq. (6) be 10. The first three POS types are as follows. INLINEFORM3 
When POS embedding is used, the attention layer and final outputs of the network in Eq. (3) become DISPLAYFORM0 
where INLINEFORM0 is the lexicon embedding for key lexical words for the INLINEFORM1 -th input.
Conjunction words play important roles in sentiment analysis BIBREF28 . For example, conjunctions such as “but" and “moreover" usually indicate the focus of texts and attract readers¡¯ attention. Therefore, conjunctions are considered in the input of the second-level LSTM.
Once a set of conjunction words is compiled, INLINEFORM0 -hot embedding is used. In our experiments, the number of conjunction words is 169. Therefore, the parameter INLINEFORM1 in Eq. (2) is set as 1.
When conjunction embedding is used for the second-level layer, the attention layer and final outputs of the network in Fig. 3(b) are calculated as follows. DISPLAYFORM0 
where INLINEFORM0 is the attention weight for the INLINEFORM1 -th input clause; INLINEFORM2 is the hidden vector of the second-level LSTM; INLINEFORM3 and INLINEFORM4 are the conjunction embeddings for the first and last words in the INLINEFORM5 -th input clause, respectively; and INLINEFORM6 is the final dense vector used for the final classification.
Shin et al. BIBREF24 also embedded lexical information into sentiment analysis. Three major differences exist between our method and the method proposed by Shin et al. BIBREF24 .
The lexicon embedding proposed by Shin et al. us-es one-hot encoding, whereas the proposed method uses a new encoding strategy that can be considered a soft one-hot encoding.
The lexicon embedding proposed by Shin et al. ex-tends the length of raw encoded vectors. However, the extension aims to keep the lengths of lexical and word embeddings equal. Their extension method also only relies on zero padding and is thus different from the proposed method.
Only sentimental words are considered in the lexicon embedding proposed by Shin et al. On the contrary, sentimental words, POS, and conjunctions are considered in our work.
The algorithmic steps of the entire learning procedure for the proposed INLINEFORM0 -hot lexicon embedding-based two-level LSTM (called INLINEFORM1 Tl-LSTM) are shown in Algorithm 1. In Algorithm 1, T1 refers to the training data that consist of clauses and the labels obtained in the first-stage labeling procedure. T2 refers to the training data that consist of sentences and the labels obtained in the second-stage labeling procedure. The structure of INLINEFORM2 Tl-LSTM is presented in Fig. 6.
 INLINEFORM0 Tl-LSTM Input: Training sets T1 and T2; dictionary of key lexical words; POS for each word; dictionary of conjunction words; character/word embeddings for each character/word.
 Output: A trained two-level LSTM for sentiment classification.
 Steps:
Construct the embedding vector for each character (including punctuation) in the clauses in T1. The embeddings include the character/word and lexicon embeddings of each character/word;
Train the first-level LSTM on the basis of the input embedding vectors and labels of the T1 text clauses;
Run the learned first-level LSTM on each clause of the text samples in T2. Record the predicted score INLINEFORM0 and the final dense vector INLINEFORM1 for each clause;
Construct the embedding vector for each clause in the text samples in T2. Each embedding vector consists of INLINEFORM0 , INLINEFORM1 , and the lexicon embedding of conjunctions of each clause;
Train the second-level LSTM on the basis of the input embedding vectors and labels of the T2 text samples.
The first-level and second-level LSTM networks consist of the final two-level LSTM.
The proposed two-level LSTM can be applied to texts with arbitrary languages. Word information is required in lexical construction regardless of whether character or word embedding is used. The reason is that the three types of lexicon embeddings are performed at the word level. Therefore, when character embedding is used, the lexicon embedding of each character is the lexicon embedding of the word containing it.
This section shows the evaluation of the proposed methodology in terms of the two-level LSTM network and each part of the lexicon embedding.
We compile three Chinese text corpora from online data for three domains, namely, “hotel", “mobile phone (mobile)", and “travel". All texts are about user reviews. Each text sample collected is first partitioned into clauses according to Chinese tokens. Three clause sets are subsequently obtained from the three text corpora.
The labels “+1", “0.5", and “0" correspond to the three sentiment classes “positive", “neutral", and “negative", respectively. The text data are labeled according to our two-stage labeling strategy.
In the first stage, only one user is invited to label each clause sample as the sentiment orientations for clauses (or sub-sentences) are easy to label.
In the second stage, five users are invited to label each text sample in the three raw data sets. The average score of the five users on each sample is calculated. Samples with average scores located in [0.6, 1] are labeled as “positive". Samples with average scores located in [0, 0.4] are labeled as “negative". Others are labeled as “neutral". The details of the labeling results are shown in Table 1.
All the training and test data and the labels are available online.
In our experiments, the five types of key lexical words introduced in Subsection 3.3.2 are manually constructed. The details of the five types of words are listed in Table 2. The conjunction words are also manually constructed. The number of conjunction words used in the experiments is 169.
In each experimental run, the training set is compiled on the basis of the training data listed in Table 1. The compiling rule is specified before each experimental run. The test data are fixed to facilitate experimental duplication and comparison by other researchers.
In our experiments, three competing algorithms, namely, BOW, CNN, and (conventional) LSTM, are used.
For BOW, term frequency-inverse document frequency is utilized to construct features. Ridge regression BIBREF29 is used as a classifier. For CNN, a three-channel CNN is used. For LSTM, one-layer and two-layer bi-LSTM with attention are adopted, and the results of the network with superior performance are presented. CNN and LSTM are performed on TensorFlow, and default parameter settings are followed.
The key parameters are searched as follows. The embedding dimensions of characters and words are searched in [100, 150, 200, 250, 300]. The parameter INLINEFORM0 in INLINEFORM1 -hot encoding is searched in INLINEFORM2 .
In this subsubsection, each of the three raw data sets (associated with their labels) shown in Table 1 is used. The clause data are not used. In other words, the training data used in this subsubsection are the same as those used in previous studies. For each data corpus, 1000 raw data samples are used as the test data, and the rest are used as the training data. The involved algorithms are detailed as follows.
CNN-C denotes the CNN with (Chinese) character embedding.
CNN-W denotes the CNN with (Chinese) word embedding.
CNN-Lex-C denotes the algorithm which also integrates polar words in CNN which is proposed by Shin et al. BIBREF24 . The (Chinese) character embedding is used.
CNN-Lex-W denotes the algorithm which also integrates polar words in CNN which is proposed by Shin et al. BIBREF24 . The (Chinese) word embedding is used.
Bi-LSTM-C denotes the BI-LSTM with (Chinese) character embedding.
Bi-LSTM-W denotes the Bi-LSTM with (Chinese) word embedding.
Lex-rule denotes the rule-based approach shows in Fig. 1. This approach is unsupervised.
BOW denotes the conventional algorithm which is based of bag-of-words features.
The accuracies of the above algorithms are listed in Table 3. Overall, Bi-LSTM outperforms CNN and BOW. This conclusion is in accordance with the conclusion that RNN performs efficiently against CNN in a broad range of natural language processing (NLP) tasks on the basis of extensive comparative studies BIBREF30 . In addition, CNN-lex outperforms CNN under both character and word embeddings, which suggests that lexicon cues are useful in sentiment analysis. Lex-rule achieves the lowest accuracies on all the three data sets. Considering that the performances of (traditional) CNN, Lex-rule, and BOW are relatively poor, they are not applied in the remaining parts.
In this experimental comparison, the proposed two-level LSTM is evaluated, whereas lexicon embedding is not used in the entire network. The primary goal is to test whether the introduced two-stage labeling and the two-level network structure are useful for sentiment analysis.
The raw and clause data listed in Table 1 are used to perform the two-level LSTM. Tl-LSTM denotes the two-level LSTM. “R+C" refer to the mixed data of raw and clause data. The test data are still the 1000 samples used in section 4.3.1 for each corpus. Table 4 shows the classification accuracies. To ensure that the results differ from those in Table 3, we explicitly add “R+C" after each algorithm in Table 4. In the last line of Table 4, the base results for each corpus in Table 3 are also listed.
On all the three data corpora, the proposed two-level network (without lexicon embedding) with character embedding, Tl-LSTM-C, outperforms all the other involved algorithms. On the travel and the mobile corpora, TI-LSTM-W outperforms Bi-LSTM-W. The results in Table 4 indicate that the performances of Tl-LSTM on the mixed training and test data (R+C) are better than those of Bi-LSTM. This comparison indicates that the proposed two-level LSTM is effective.
In addition, for the involved algorithms, most results achieved on “R+C" are better than the best results only achieved on `R'listed in Table 3. This comparison suggests that the introduced two-stage labeling is useful.
The results also show that in the two-level LSTM, character embedding is more effective than word embedding.
In this experimental run, lexicon embedding is used in the proposed two-level LSTM or INLINEFORM0 Tl-LSTM. Table 5 shows the results. The optimal parameter INLINEFORM1 is about 11.
The performances of TI-LSTM with lexicon embedding (i.e., INLINEFORM0 Tl-LSTM) are consistently better than those of TI-LSTM without lexicon embedding (i.e., Tl-LSTM) listed in Table 5. The improved accuracies of INLINEFORM1 TI-LSTM over Tl-LSTM on the three data corpora are explicitly listed in Table 6.
The experimental evaluation discussed in Subsection 4.3 verifies the effectiveness of the proposed method, INLINEFORM0 Tl-LSTM. Unlike the conventional RNN, INLINEFORM1 Tl-LSTM contains lexicon embedding that consists of new technique and components, including INLINEFORM2 -hot encoding, embedding for polar words, embedding for POS, and embedding for conjunctions. Therefore, this subsection evaluates the performances of the involved technique and embeddings separately.
Our INLINEFORM0 -hot encoding differs from one-hot encoding in two aspects. The first aspect is that the nonzero values in one-hot encoding are only equal to 1, whereas the nonzero values in INLINEFORM1 -hot encoding are INLINEFORM2 . The second aspect is that only one element in one-hot encoding is nonzero, whereas n elements in INLINEFORM3 -hot encoding are nonzero.
In this experiment, we test whether INLINEFORM0 -hot encoding is useful in two experimental runs. In the first run, the value of INLINEFORM1 is manually set to 0.5 and 1 in the experimental run without optimization. The parameter INLINEFORM2 in Eq. (6) is set as 15. The classification accuracies vary according to different INLINEFORM3 values on all the three data corpora. When INLINEFORM4 equals 1, the accuracies are the lowest in most cases shown in Fig. 7.
The results shown in Fig. 7 indicate that the value of INLINEFORM0 does affect the performance of the entire network. Consequently, the classical one-hot encoding, which fixes the value of nonzero elements as 1, is ineffective. In our experiments, the learned value of INLINEFORM1 is approximate 0.4.
In the second run, the performances under different INLINEFORM0 (i.e., 1, 5, 10, 15) are tested. Table 7 shows the comparison results. The value of INLINEFORM1 does affect the performance of the entire network, thereby indicating that the introduction of the INLINEFORM2 -duplicated strategy in encoding is effective. In the experiments, when INLINEFORM3 is increasing, the accuracies first increase and then decrease. The main reason may lie in the fact that when INLINEFORM4 becomes large, the proportion of lexicon embedding becomes large accordingly. An over-length input feature vector may incur “curse of dimensionality" and thus weaken the performance of the proposed two-level network.
In this experimental run, we test whether the labeled polar (negative and positive) words do affect the performance of the entire method when they are used in lexicon embedding. To this end, we order the polar words according to their frequencies in the training data. Top 0%, 50%, 100% polar words are used. The corresponding classification accuracies are depicted in Fig. 8.
In most cases, the accuracies are the lowest when no polar words are used in the lexicon embedding. When all polar words are used, the proposed network achieves the highest accuracies.
In the experiment, only one user is invited to manually compile the dictionary for a data corpus. One and a half hour is needed for each data corpus. In our viewpoint, it is worth manually compiling the polar words for sentiment analysis by considering the performance improvement and time-consumption.
In this experimental run, we test whether POS cues do play positive roles in the entire model. To this end, we remove POS in the lexicon embedding of the proposed method. The results are shown in Fig. 9.
In most cases, the accuracies with POS embedding are greater than those without POS embedding, thereby indicating that the application of POS to lexicon embedding is useful.
In this experimental run, we test whether conjunction cues do play positive roles in the entire model. To this end, the lexicon embedding for conjunction words is also removed from the proposed method. The results are shown in Fig. 10.
The algorithm with conjunction embedding outperforms that without conjunction embedding consistently, thereby indicating that the application of conjunction to lexicon embedding is useful.
High-quality labels are crucial for learning systems. Nevertheless, texts with mixed sentiments are difficult for humans to label in text sentiment classification. In this study, a new labeling strategy is introduced to partition texts into those with pure and mixed sentiment orientations. These two categories of texts are labeled using different processes. A two-level network is accordingly proposed to utilize the two labeled data in our two-stage labeling strategy. Lexical cues (e.g., polar words, POS, conjunction words) are particularly useful in sentiment analysis. These lexical cues are used in our two-level network, and a new encoding strategy, that is, INLINEFORM0 -hot encoding, is introduced. INLINEFORM1 -hot encoding is motivated by one-hot encoding. However, the former alleviates the drawbacks of the latter. Three Chinese sentiment text data corpora are compiled to verify the effectiveness of the proposed methodology. Our proposed method achieves the highest accuracies on these three data corpora.
The proposed two-level network and lexicon embedding can also be applied to other types of deep neural networks. In our future work, we will extend our main idea into several networks and text mining applications.
The authors wish to thank Zefeng Han, Qing Yin, Lei Yang, Xiaonan Wang, Nan Chen, Rujing Yao, Lihong Guo, Pinglong Zhao for the labeling of the experimental data.

Figure and table captions:
Figure 1. A lexicon-based approach for sentiment classification.
Figure 2. Two representative LSTM structures for text classification: a bi-directional LSTM (left), and a bi-directional LSTM with attention (right).
Figure 3. The proposed two-level LSTM network.
Figure 4. The histogram of the values in word embedding vectors. Most values are smaller than 1.
Figure 5. The first-level LSTM with lexicon embedding in both the input and attention layers.
Figure 6. The whole two-level LSTM network with lexicon embedding in both the input and attention layers.
TABLE 1 Details of the three data corpora. Each corpus consists of raw samples (sentences or paragraphs) and partitioned clauses (sub-sentences).
TABLE 2 Numbers of five types of key lexical words.
TABLE 4 The classification accuracies of two-level LSTM without lexicon embedding.
TABLE 3 The classification accuracies of existing algorithms on raw samples.
Figure 7. Classification accuracies under different ρ values. #1-C and #1-W represent ρTl-LSTM-C and ρTl-LSTM-W on the first (travel) corpus, respectively; #2-C and #2-W represent ρTl-LSTM-C and ρTl-LSTMW on the second (hotel) corpus, respectively; #3-C and #3-W represent ρTl-LSTM-C and ρTl-LSTM-W on the third (hotel) corpus, respectively.
TABLE 6 The accuracy improvement of two-level LSTM when lexicon embedding was used over those of two-level LSTM without lexicon embedding. The values of the last row are the accuracy improvement over the highest accuracies on each data corpus with existing algorithms.
TABLE 7 The accuracies of ρTl-LSTM with different n values in ρ-hot encoding.
TABLE 5 The classification accuracies of two-level LSTM with lexicon embedding.
Figure 8. Classification accuracies under different proportions of polar words. #1-C and #1-W represent ρTl-LSTM-C and ρTl-LSTM-W on the first (travel) corpus, respectively; #2-C and #2-W represent ρTl-LSTMC and ρTl-LSTM-W on the second (hotel) corpus, respectively; #3-C and #3-W represent ρTl-LSTM-C and ρTl-LSTM-W on the third (hotel) corpus, respectively.
Figure 10. Classification accuracies with and without conjunction in lexicon embedding. #1-C and #1-W represent ρTl-LSTM-C and ρTl-LSTMW on the first (travel) corpus, respectively; #2-C and #2-W represent ρTlLSTM-C and ρTl-LSTM-W on the second (hotel) corpus, respectively; #3-C and #3-W represent ρTl-LSTM-C and ρTl-LSTM-W on the third (hotel) corpus, respectively.
Figure 9. Classification accuracies with and without POS in lexicon embedding. #1-C and #1-W represent ρTl-LSTM-C and ρTl-LSTM-W on the first (travel) corpus, respectively; #2-C and #2-W represent ρTlLSTM-C and ρTl-LSTM-W on the second (hotel) corpus, respectively; #3-C and #3-W represent ρTl-LSTM-C and ρTl-LSTM-W on the third (hotel) corpus, respectively.

Questions about this paper:
How long are the datasets?
What are the sources of the data?
What is the new labeling strategy?

Answers:
Travel dataset contains 4100 raw samples, 11291 clauses, Hotel dataset contains 3825 raw samples, 11264 clauses, and the Mobile dataset contains 3483 raw samples and 8118 clauses
User reviews written in Chinese collected online for hotel, mobile phone, and travel domains
They use a two-stage labeling strategy where in the first stage single annotators label a large number of short texts with relatively pure sentiment orientations and in the second stage multiple annotators label few text samples with mixed sentiment orientations