A Deep Neural Architecture for Sentence-level Sentiment Classification in Twitter Social Networking

Abstract: This paper introduces a novel deep learning framework including a lexicon-based approach for sentence-level prediction of sentiment label distribution. We propose to first apply semantic rules and then use a Deep Convolutional Neural Network (DeepCNN) for character-level embeddings in order to increase information for word-level embedding. After that, a Bidirectional Long Short-Term Memory Network (Bi-LSTM) produces a sentence-wide feature representation from the word-level embedding. We evaluate our approach on three Twitter sentiment classification datasets. Experimental results show that our model can improve the classification accuracy of sentence-level sentiment analysis in Twitter social networking.

Twitter sentiment classification have intensively researched in recent years BIBREF0 BIBREF1 . Different approaches were developed for Twitter sentiment classification by using machine learning such as Support Vector Machine (SVM) with rule-based features BIBREF2 and the combination of SVMs and Naive Bayes (NB) BIBREF3 . In addition, hybrid approaches combining lexicon-based and machine learning methods also achieved high performance described in BIBREF4 . However, a problem of traditional machine learning is how to define a feature extractor for a specific domain in order to extract important features.
Deep learning models are different from traditional machine learning methods in that a deep learning model does not depend on feature extractors because features are extracted during training progress. The use of deep learning methods becomes to achieve remarkable results for sentiment analysis BIBREF5 BIBREF6 BIBREF7 . Some researchers used Convolutional Neural Network (CNN) for sentiment classification. CNN models have been shown to be effective for NLP. For example, BIBREF6 proposed various kinds of CNN to learn sentiment-bearing sentence vectors, BIBREF5 adopted two CNNs in character-level to sentence-level representation for sentiment analysis. BIBREF7 constructs experiments on a character-level CNN for several large-scale datasets. In addition, Long Short-Term Memory (LSTM) is another state-of-the-art semantic composition model for sentiment classification with many variants described in BIBREF8 . The studies reveal that using a CNN is useful in extracting information and finding feature detectors from texts. In addition, a LSTM can be good in maintaining word order and the context of words. However, in some important aspects, the use of CNN or LSTM separately may not capture enough information.
Inspired by the models above, the goal of this research is using a Deep Convolutional Neural Network (DeepCNN) to exploit the information of characters of words in order to support word-level embedding. A Bi-LSTM produces a sentence-wide feature representation based on these embeddings. The Bi-LSTM is a version of BIBREF9 with Full Gradient described in BIBREF10 . In addition, the rules-based approach also effects classification accuracy by focusing on important sub-sentences expressing the main sentiment of a tweet while removing unnecessary parts of a tweet. The paper makes the following contributions:
The organization of the present paper is as follows: In section 2, we describe the model architecture which introduces the structure of the model. We explain the basic idea of model and the way of constructing the model. Section 3 show results and analysis and section 4 summarize this paper.
Our proposed model consists of a deep learning classifier and a tweet processor. The deep learning classifier is a combination of DeepCNN and Bi-LSTM. The tweet processor standardizes tweets and then applies semantic rules on datasets. We construct a framework that treats the deep learning classifier and the tweet processor as two distinct components. We believe that standardizing data is an important step to achieve high accuracy. To formulate our problem in increasing the accuracy of the classifier, we illustrate our model in Figure. FIGREF4 as follows:
Tweets are firstly considered via a processor based on preprocessing steps BIBREF0 and the semantic rules-based method BIBREF11 in order to standardize tweets and capture only important information containing the main sentiment of a tweet.
We use DeepCNN with Wide convolution for character-level embeddings. A wide convolution can learn to recognize specific n-grams at every position in a word that allows features to be extracted independently of these positions in the word. These features maintain the order and relative positions of characters. A DeepCNN is constructed by two wide convolution layers and the need of multiple wide convolution layers is widely accepted that a model constructing by multiple processing layers have the ability to learn representations of data with higher levels of abstraction BIBREF12 . Therefore, we use DeepCNN for character-level embeddings to support morphological and shape information for a word. The DeepCNN produces INLINEFORM0 global fixed-sized feature vectors for INLINEFORM1 words.
A combination of the global fixed-size feature vectors and word-level embedding is fed into Bi-LSTM. The Bi-LSTM produces a sentence-level representation by maintaining the order of words.
Our work is philosophically similar to BIBREF5 . However, our model is distinguished with their approaches in two aspects:
Using DeepCNN with two wide convolution layers to increase representation with multiple levels of abstraction.
Integrating global character fixed-sized feature vectors with word-level embedding to extract a sentence-wide feature set via Bi-LSTM. This deals with three main problems: (i) Sentences have any different size; (ii) The semantic and the syntactic of words in a sentence are captured in order to increase information for a word; (iii) Important information of characters that can appear at any position in a word are extracted.
In sub-section B, we introduce various kinds of dataset. The modules of our model are constructed in other sub-sections.
Stanford - Twitter Sentiment Corpus (STS Corpus): STS Corpus contains 1,600K training tweets collected by a crawler from BIBREF0 . BIBREF0 constructed a test set manually with 177 negative and 182 positive tweets. The Stanford test set is small. However, it has been widely used in different evaluation tasks BIBREF0 BIBREF5 BIBREF13 .
Sanders - Twitter Sentiment Corpus: This dataset consists of hand-classified tweets collected by using search terms: INLINEFORM0 , #google, #microsoft and #twitter. We construct the dataset as BIBREF14 for binary classification.
Health Care Reform (HCR): This dataset was constructed by crawling tweets containing the hashtag #hcr BIBREF15 . Task is to predict positive/negative tweets BIBREF14 .
We firstly take unique properties of Twitter in order to reduce the feature space such as Username, Usage of links, None, URLs and Repeated Letters. We then process retweets, stop words, links, URLs, mentions, punctuation and accentuation. For emoticons, BIBREF0 revealed that the training process makes the use of emoticons as noisy labels and they stripped the emoticons out from their training dataset because BIBREF0 believed that if we consider the emoticons, there is a negative impact on the accuracies of classifiers. In addition, removing emoticons makes the classifiers learns from other features (e.g. unigrams and bi-grams) presented in tweets and the classifiers only use these non-emoticon features to predict the sentiment of tweets. However, there is a problem is that if the test set contains emoticons, they do not influence the classifiers because emoticon features do not contain in its training data. This is a limitation of BIBREF0 , because the emoticon features would be useful when classifying test data. Therefore, we keep emoticon features in the datasets because deep learning models can capture more information from emoticon features for increasing classification accuracy.
In Twitter social networking, people express their opinions containing sub-sentences. These sub-sentences using specific PoS particles (Conjunction and Conjunctive adverbs), like "but, while, however, despite, however" have different polarities. However, the overall sentiment of tweets often focus on certain sub-sentences. For example:
@lonedog bwahahah...you are amazing! However, it was quite the letdown.
@kirstiealley my dentist is great but she's expensive...=(
In two tweets above, the overall sentiment is negative. However, the main sentiment is only in the sub-sentences following but and however. This inspires a processing step to remove unessential parts in a tweet. Rule-based approach can assists these problems in handling negation and dealing with specific PoS particles led to effectively affect the final output of classification BIBREF11 BIBREF16 . BIBREF11 summarized a full presentation of their semantic rules approach and devised ten semantic rules in their hybrid approach based on the presentation of BIBREF16 . We use five rules in the semantic rules set because other five rules are only used to compute polarity of words after POS tagging or Parsing steps. We follow the same naming convention for rules utilized by BIBREF11 to represent the rules utilized in our proposed method. The rules utilized in the proposed method are displayed in Table TABREF15 in which is included examples from STS Corpus and output after using the rules. Table TABREF16 illustrates the number of processed sentences on each dataset.
To construct embedding inputs for our model, we use a fixed-sized word vocabulary INLINEFORM0 and a fixed-sized character vocabulary INLINEFORM1 . Given a word INLINEFORM2 is composed from characters INLINEFORM3 , the character-level embeddings are encoded by column vectors INLINEFORM4 in the embedding matrix INLINEFORM5 , where INLINEFORM6 is the size of the character vocabulary. For word-level embedding INLINEFORM7 , we use a pre-trained word-level embedding with dimension 200 or 300. A pre-trained word-level embedding can capture the syntactic and semantic information of words BIBREF17 . We build every word INLINEFORM8 into an embedding INLINEFORM9 which is constructed by two sub-vectors: the word-level embedding INLINEFORM10 and the character fixed-size feature vector INLINEFORM11 of INLINEFORM12 where INLINEFORM13 is the length of the filter of wide convolutions. We have INLINEFORM14 character fixed-size feature vectors corresponding to word-level embedding in a sentence.
DeepCNN in the deep learning module is illustrated in Figure. FIGREF22 . The DeepCNN has two wide convolution layers. The first layer extract local features around each character windows of the given word and using a max pooling over character windows to produce a global fixed-sized feature vector for the word. The second layer retrieves important context characters and transforms the representation at previous level into a representation at higher abstract level. We have INLINEFORM0 global character fixed-sized feature vectors for INLINEFORM1 words.
In the next step of Figure. FIGREF4 , we construct the vector INLINEFORM0 by concatenating the word-level embedding with the global character fixed-size feature vectors. The input of Bi-LSTM is a sequence of embeddings INLINEFORM1 . The use of the global character fixed-size feature vectors increases the relationship of words in the word-level embedding. The purpose of this Bi-LSTM is to capture the context of words in a sentence and maintain the order of words toward to extract sentence-level representation. The top of the model is a softmax function to predict sentiment label. We describe in detail the kinds of CNN and LSTM that we use in next sub-part 1 and 2.
The one-dimensional convolution called time-delay neural net has a filter vector INLINEFORM0 and take the dot product of filter INLINEFORM1 with each m-grams in the sequence of characters INLINEFORM2 of a word in order to obtain a sequence INLINEFORM3 : DISPLAYFORM0 
Based on Equation 1, we have two types of convolutions that depend on the range of the index INLINEFORM0 . The narrow type of convolution requires that INLINEFORM1 and produce a sequence INLINEFORM2 . The wide type of convolution does not require on INLINEFORM3 or INLINEFORM4 and produce a sequence INLINEFORM5 . Out-of-range input values INLINEFORM6 where INLINEFORM7 or INLINEFORM8 are taken to be zero. We use wide convolution for our model.
Given a word INLINEFORM0 composed of INLINEFORM1 characters INLINEFORM2 , we take a character embedding INLINEFORM3 for each character INLINEFORM4 and construct a character matrix INLINEFORM5 as following Equation. 2: DISPLAYFORM0 
The values of the embeddings INLINEFORM0 are parameters that are optimized during training. The trained weights in the filter INLINEFORM1 correspond to a feature detector which learns to recognize a specific class of n-grams. The n-grams have size INLINEFORM2 . The use of a wide convolution has some advantages more than a narrow convolution because a wide convolution ensures that all weights of filter reach the whole characters of a word at the margins. The resulting matrix has dimension INLINEFORM3 .
Long Short-Term Memory networks usually called LSTMs are a improved version of RNN. The core idea behind LSTMs is the cell state which can maintain its state over time, and non-linear gating units which regulate the information flow into and out of the cell. The LSTM architecture that we used in our proposed model is described in BIBREF9 . A single LSTM memory cell is implemented by the following composite function: DISPLAYFORM0 DISPLAYFORM1 
where INLINEFORM0 is the logistic sigmoid function, INLINEFORM1 and INLINEFORM2 are the input gate, forget gate, output gate, cell and cell input activation vectors respectively. All of them have a same size as the hidden vector INLINEFORM3 . INLINEFORM4 is the hidden-input gate matrix, INLINEFORM5 is the input-output gate matrix. The bias terms which are added to INLINEFORM6 and INLINEFORM7 have been omitted for clarity. In addition, we also use the full gradient for calculating with full backpropagation through time (BPTT) described in BIBREF10 . A LSTM gradients using finite differences could be checked and making practical implementations more reliable.
For regularization, we use a constraint on INLINEFORM0 of the weight vectors BIBREF18 .
For the Stanford Twitter Sentiment Corpus, we use the number of samples as BIBREF5 . The training data is selected 80K tweets for a training data and 16K tweets for the development set randomly from the training data of BIBREF0 . We conduct a binary prediction for STS Corpus.
For Sander dataset, we use standard 10-fold cross validation as BIBREF14 . We construct the development set by selecting 10% randomly from 9-fold training data.
In Health Care Reform Corpus, we also select 10% randomly for the development set in a training set and construct as BIBREF14 for comparison. We describe the summary of datasets in Table III.
for all datasets, the filter window size ( INLINEFORM0 ) is 7 with 6 feature maps each for the first wide convolution layer, the second wide convolution layer has a filter window size of 5 with 14 feature maps each. Dropout rate ( INLINEFORM1 ) is 0.5, INLINEFORM2 constraint, learning rate is 0.1 and momentum of 0.9. Mini-batch size for STS Corpus is 100 and others are 4. In addition, training is done through stochastic gradient descent over shuffled mini-batches with Adadelta update rule BIBREF19 .
we use the publicly available Word2Vec trained from 100 billion words from Google and TwitterGlove of Stanford is performed on aggregated global word-word co-occurrence statistics from a corpus. Word2Vec has dimensionality of 300 and Twitter Glove have dimensionality of 200. Words that do not present in the set of pre-train words are initialized randomly.
Table IV shows the result of our model for sentiment classification against other models. We compare our model performance with the approaches of BIBREF0 BIBREF5 on STS Corpus. BIBREF0 reported the results of Maximum Entropy (MaxEnt), NB, SVM on STS Corpus having good performance in previous time. The model of BIBREF5 is a state-of-the-art so far by using a CharSCNN. As can be seen, 86.63 is the best prediction accuracy of our model so far for the STS Corpus.
For Sanders and HCR datasets, we compare results with the model of BIBREF14 that used a ensemble of multiple base classifiers (ENS) such as NB, Random Forest (RF), SVM and Logistic Regression (LR). The ENS model is combined with bag-of-words (BoW), feature hashing (FH) and lexicons. The model of BIBREF14 is a state-of-the-art on Sanders and HCR datasets. Our models outperform the model of BIBREF14 for the Sanders dataset and HCR dataset.
As can be seen, the models with SR outperforms the model with no SR. Semantic rules is effective in order to increase classification accuracy. We evaluate the efficiency of SR for the model in Table V of our full paper . We also conduct two experiments on two separate models: DeepCNN and Bi-LSTM in order to show the effectiveness of combination of DeepCNN and Bi-LSTM. In addition, the model using TwitterGlove outperform the model using GoogleW2V because TwitterGlove captures more information in Twitter than GoogleW2V. These results show that the character-level information and SR have a great impact on Twitter Data. The pre-train word vectors are good, universal feature extractors. The difference between our model and other approaches is the ability of our model to capture important features by using SR and combine these features at high benefit. The use of DeepCNN can learn a representation of words in higher abstract level. The combination of global character fixed-sized feature vectors and a word embedding helps the model to find important detectors for particles such as 'not' that negate sentiment and potentiate sentiment such as 'too', 'so' standing beside expected features. The model not only learns to recognize single n-grams, but also patterns in n-grams lead to form a structure significance of a sentence.
In the present work, we have pointed out that the use of character embeddings through a DeepCNN to enhance information for word embeddings built on top of Word2Vec or TwitterGlove improves classification accuracy in Tweet sentiment classification. Our results add to the well-establish evidence that character vectors are an important ingredient for word-level in deep learning for NLP. In addition, semantic rules contribute handling non-essential sub-tweets in order to improve classification accuracy.

Figure and table captions:
Figure 1. The overview of a deep learning system.
Table II THE NUMBER OF TWEETS ARE PROCESSED BY USING SEMANTIC RULES
Table I SEMANTIC RULES [12]
Figure 2. Deep Convolutional Neural Network (DeepCNN) for the sequence of character embeddings of a word. For example with 1 region size is 2 and 4 feature maps in the first convolution and 1 region size is 3 with 3 feature maps in the second convolution.
Table IV ACCURACY OF DIFFERENT MODELS FOR BINARY CLASSIFICATION
Table III SUMMARY STATISTICS FOR THE DATASETS AFTER USING SEMANTIC RULES. c: THE NUMBER OF CLASSES. N : THE NUMBER OF TWEETS. lw : MAXIMUM SENTENCE LENGTH. lc : MAXIMUM CHARACTER LENGTH. |Vw|: WORD ALPHABET SIZE. |Vc|: CHARACTER ALPHABET SIZE.

Questions about this paper:
What were their results on the three datasets?
What semantic rules are proposed?

Answers:
accuracy of 86.63 on STS, 85.14 on Sanders and 80.9 on HCR
rules that compute polarity of words after POS tagging or parsing steps