Introduction
Determining the sentiment polarity of tweets has become a landmark homework exercise in natural language processing (NLP) and data science classes. This is perhaps because the task is easy to understand and it is also easy to get good results with very simple methods (e.g. positive - negative words counting). The practical applications of this task are wide, from monitoring popular events (e.g. Presidential debates, Oscars, etc.) to extracting trading signals by monitoring tweets about public companies. These applications often benefit greatly from the best possible accuracy, which is why the SemEval-2017 Twitter competition promotes research in this area. The competition is divided into five subtasks which involve standard classification, ordinal classification and distributional estimation. For a more detailed description see BIBREF0 .
In the last few years, deep learning techniques have significantly out-performed traditional methods in several NLP tasks BIBREF1 , BIBREF2 , and sentiment analysis is no exception to this trend BIBREF3 . In fact, previous iterations of the SemEval Twitter sentiment analysis competition have already established their power over other approaches BIBREF4 , BIBREF5 , BIBREF6 . Two of the most popular deep learning techniques for sentiment analysis are CNNs and LSTMs. Consequently, in an effort to build a state-of-the-art Twitter sentiment classifier, we explore both models and build a system which combines both.
This paper is organized as follows. In sec. SECREF2 we describe the architecture of the CNN and the LSTM used in our system. In sec. SECREF3 we expand on the three training phases used in our system. In sec. SECREF4 we discuss the various tricks that were used to fine tune the system for each individual subtasks. Finally in sec. SECREF5 we present the performance of the system and in sec. SECREF6 we outline our main conclusions.
CNN
Let us now describe the architecture of the CNN we worked with. Its architecture is almost identical to the CNN of BIBREF8 . A smaller version of our model is illustrated on Fig. FIGREF2 . The input of the network are the tweets, which are tokenized into words. Each word is mapped to a word vector representation, i.e. a word embedding, such that an entire tweet can be mapped to a matrix of size INLINEFORM0 , where INLINEFORM1 is the number of words in the tweet and INLINEFORM2 is the dimension of the embedding space (we chose INLINEFORM3 ). We follow BIBREF8 zero-padding strategy such that all tweets have the same matrix dimension INLINEFORM4 , where we chose INLINEFORM5 . We then apply several convolution operations of various sizes to this matrix. A single convolution involves a filtering matrix INLINEFORM6 where INLINEFORM7 is the size of the convolution, meaning the number of words it spans. The convolution operation is defined as DISPLAYFORM0
where INLINEFORM0 is a bias term and f(x) is a non-linear function, which we chose to be the relu function. The output INLINEFORM1 is therefore a concatenation of the convolution operator over all possible window of words in the tweet. Note that because of the zero-padding strategy we use, we are effectively applying wide convolutions BIBREF9 . We can use multiple filtering matrices to learn different features, and additionally we can use multiple convolution sizes to focus on smaller or larger regions of the tweets. In practice, we used three filter sizes (either INLINEFORM2 , INLINEFORM3 or INLINEFORM4 depending on the model) and we used a total of 200 filtering matrices for each filter size.
We then apply a max-pooling operation to each convolution INLINEFORM0 . The max-pooling operation extracts the most important feature for each convolution, independently of where in the tweet this feature is located. In other words, the CNN's structure effectively extracts the most important n-grams in the embedding space, which is why we believe these systems are good at sentence classification. The max-pooling operation also allows us to combine all the INLINEFORM1 of each filter into one vector INLINEFORM2 where m is the total number of filters (in our case INLINEFORM3 ). This vector then goes through a small fully connected hidden layer of size 30, which is then in turn passed through a softmax layer to give the final classification probabilities. To reduce over-fitting, we add a dropout layer BIBREF10 after the max-pooling layer and after the fully connected hidden layer, with a dropout probability of 50 INLINEFORM4 during training.
LSTM
Let us now describe the architecture of the LSTM system we worked with. A smaller version of our model is illustrated on Fig. FIGREF3 . Its main building blocks are two LSTM units. LSTMs are part of the recurrent neural networks (RNN) family, which are neural networks that are constructed to deal with sequential data by sharing their internal weights across the sequence. For each element in the sequence, that is for each word in the tweet, the RNN uses the current word embedding and its previous hidden state to compute the next hidden state. In its simplest version, the hidden state INLINEFORM0 (where m is the dimension of the RNN, which we pick to be INLINEFORM1 ) at time INLINEFORM2 is computed by DISPLAYFORM0
where INLINEFORM0 is the current word embedding, INLINEFORM1 and INLINEFORM2 are weight matrices, INLINEFORM3 is a bias term and INLINEFORM4 is a non-linear function, usually chosen to be INLINEFORM5 . The initial hidden state is chosen to be a vector of zeros. Unfortunately this simple RNN suffers from the exploding and vanishing gradient problem during the backpropagation training stage BIBREF11 . LSTMs solve this problem by having a more complex internal structure which allows LSTMs to remember information for either long or short terms BIBREF12 . The hidden state of an LSTM unit is computed by BIBREF13 DISPLAYFORM0
where INLINEFORM0 is called the input gate, INLINEFORM1 is the forget gate, INLINEFORM2 is the cell state, INLINEFORM3 is the regular hidden state, INLINEFORM4 is the sigmoid function, and INLINEFORM5 is the Hadamard product.
One drawback from the LSTM is that it does not sufficiently take into account post word information because the sentence is read only in one direction; forward. To solve this problem, we use what is known as a bidirectional LSTM, which is two LSTMs whose outputs are stacked together. One LSTM reads the sentence forward, and the other LSTM reads it backward. We concatenate the hidden states of each LSTM after they processed their respective final word. This gives a vector of dimension INLINEFORM0 , which is fed to a fully connected hidden layer of size 30, and then passed through a softmax layer to give the final classification probabilities. Here again we use dropout to reduce over-fitting; we add a dropout layer before and after the LSTMs, and after the fully connected hidden layer, with a dropout probability of 50 INLINEFORM1 during training.
Training
To train those models we had access to 49,693 human labeled tweets for subtask A, 30,849 tweets for subtasks (C, E) and 18,948 tweets for subtasks (B, D). In addition to this human labeled data, we collected 100 million unique unlabeled English tweets using the Twitter streaming API. From this unlabeled dataset, we extracted a distant dataset of 5 million positive tweets and 5 million negative tweets. To extract this distant dataset we used the strategy of BIBREF14 , that is we simply associate positive tweets with the presence of positive emoticons (e.g. “:)”) and vice versa for negative tweets. Those three datasets (unlabeled, distant and labeled) were used separately in the three training stages which we now present. Note that our training strategy is very similar to the one used in BIBREF5 , BIBREF6 .
Pre-processing
Before feeding the tweets to any training stage, they are pre-processed using the following procedure:
URLs are replaced by the INLINEFORM0 url INLINEFORM1 token.
Several emoticons are replaced by the tokens INLINEFORM0 smile INLINEFORM1 , INLINEFORM2 sadface INLINEFORM3 , INLINEFORM4 lolface INLINEFORM5 or INLINEFORM6 neutralface INLINEFORM7 .
Any letter repeated more than 2 times in a row is replaced by 2 repetitions of that letter (for example, “sooooo” is replaced by “soo”).
All tweets are lowercased.
Unsupervised training
We start by using the 100 million unlabeled tweets to pre-train the word embeddings which will later be used in the CNN and LSTM. To do so, we experimented with 3 unsupervised learning algorithms, Google's Word2vec BIBREF15 , BIBREF16 , Facebook's FastText BIBREF17 and Stanford's GloVe BIBREF18 . Word2vec learns word vector representations by attempting to predict context words around an input word. FastText is very similar to Word2vec but it also uses subword information in the prediction model. GloVe on the other hand is a model based on global word-word co-occurrence statistics. For all three algorithms we used the code provided by the authors with their default settings.
Distant training
The embeddings learned in the unsupervised phase contain very little information about the sentiment polarity of the words since the context for a positive word (ex. “good”) tends to be very similar to the context of a negative word (ex. “bad”). To add polarity information to the embeddings, we follow the unsupervised training by a fine tuning of the embeddings via a distant training phase. To do so, we use the CNN described in sec. SECREF2 and initialize the embeddings with the ones learned in the unsupervised phase. We then use the distant dataset to train the CNN to classify noisy positive tweets vs. noisy negative tweets. The first epoch of the training is done with the embeddings frozen in order to minimize large changes in the embeddings. We then unfreeze the embeddings and train for 6 more epochs. After this training stage, words with very different sentiment polarity (ex. “good” vs. “bad”) are far apart in the embedding space.
Supervised training
The final training stage uses the human labeled data provided by SemEval-2017. We initialize the embeddings in the CNN and LSTM models with the fine tuned embeddings of the distant training phase, and freeze them for the first INLINEFORM0 epochs. We then train for another INLINEFORM1 epochs with unfrozen embeddings and a learning rate reduced by a factor of 10. We pick the cross-entropy as the loss function, and we weight it by the inverse frequency of the true classes to counteract the imbalanced dataset. The loss is minimized using the Adam optimizer BIBREF19 with initial learning rate of 0.001. The models were implemented in TensorFlow and experiments were run on a GeForce GTX Titan X GPU.
To reduce variance and boost accuracy, we ensemble 10 CNNs and 10 LSTMs together through soft voting. The models ensembled have different random weight initializations, different number of epochs (from 4 to 20 in total), different set of filter sizes (either INLINEFORM0 , INLINEFORM1 or INLINEFORM2 ) and different embedding pre-training algorithms (either Word2vec or FastText).
Subtask specific tricks
The models described in sec. SECREF2 and the training method described in sec. SECREF3 are used in the same way for all five subtasks, with a few special exceptions which we now address. Clearly, the output dimension differs depending on the subtask, for subtask A the output dimension is 3, while for B and D it is 2 and for subtask C and E it is 5. Furthermore, for quantification subtasks (D and E), we use the probability average approach of BIBREF20 to convert the output probabilities into sentiment distributions.
Finally for subtasks that have a topic associated with the tweet (B, C, D and E), we add two special steps which we noticed improves the accuracy during the cross-validation phase. First, if any of the words in the topic is not explicitly mentioned in the tweet, we add those missing words at the end of the tweet in the pre-processing phase. Second, we concatenate to the regular word embeddings another embedding space of dimension 5 which has only 2 possible vectors. One of these 2 vectors indicates that the current word is part of the topic, while the other vector indicates that the current word is not part of the topic.
Results
Let us now discuss the results obtained from this system. In order to assess the performance of each model and their variations, we first show their scores on the historical Twitter test set of 2013, 2014, 2015 and 2016 without using any of those sets in the training dataset, just like it was required for the 2016 edition of this competition. For brevity, we only focus on task A since it tends to be the most popular one. Moreover, in order to be consistent with historical editions of this competition, we use the average INLINEFORM0 score of the positive and negative class as the metric of interest. This is different from the macro-average recall which is used in the 2017 edition, but this should not affect the conclusions of this analysis significantly since we found that the two metrics were highly correlated. The results are summarized in Table TABREF16 . This table is not meant to be an exhaustive list of all the experiments performed, but it does illustrate the relative performances of the most important variations on the models explored here.
We can see from Table TABREF16 that the GloVe unsupervised algorithm gives a lower score than both FastText and Word2vec. It is for this reason that we did not include the GloVe variation in the ensemble model. We also note that the absence of class weights or the absence of a distant training stage lowers the scores significantly, which demonstrates that these are sound additions. Except for these three variations, the other models have similar scores. However, the ensemble model effectively outperforms all the other individual models. Indeed, while these individual models give similar scores, their outputs are sufficiently uncorrelated such that ensembling them gives the score a small boost. To get a sense of how correlated with each other these models are, we can compute the Pearson correlation coefficient between the output probabilities of any pairs of models, see Table TABREF17 . From this table we can see that the most uncorrelated models come from different supervised learning models (CNN vs. LSTM) and from different unsupervised learning algorithms (Word2vec vs. FastText).
For the predictions on the 2017 test set, the system is retrained on all available training data, which includes previous years testing data. The results of our system on the 2017 test set are shown on Table TABREF18 . Our system achieved the best scores on all of the five English subtasks. For subtask A, there is actually a tie between our submission and another team (DataStories), but note that with respect to the other metrics (accuracy and INLINEFORM0 score) our submission ranks higher.
Conclusion
In this paper we presented the system we used to compete in the SemEval-2017 Twitter sentiment analysis competition. Our goal was to experiment with deep learning models along with modern training strategies in an effort to build the best possible sentiment classifier for tweets. The final model we used was an ensemble of 10 CNNs and 10 LSTMs with different hyper-parameters and different pre-training strategies. We participated in all of the English subtasks, and obtained first rank in all of them.
For future work, it would be interesting to explore systems that combine a CNN and an LSTM more organically than through an ensemble model, perhaps a model similar to the one of BIBREF21 . It would also be interesting to analyze the dependence of the amount of unlabeled and distant data on the performance of the models.
Acknowledgments
We thank Karl Stratos, Anju Kambadur, Liang Zhou, Alexander M. Rush, David Rosenberg and Biye Li for their help on this project.

Question: What were the five English subtasks?
Answer:  five subtasks which involve standard classification, ordinal classification and distributional estimation. For a more detailed description see BIBREF0

Question: How many CNNs and LSTMs were ensembled?
Answer: 10 CNNs and 10 LSTMs