Tweet2Vec: Learning Tweet Embeddings Using Character-level CNN-LSTM Encoder-Decoder

Abstract: We present Tweet2Vec, a novel method for generating general-purpose vector representation of tweets. The model learns tweet embeddings using character-level CNN-LSTM encoder-decoder. We trained our model on 3 million, randomly selected English-language tweets. The model was evaluated using two methods: tweet semantic similarity and tweet sentiment categorization, outperforming the previous state-of-the-art in both tasks. The evaluations demonstrate the power of the tweet embeddings generated by our model for various tweet categorization tasks. The vector representations generated by our model are generic, and hence can be applied to a variety of tasks. Though the model presented in this paper is trained on English-language tweets, the method presented can be used to learn tweet embeddings for different languages.

In recent years, the micro-blogging site Twitter has become a major social media platform with hundreds of millions of users. The short (140 character limit), noisy and idiosyncratic nature of tweets make standard information retrieval and data mining methods ill-suited to Twitter. Consequently, there has been an ever growing body of IR and data mining literature focusing on Twitter. However, most of these works employ extensive feature engineering to create task-specific, hand-crafted features. This is time consuming and inefficient as new features need to be engineered for every task.
In this paper, we present Tweet2Vec, a method for generating general-purpose vector representation of tweets that can be used for any classification task. Tweet2Vec removes the need for expansive feature engineering and can be used to train any standard off-the-shelf classifier (e.g., logistic regression, svm, etc). Tweet2Vec uses a CNN-LSTM encoder-decoder model that operates at the character level to learn and generate vector representation of tweets. Our method is especially useful for natural language processing tasks on Twitter where it is particularly difficult to engineer features, such as speech-act classification and stance detection (as shown in our previous works on these topics BIBREF0 , BIBREF1 ).
There has been several works on generating embeddings for words, most famously Word2Vec by Mikolov et al. BIBREF2 ). There has also been a number of different works that use encoder-decoder models based on long short-term memory (LSTM) BIBREF3 , and gated recurrent neural networks (GRU) BIBREF4 . These methods have been used mostly in the context of machine translation. The encoder maps the sentence from the source language to a vector representation, while the decoder conditions on this encoded vector for translating it to the target language. Perhaps the work most related to ours is the work of Le and Mikolov le2014distributed, where they extended the Word2Vec model to generate representations for sentences (called ParagraphVec). However, these models all function at the word level, making them ill-suited to the extremely noisy and idiosyncratic nature of tweets. Our character-level model, on the other hand, can better deal with the noise and idiosyncrasies in tweets. We plan to make our model and the data used to train it publicly available to be used by other researchers that work with tweets.
In this section, we describe the CNN-LSTM encoder-decoder model that operates at the character level and generates vector representation of tweets. The encoder consists of convolutional layers to extract features from the characters and an LSTM layer to encode the sequence of features to a vector representation, while the decoder consists of two LSTM layers which predict the character at each time step from the output of encoder.
Character-level CNN (CharCNN) is a slight variant of the deep character-level convolutional neural network introduced by Zhang et al BIBREF5 . In this model, we perform temporal convolutional and temporal max-pooling operations, which computes one-dimensional convolution and pooling functions, respectively, between input and output. Given a discrete input function INLINEFORM0 , a discrete kernel function INLINEFORM1 and stride INLINEFORM2 , the convolution INLINEFORM3 between INLINEFORM4 and INLINEFORM5 and pooling operation INLINEFORM6 of INLINEFORM7 is calculated as: DISPLAYFORM0 DISPLAYFORM1 
where INLINEFORM0 is an offset constant.
We adapted this model, which employs temporal convolution and pooling operations, for tweets. The character set includes the English alphabets, numbers, special characters and unknown character. There are 70 characters in total, given below:
abcdefghijklmnopqrstuvwxyz0123456789
-,;.!?:'"/\|_#$%&^*~`+-=<>()[]{}
Each character in the tweets can be encoded using one-hot vector INLINEFORM0 . Hence, the tweets are represented as a binary matrix INLINEFORM1 with padding wherever necessary, where 150 is the maximum number of characters in a tweet (140 tweet characters and padding) and 70 is the size of the character set.
Each tweet, in the form of a matrix, is now fed into a deep model consisting of four 1-d convolutional layers. A convolution operation employs a filter INLINEFORM0 , to extract n-gram character feature from a sliding window of INLINEFORM1 characters at the first layer and learns abstract textual features in the subsequent layers. The convolution in the first layer operates on sliding windows of character (size INLINEFORM2 ), and the convolutions in deeper layers are defined in a similar way. Generally, for tweet INLINEFORM3 , a feature INLINEFORM4 at layer INLINEFORM5 is generated by: DISPLAYFORM0 
where INLINEFORM0 , INLINEFORM1 is the bias at layer INLINEFORM2 and INLINEFORM3 is a rectified linear unit.
This filter INLINEFORM0 is applied across all possible windows of characters in the tweet to produce a feature map. The output of the convolutional layer is followed by a 1-d max-overtime pooling operation BIBREF6 over the feature map and selects the maximum value as the prominent feature from the current filter. In this way, we apply INLINEFORM1 filters at each layer. Pooling size may vary at each layer (given by INLINEFORM2 at layer INLINEFORM3 ). The pooling operation shrinks the size of the feature representation and filters out trivial features like unnecessary combination of characters. The window length INLINEFORM4 , number of filters INLINEFORM5 , pooling size INLINEFORM6 at each layer are given in Table TABREF6 .
.93
.91
We define INLINEFORM0 to denote the character-level CNN operation on input tweet matrix INLINEFORM1 . The output from the last convolutional layer of CharCNN(T) (size: INLINEFORM2 ) is subsequently given as input to the LSTM layer. Since LSTM works on sequences (explained in Section SECREF8 and SECREF11 ), pooling operation is restricted to the first two layers of the model (as shown in Table TABREF6 ).
In this section we briefly describe the LSTM model BIBREF7 . Given an input sequence INLINEFORM0 ( INLINEFORM1 ), LSTM computes the hidden vector sequence INLINEFORM2 ( INLINEFORM3 ) and and output vector sequence INLINEFORM4 ( INLINEFORM5 ). At each time step, the output of the module is controlled by a set of gates as a function of the previous hidden state INLINEFORM6 and the input at the current time step INLINEFORM7 , the forget gate INLINEFORM8 , the input gate INLINEFORM9 , and the output gate INLINEFORM10 . These gates collectively decide the transitions of the current memory cell INLINEFORM11 and the current hidden state INLINEFORM12 . The LSTM transition functions are defined as follows: DISPLAYFORM0 
Here, INLINEFORM0 is the INLINEFORM1 function that has an output in [0, 1], INLINEFORM2 denotes the hyperbolic tangent function that has an output in INLINEFORM3 , and INLINEFORM4 denotes the component-wise multiplication. The extent to which the information in the old memory cell is discarded is controlled by INLINEFORM5 , while INLINEFORM6 controls the extent to which new information is stored in the current memory cell, and INLINEFORM7 is the output based on the memory cell INLINEFORM8 . LSTM is explicitly designed for learning long-term dependencies, and therefore we choose LSTM after the convolution layer to learn dependencies in the sequence of extracted features. In sequence-to-sequence generation tasks, an LSTM defines a distribution over outputs and sequentially predicts tokens using a softmax function. DISPLAYFORM0 
where INLINEFORM0 is the activation function. For simplicity, we define INLINEFORM1 to denote the LSTM operation on input INLINEFORM2 at time-step INLINEFORM3 and the previous hidden state INLINEFORM4 .
The CNN-LSTM encoder-decoder model draws on the intuition that the sequence of features (e.g. character and word n-grams) extracted from CNN can be encoded into a vector representation using LSTM that can embed the meaning of the whole tweet. Figure FIGREF7 illustrates the complete encoder-decoder model. The input and output to the model are the tweet represented as a matrix where each row is the one-hot vector representation of the characters. The procedure for encoding and decoding is explained in the following section.
Given a tweet in the matrix form T (size: INLINEFORM0 ), the CNN (Section SECREF2 ) extracts the features from the character representation. The one-dimensional convolution involves a filter vector sliding over a sequence and detecting features at different positions. The new successive higher-order window representations then are fed into LSTM (Section SECREF8 ). Since LSTM extracts representation from sequence input, we will not apply pooling after convolution at the higher layers of Character-level CNN model. The encoding procedure can be summarized as: DISPLAYFORM0 DISPLAYFORM1 
where INLINEFORM0 is an extracted feature matrix where each row can be considered as a time-step for the LSTM and INLINEFORM1 is the hidden representation at time-step INLINEFORM2 . LSTM operates on each row of the INLINEFORM3 along with the hidden vectors from previous time-step to produce embedding for the subsequent time-steps. The vector output at the final time-step, INLINEFORM4 , is used to represent the entire tweet. In our case, the size of the INLINEFORM5 is 256.
The decoder operates on the encoded representation with two layers of LSTMs. In the initial time-step, the end-to-end output from the encoding procedure is used as the original input into first LSTM layer. The last LSTM decoder generates each character, INLINEFORM0 , sequentially and combines it with previously generated hidden vectors of size 128, INLINEFORM1 , for the next time-step prediction. The prediction of character at each time step is given by: DISPLAYFORM0 
where INLINEFORM0 refers to the character at time-step INLINEFORM1 , INLINEFORM2 represents the one-hot vector of the character at time-step INLINEFORM3 . The result from the softmax is a decoded tweet matrix INLINEFORM4 , which is eventually compared with the actual tweet or a synonym-replaced version of the tweet (explained in Section SECREF3 ) for learning the parameters of the model.
We trained the CNN-LSTM encoder-decoder model on 3 million randomly selected English-language tweets populated using data augmentation techniques, which are useful for controlling generalization error for deep learning models. Data augmentation, in our context, refers to replicating tweet and replacing some of the words in the replicated tweets with their synonyms. These synonyms are obtained from WordNet BIBREF8 which contains words grouped together on the basis of their meanings. This involves selection of replaceable words (example of non-replaceable words are stopwords, user names, hash tags, etc) from the tweet and the number of words INLINEFORM0 to be replaced. The probability of the number, INLINEFORM1 , is given by a geometric distribution with parameter INLINEFORM2 in which INLINEFORM3 . Words generally have several synonyms, thus the synonym index INLINEFORM4 , of a given word is also determined by another geometric distribution in which INLINEFORM5 . In our encoder-decoder model, we decode the encoded representation to the actual tweet or a synonym-replaced version of the tweet from the augmented data. We used INLINEFORM6 , INLINEFORM7 for our training. We also make sure that the POS tags of the replaced words are not completely different from the actual words. For regularization, we apply a dropout mechanism after the penultimate layer. This prevents co-adaptation of hidden units by randomly setting a proportion INLINEFORM8 of the hidden units to zero (for our case, we set INLINEFORM9 ).
To learn the model parameters, we minimize the cross-entropy loss as the training objective using the Adam Optimization algorithm BIBREF9 . It is given by DISPLAYFORM0 
where p is the true distribution (one-hot vector representing characters in the tweet) and q is the output of the softmax. This, in turn, corresponds to computing the negative log-probability of the true class.
We evaluated our model using two classification tasks: Tweet semantic relatedness and Tweet sentiment classification.
The first evaluation is based on the SemEval 2015-Task 1: Paraphrase and Semantic Similarity in Twitter BIBREF10 . Given a pair of tweets, the goal is to predict their semantic equivalence (i.e., if they express the same or very similar meaning), through a binary yes/no judgement. The dataset provided for this task contains 18K tweet pairs for training and 1K pairs for testing, with INLINEFORM0 of these pairs being paraphrases, and INLINEFORM1 non-paraphrases. We first extract the vector representation of all the tweets in the dataset using our Tweet2Vec model. We use two features to represent a tweet pair. Given two tweet vectors INLINEFORM2 and INLINEFORM3 , we compute their element-wise product INLINEFORM4 and their absolute difference INLINEFORM5 and concatenate them together (Similar to BIBREF11 ). We then train a logistic regression model on these features using the dataset. Cross-validation is used for tuning the threshold for classification. In contrast to our model, most of the methods used for this task were largely based on extensive use of feature engineering, or a combination of feature engineering with semantic spaces. Table 2 shows the performance of our model compared to the top four models in the SemEval 2015 competition, and also a model that was trained using ParagraphVec. Our model (Tweet2Vec) outperforms all these models, without resorting to extensive task-specific feature engineering.
.93
.91
The second evaluation is based on the SemEval 2015-Task 10B: Twitter Message Polarity Classification BIBREF12 . Given a tweet, the task is to classify it as either positive, negative or neutral in sentiment. The size of the training and test sets were 9,520 tweets and 2,380 tweets respectively ( INLINEFORM0 positive, INLINEFORM1 negative, and INLINEFORM2 neutral).
As with the last task, we first extract the vector representation of all the tweets in the dataset using Tweet2Vec and use that to train a logistic regression classifier using the vector representations. Even though there are three classes, the SemEval task is a binary task. The performance is measured as the average F1-score of the positive and the negative class. Table 3 shows the performance of our model compared to the top four models in the SemEval 2015 competition (note that only the F1-score is reported by SemEval for this task) and ParagraphVec. Our model outperforms all these models, again without resorting to any feature engineering.
.93
.91
In this paper, we presented Tweet2Vec, a novel method for generating general-purpose vector representation of tweets, using a character-level CNN-LSTM encoder-decoder architecture. To the best of our knowledge, ours is the first attempt at learning and applying character-level tweet embeddings. Our character-level model can deal with the noisy and peculiar nature of tweets better than methods that generate embeddings at the word level. Our model is also robust to synonyms with the help of our data augmentation technique using WordNet.
The vector representations generated by our model are generic, and thus can be applied to tasks of different nature. We evaluated our model using two different SemEval 2015 tasks: Twitter semantic relatedness, and sentiment classification. Simple, off-the-shelf logistic regression classifiers trained using the vector representations generated by our model outperformed the top-performing methods for both tasks, without the need for any extensive feature engineering. This was despite the fact that due to resource limitations, our Tweet2Vec model was trained on a relatively small set (3 million tweets). Also, our method outperformed ParagraphVec, which is an extension of Word2Vec to handle sentences. This is a small but noteworthy illustration of why our tweet embeddings are best-suited to deal with the noise and idiosyncrasies of tweets.
For future work, we plan to extend the method to include: 1) Augmentation of data through reordering the words in the tweets to make the model robust to word-order, 2) Exploiting attention mechanism BIBREF13 in our model to improve alignment of words in tweets during decoding, which could improve the overall performance.

Figure and table captions:
Table 1: Layer Parameters of CharCNN
Table 2: Results of the paraphrase and semantic similarity in Twitter task.
Table 3: Results of Twitter sentiment classification task.

Questions about this paper:
by how much did their system outperform previous tasks?
what are the previous state of the art for sentiment categorization?
what are the previous state of the art for tweet semantic similarity?

Answers:
On paraphrase and semantic similarity proposed model has F1 score of 0.677 compared to best previous model result of 0.674, while on sentiment classification it has 0.656 compared to 0.648 of best previous result.
INESC-ID, lsislif, unitn and Webis.
nnfeats, ikr, linearsvm and svckernel.