Interpreting the Syntactic and Social Elements of the Tweet Representations via Elementary Property Prediction Tasks

Abstract: Research in social media analysis is experiencing a recent surge with a large number of works applying representation learning models to solve high-level syntactico-semantic tasks such as sentiment analysis, semantic textual similarity computation, hashtag prediction and so on. Although the performance of the representation learning models are better than the traditional baselines for the tasks, little is known about the core properties of a tweet encoded within the representations. Understanding these core properties would empower us in making generalizable conclusions about the quality of representations. Our work presented here constitutes the first step in opening the black-box of vector embedding for social media posts, with emphasis on tweets in particular. In order to understand the core properties encoded in a tweet representation, we evaluate the representations to estimate the extent to which it can model each of those properties such as tweet length, presence of words, hashtags, mentions, capitalization, and so on. This is done with the help of multiple classifiers which take the representation as input. Essentially, each classifier evaluates one of the syntactic or social properties which are arguably salient for a tweet. This is also the first holistic study on extensively analysing the ability to encode these properties for a wide variety of tweet representation models including the traditional unsupervised methods (BOW, LDA), unsupervised representation learning methods (Siamese CBOW, Tweet2Vec) as well as supervised methods (CNN, BLSTM).

Research in social media analysis is recently seeing a surge in the number of research works applying representation learning models to solve high-level syntactico-semantic tasks such as sentiment analysis [1], semantic textual similarity computation [2], hashtag prediction [3] and so on. Though the performance of the representation learning models are better than the traditional models for all the tasks, little is known about the core properties of a tweet encoded within the representations. In a recent work, Hill et al. [4] perform a comparison of different sentence representation models by evaluating them for different high-level semantic tasks such as paraphrase identification, sentiment classification, question answering, document retrieval and so on. This type of coarse-grained analysis is opaque as it does not clearly reveal the kind of information encoded by the representations. Our work presented here constitutes the first step in opening the black-box of vector embeddings for social media posts, particularly tweets.
Essentially we ask the following question: “What are the core properties encoded in the given tweet representation?”. We explicitly group the set of these properties into two categories: syntactic and social. Syntactic category includes properties such as tweet length, the order of words in it, the words themselves, slang words, hashtags and named entities in the tweet. On the other hand, social properties consist of `is reply', and `reply time'. We investigate the degree to which the tweet representations encode these properties. We assume that if we cannot train a classifier to predict a property based on its tweet representation, then this property is not encoded in this representation. For example, the model which preserves the tweet length should perform well in predicting the length given the representation generated from the model. Though these elementary property prediction tasks are not directly related to any downstream application, knowing that the model is good at modeling a particular property (e.g., the social properties) indicates that it could excel in correlated applications (e.g., user profiling task). In this work we perform an extensive evaluation of 9 unsupervised and 4 supervised tweet representation models, using 8 different properties. The most relevant work is that of Adi et al. [5], which investigates three sentence properties in comparing unsupervised sentence representation models such as average of words vectors and LSTM auto-encoders. We differ from their work in two ways: (1) While they focus on sentences, we focus on social media posts which opens up the challenge of considering multiple salient properties such as hashtags, named entities, conversations and so on. (2) While they work with only unsupervised representation-learning models, we investigate the traditional unsupervised methods (BOW, LDA), unsupervised representation learning methods (Siamese CBOW, Tweet2Vec), as well as supervised methods (CNN, BLSTM).
Our main contributions are summarized below.
The paper is organized as follows. Sections 2 and 3 discuss the set of proposed elementary property prediction tasks and the models considered for this study respectively. Section 4 and 5 presents the experiment setup and result analysis respectively. We conclude the work with a brief summary in Section 5.
In this section we list down the set of proposed elementary property prediction tasks to test the characteristics of a tweet embedding. Table TABREF4 explains all the tasks considered in this study. Note that we use a neural network to build the elementary property prediction task classifier which has the following two layers in order: the representation layer and the softmax layer on top whose size varies according to the specific task. When there are more than one input for a task, we concatenate embeddings for each input.
[1]https://noisy-text.github.io/norm-shared-task.html
In this section we list down the set of models considered in the study.
Bag Of Words (BOW) [17] - This simple representation captures the TF-IDF value of an n-gram. We pick top 50K n-grams, with the value of `n' going up to 5.
Latent Dirichlet Allocation (LDA) [18] - We use the topic distribution resulting by running LDA with number of topics as 200, as tweet representation.
Bag Of Means (BOM) - We take the average of the word embeddings obtained by running the Glove [12] model on 2 billion tweets with embedding size as 200.
Deep Structured Semantic Models (DSSM) [9] - This is a deep encoder trained to represent query and document in common space, for document ranking. We use the publicly available pre-trained encoder to encode the tweets.
Convolutional DSSM (CDSSM) [10] - This is the convolutional variant of DSSM.
Paragraph2Vec (PV) [13] - This model based on Word2Vec [15] learns embedding for a document which is good in predicting the words within it. We use the BOW variant with embedding size and window size of 200 and 10 respectively.
Skip-Thought Vectors (STV) [6] - This is a GRU [16] encoder trained to predict adjacent sentences in a books corpus. We use the recommended combine-skip (4800-dimensional) vectors from the publicly available encoder.
Tweet2Vec (T2V) [3] - This is a character composition model working directly on the character sequences to predict the user-annotated hashtags in a tweet. We use publicly available encoder, which was trained on 2 million tweets.
Siamese CBOW (SCBOW) [2] - This model uses averaging of word vectors to represent a sentence, and the objective and data used here is the same as that for STV. Note that this is different from BOW because the word vectors here are optimized for sentence representation.
Convolutional Neural Network (CNN) - This is a simple CNN proposed in [7].
Long Short Term Memory Network (LSTM) [14] - This is a vanilla LSTM based recurrent model, applied from start to the end of a tweet, and the last hidden vector is used as tweet representation.
Bi-directional LSTM (BLSTM) [14] - This extends LSTM by using two LSTM networks, processing a tweet left-to-right and right-to-left respectively. Tweet is represented by concatenating the last hidden vector of both the LSTMs.
FastText (FT) [8] - This is a simple architecture which averages the n-gram vectors to represent a tweet, followed by the softmax in the final layer. This simple model has been shown to be effective for the text classification task.
In this section we perform an extensive evaluation of all the models in an attempt to find the significance of different representation models. Essentially we study every model (with optimal settings reported in the corresponding paper) with respect to the following three perspectives.
Fine-grained analysis of various supervised and unsupervised models discussed in Section SECREF3 , across various dimensions discussed in Section SECREF4 , is presented in Table TABREF30 . The codes used to conduct our experiments are publicly accessible at: https://github.com/ganeshjawahar/fine-tweet/.
We summarize the results of property prediction tasks in Table TABREF31 . Length prediction turns out to be a difficult task for most of the models. Models which rely on the recurrent architectures such as LSTM, STV, T2V have sufficient capacity to perform well in modeling the tweet length. Also BLSTM is the best in modeling slang words. BLSTM outperforms the LSTM variant in all the tasks except `Content', which signifies the power of using the information flowing from both the directions of the tweet. T2V which is expected to perform well in this task because of its ability to work at a more fine level (i.e., characters) performs the worst. In fact T2V does not outperform other models in any task, which could be mainly due to the fact that the hashtags which are used for supervision in learning tweet representations reduces the generalization capability of the tweets beyond hashtag prediction. Prediction tasks such as `Content' and `Hashtag' seem to be less difficult as all the models perform nearly optimal for them. The superior performance of all the models for the `Content' task in particular is unlike the relatively lower performance reported for in [5], mainly because of the short length of the tweets. The most surprising result is when the BOM model turned out to be the best in `Word Order' task, as the model by nature loses the word order. This might be due to the correlation between word order patterns and the occurrences of specific words. BOM has also proven to perform well for identifying the named entities in the tweet.
STV is good for most of the social tasks. We believe the main reason for STV's performance is two-fold: (a) the inter-sentential features extracted from STV's encoder by the prediction of the surrounding sentences in the books corpus contains rich social elements that are vital for social tasks (e.g., user profiling), (b) the recurrent structure in both the encoder and decoder persists useful information in the memory nicely. The second claim is further substantiated by observing the poor performance of SCBOW whose objective is also similar to STV, but with a simpler architecture (i.e., word vector averaging). In future it would be interesting to create such a model for Twitter conversations or chronologically ordered topical tweets so as to directly capture the latent social features from Twitter.
This setup captures the behavior of the model with the increase in the context size, which is defined in terms of number of words. For `Word Order' task, we see the performance of all the models to be negatively correlated with the tweet length, which is expected. On the other hand, there is no correlation between the tweet length and the performance of all the models for the tasks such as `Slang Words', `Content', `Hashtag', `NE', and `Is Reply'. For social tasks such as `Is Reply' and `Reply Time', we see a positive correlation between the tweet length and the performance of all the models. This finding is intuitive in social media analysis where additional context is mostly helpful in modeling the social behavior.
This test essentially captures the importance of “natural word order”. We found that LDA was invariant to the reordering of the words in the tweet for most of the tasks. This result is not surprising as LDA considers each word in the tweet independently. CNN, LSTM and BLSTM rely on the word order significantly to perform well for most of the prediction tasks.
This work proposed a set of elementary property prediction tasks to understand different tweet representations in an application independent, fine-grained fashion. The open nature of social media not only poses a plethora of opportunities to understand the basic characteristics of the posts, but also helped us draw novel insights about different representation models. We observed that among supervised models, CNN, LSTM and BLSTM encapsulates most of the syntactic and social properties with a great accuracy, while BOW, DSSM, STV and T2V does that among the unsupervised models. Tweet length affects the task prediction accuracies, but we found that all models behave similarly under variation in tweet length. Finally while LDA is insensitive to input word order, CNN, LSTM and BLSTM are extremely sensitive to word order.
[1] Tang, D., Wei, F., Qin, B., Yang, N., Liu, T., & Zhou, M.: Sentiment Embeddings with Applications to Sentiment Analysis. In: TKDE. (2016) 28(2) 496-509
[2] Kenter, T., Borisov, A., & de Rijke, M.: Siamese CBOW: Optimizing Word Embeddings for Sentence Representations. In: ACL. (2016) 941-951
[3] Dhingra, B., Zhou, Z., Fitzpatrick, D., Muehl, M., & Cohen, W. W.: Tweet2Vec: Character-Based Distributed Representations for Social Media. In: ACL. (2016)
[4] Hill, F., Cho, K.,& Korhonen, A.: Learning distributed representations of sentences from unlabelled data. In: NAACL. (2016)
[5] Adi, Y., Kermany, E., Belinkov, Y., Lavi, O., & Goldberg, Y.: Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks. arXiv preprint arXiv:1608.04207. (2016)
[6] Kiros, R., Zhu, Y., Salakhutdinov, R. R., Zemel, R., Urtasun, R., Torralba, A., & Fidler, S.: Skip-thought vectors. In: NIPS. (2015) 3294-3302
[7] Kim, Y.: Convolutional neural networks for sentence classification. In: EMNLP. (2014)
[8] Joulin, A., Grave, E., Bojanowski, P., & Mikolov, T.: Bag of Tricks for Efficient Text Classification. arXiv preprint arXiv:1607.01759. (2016)
[9] Huang, P. S., He, X., Gao, J., Deng, L., Acero, A., & Heck, L.: Learning deep structured semantic models for web search using clickthrough data. In: CIKM. (2013)
[10] Shen, Y., He, X., Gao, J., Deng, L., & Mesnil, G.: A latent semantic model with convolutional-pooling structure for information retrieval. In: CIKM. (2014)
[11] Ritter, A., Clark, S., Mausam, & Etzioni, O.: Named entity recognition in tweets: an experimental study. In: EMNLP. (2011) 1524-1534
[12] Pennington, J., Socher, R., & Manning, C. D.: Glove: Global Vectors for Word Representation. In: EMNLP. (2014) 1532-43
[13] Le, Q. V., & Mikolov, T.: Distributed Representations of Sentences and Documents. In: ICML. (2014) 1188-1196
[14] Graves, A., Mohamed, A. R., & Hinton, G.: Speech recognition with deep recurrent neural networks. In: ICASSP. (2013) 6645-6649
[15] Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., & Dean, J.: Distributed representations of words and phrases and their compositionality. In: NIPS. (2013) 3111-3119
[16] Cho, K., Van Merriënboer, B., Bahdanau, D., & Bengio, Y.: On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv:1409.1259. (2014)
[17] Harris, Z. S.: Distributional structure. In: Word. (1954) 146-162
[18] Blei, D. M., Ng, A. Y., & Jordan, M. I.: Latent dirichlet allocation. In: JMLR. (2003)

Figure and table captions:
Table 1: Details of the Set of Proposed Elementary Property Prediction Tasks
Table 2: Fine-grained analysis of supervised/unsupervised models
Table 3: Elementary Property Prediction Task F1-Score (%) - Performance Comparison

Questions about this paper:
What conclusions do the authors draw from their experiments?
In what way does each classifier evaluate one of the syntactic or social properties which are salient for a tweet?

Answers:
Supervised models CNN, LSTM and BLSTM and unsupervised models BOW, DSSM, STV and T2V can  encapsulate most of the syntactic and social properties. Tweet length affects the task prediction accuracies for all models. LDA is insensitive to input word order, but, CNN, LSTM
and BLSTM are not.
Through 8 different property prediction tasks