Enhanced Twitter Sentiment Classification Using Contextual Information

Abstract: The rise in popularity and ubiquity of Twitter has made sentiment analysis of tweets an important and well-covered area of research. However, the 140 character limit imposed on tweets makes it hard to use standard linguistic methods for sentiment classification. On the other hand, what tweets lack in structure they make up with sheer volume and rich metadata. This metadata includes geolocation, temporal and author information. We hypothesize that sentiment is dependent on all these contextual factors. Different locations, times and authors have different emotional valences. In this paper, we explored this hypothesis by utilizing distant supervision to collect millions of labelled tweets from different locations, times and authors. We used this data to analyse the variation of tweet sentiments across different authors, times and locations. Once we explored and understood the relationship between these variables and sentiment, we used a Bayesian approach to combine these variables with more standard linguistic features such as n-grams to create a Twitter sentiment classifier. This combined classifier outperforms the purely linguistic classifier, showing that integrating the rich contextual information available on Twitter into sentiment classification is a promising direction of research.

Twitter is a micro-blogging platform and a social network where users can publish and exchange short messages of up to 140 characters long (also known as tweets). Twitter has seen a great rise in popularity in recent years because of its availability and ease-of-use. This rise in popularity and the public nature of Twitter (less than 10% of Twitter accounts are private BIBREF0 ) have made it an important tool for studying the behaviour and attitude of people.
One area of research that has attracted great attention in the last few years is that of tweet sentiment classification. Through sentiment classification and analysis, one can get a picture of people's attitudes about particular topics on Twitter. This can be used for measuring people's attitudes towards brands, political candidates, and social issues.
There have been several works that do sentiment classification on Twitter using standard sentiment classification techniques, with variations of n-gram and bag of words being the most common. There have been attempts at using more advanced syntactic features as is done in sentiment classification for other domains BIBREF1 , BIBREF2 , however the 140 character limit imposed on tweets makes this hard to do as each article in the Twitter training set consists of sentences of no more than several words, many of them with irregular form BIBREF3 .
On the other hand, what tweets lack in structure they make up with sheer volume and rich metadata. This metadata includes geolocation, temporal and author information. We hypothesize that sentiment is dependent on all these contextual factors. Different locations, times and authors have different emotional valences. For instance, people are generally happier on weekends and certain hours of the day, more depressed at the end of summer holidays, and happier in certain states in the United States. Moreover, people have different baseline emotional valences from one another. These claims are supported for example by the annual Gallup poll that ranks states from most happy to least happy BIBREF4 , or the work by Csikszentmihalyi and Hunter BIBREF5 that showed reported happiness varies significantly by day of week and time of day. We believe these factors manifest themselves in sentiments expressed in tweets and that by accounting for these factors, we can improve sentiment classification on Twitter.
In this work, we explored this hypothesis by utilizing distant supervision BIBREF6 to collect millions of labelled tweets from different locations (within the USA), times of day, days of the week, months and authors. We used this data to analyse the variation of tweet sentiments across the aforementioned categories. We then used a Bayesian approach to incorporate the relationship between these factors and tweet sentiments into standard n-gram based Twitter sentiment classification.
This paper is structured as follows. In the next sections we will review related work on sentiment classification, followed by a detailed explanation of our approach and our data collection, annotation and processing efforts. After that, we describe our baseline n-gram sentiment classifier model, followed by the explanation of how the baseline model is extended to incorporate contextual information. Next, we describe our analysis of the variation of sentiment within each of the contextual categories. We then evaluate our models and finally summarize our findings and contributions and discuss possible paths for future work.
Sentiment analysis and classification of text is a problem that has been well studied across many different domains, such as blogs, movie reviews, and product reviews (e.g., BIBREF7 , BIBREF8 , BIBREF9 ). There is also extensive work on sentiment analysis for Twitter. Most of the work on Twitter sentiment classification either focuses on different machine learning techniques (e.g., BIBREF10 , BIBREF11 ), novel features (e.g., BIBREF12 , BIBREF13 , BIBREF3 ), new data collection and labelling techniques (e.g., BIBREF6 ) or the application of sentiment classification to analyse the attitude of people about certain topics on Twitter (e.g., BIBREF14 , BIBREF15 ). These are just some examples of the extensive research already done on Twitter sentiment classification and analysis.
There has also been previous work on measuring the happiness of people in different contexts (location, time, etc). This has been done mostly through traditional land-line polling BIBREF5 , BIBREF4 , with Gallup's annual happiness index being a prime example BIBREF4 . More recently, some have utilized Twitter to measure people's mood and happiness and have found Twitter to be a generally good measure of the public's overall happiness, well-being and mood. For example, Bollen et al. BIBREF15 used Twitter to measure the daily mood of the public and compare that to the record of social, political, cultural and economic events in the real world. They found that these events have a significant effect on the public mood as measured through Twitter. Another example would be the work of Mitchell et al. BIBREF16 , in which they estimated the happiness levels of different states and cities in the USA using Twitter and found statistically significant correlations between happiness level and the demographic characteristics (such as obesity rates and education levels) of those regions. Finally, improving natural language processing by incorporating contextual information has been successfully attempted before BIBREF17 , BIBREF18 ; but as far as we are aware, this has not been attempted for sentiment classification.
In this work, we combined the sentiment analysis of different authors, locations, times and dates as measured through labelled Twitter data with standard word-based sentiment classification methods to create a context-dependent sentiment classifier. As far as we can tell, there has not been significant previous work on Twitter sentiment classification that has achieved this.
The main hypothesis behind this work is that the average sentiment of messages on Twitter is different in different contexts. Specifically, tweets in different spatial, temporal and authorial contexts have on average different sentiments. Basically, these factors (many of which are environmental) have an affect on the emotional states of people which in turn have an effect on the sentiments people express on Twitter and elsewhere. In this paper, we used this contextual information to better predict the sentiment of tweets.
Luckily, tweets are tagged with very rich metadata, including location, timestamp, and author information. By analysing labelled data collected from these different contexts, we calculated prior probabilities of negative and positive sentiments for each of the contextual categories shown below:
This means that for every item in each of these categories, we calculated a probability of sentiment being positive or negative based on historical tweets. For example, if seven out of ten historical tweets made on Friday were positive then the prior probability of a sentiment being positive for tweets sent out on Friday is INLINEFORM0 and the prior probability of a sentiment being negative is INLINEFORM1 . We then trained a Bayesian sentiment classifier using a combination of these prior probabilities and standard n-gram models. The model is described in great detail in the "Baseline Model" and "Contextual Model" sections of this paper.
In order to do a comprehensive analysis of sentiment of tweets across aforementioned contextual categories, a large amount of labelled data was required. We needed thousands of tweets for every item in each of the categories (e.g. thousands of tweets per hour of day, or state in the US). Therefore, creating a corpus using human-annotated data would have been impractical. Instead, we turned to distant supervision techniques to obtain our corpus. Distant supervision allows us to have noisy but large amounts of annotated tweets.
There are different methods of obtaining labelled data using distant supervision BIBREF1 , BIBREF6 , BIBREF19 , BIBREF12 . We used emoticons to label tweets as positive or negative, an approach that was introduced by Read BIBREF1 and used in multiple works BIBREF6 , BIBREF12 . We collected millions of English-language tweets from different times, dates, authors and US states. We used a total of six emoticons, three mapping to positive and three mapping to negative sentiment (table TABREF7 ). We identified more than 120 positive and negative ASCII emoticons and unicode emojis, but we decided to only use the six most common emoticons in order to avoid possible selection biases. For example, people who use obscure emoticons and emojis might have a different base sentiment from those who do not. Using the six most commonly used emoticons limits this bias. Since there are no "neutral" emoticons, our dataset is limited to tweets with positive or negative sentiments. Accordingly, in this work we are only concerned with analysing and classifying the polarity of tweets (negative vs. positive) and not their subjectivity (neutral vs. non-neutral). Below we will explain our data collection and corpus in greater detail.
We collected two datasets, one massive and labelled through distant supervision, the other small and labelled by humans. The massive dataset was used to calculate the prior probabilities for each of our contextual categories. Both datasets were used to train and test our sentiment classifier. The human-labelled dataset was used as a sanity check to make sure the dataset labelled using the emoticons classifier was not too noisy and that the human and emoticon labels matched for a majority of tweets.
We collected a total of 18 million, geo-tagged, English-language tweets over three years, from January 1st, 2012 to January 1st, 2015, evenly divided across all 36 months, using Historical PowerTrack for Twitter provided by GNIP. We created geolocation bounding boxes for each of the 50 states which were used to collect our dataset. All 18 million tweets originated from one of the 50 states and are tagged as such. Moreover, all tweets contained one of the six emoticons in Table TABREF7 and were labelled as either positive or negative based on the emoticon. Out of the 18 million tweets, INLINEFORM0 million ( INLINEFORM1 ) were labelled as positive and INLINEFORM2 million ( INLINEFORM3 ) were labelled as negative. The 18 million tweets came from INLINEFORM4 distinct users.
We randomly selected 3000 tweets from our large dataset and had all their emoticons stripped. We then had these tweets labelled as positive or negative by three human annotators. We measured the inter-annotator agreement using Fleiss' kappa, which calculates the degree of agreement in classification over that which would be expected by chance BIBREF20 . The kappa score for the three annotators was INLINEFORM0 , which means that there were disagreements in sentiment for a small portion of the tweets. However, the number of tweets that were labelled the same by at least two of the three human annotator was 2908 out of of the 3000 tweets ( INLINEFORM1 ). Of these 2908 tweets, INLINEFORM2 were labelled as positive and INLINEFORM3 as negative.
We then measured the agreement between the human labels and emoticon-based labels, using only tweets that were labelled the same by at least two of the three human annotators (the majority label was used as the label for the tweet). Table TABREF13 shows the confusion matrix between human and emoticon-based annotations. As you can see, INLINEFORM0 of all labels matched ( INLINEFORM1 ).
These results are very promising and show that using emoticon-based distant supervision to label the sentiment of tweets is an acceptable method. Though there is some noise introduced to the dataset (as evidenced by the INLINEFORM0 of tweets whose human labels did not match their emoticon labels), the sheer volume of labelled data that this method makes accessible, far outweighs the relatively small amount of noise introduced.
Since the data is labelled using emoticons, we stripped all emoticons from the training data. This ensures that emoticons are not used as a feature in our sentiment classifier. A large portion of tweets contain links to other websites. These links are mostly not meaningful semantically and thus can not help in sentiment classification. Therefore, all links in tweets were replaced with the token "URL". Similarly, all mentions of usernames (which are denoted by the @ symbol) were replaced with the token "USERNAME", since they also can not help in sentiment classification. Tweets also contain very informal language and as such, characters in words are often repeated for emphasis (e.g., the word good is used with an arbitrary number of o's in many tweets). Any character that was repeated more than two times was removed (e.g., goooood was replaced with good). Finally, all words in the tweets were stemmed using Porter Stemming BIBREF21 .
For our baseline sentiment classification model, we used our massive dataset to train a negative and positive n-gram language model from the negative and positive tweets.
As our baseline model, we built purely linguistic bigram models in Python, utilizing some components from NLTK BIBREF22 . These models used a vocabulary that was filtered to remove words occurring 5 or fewer times. Probability distributions were calculated using Kneser-Ney smoothing BIBREF23 . In addition to Kneser-Ney smoothing, the bigram models also used “backoff” smoothing BIBREF24 , in which an n-gram model falls back on an INLINEFORM0 -gram model for words that were unobserved in the n-gram context.
In order to classify the sentiment of a new tweet, its probability of fit is calculated using both the negative and positive bigram models. Equation EQREF15 below shows our models through a Bayesian lens. DISPLAYFORM0 
Here INLINEFORM0 can be INLINEFORM1 or INLINEFORM2 , corresponding to the hypothesis that the sentiment of the tweet is positive or negative respectively. INLINEFORM3 is the sequence of INLINEFORM4 words, written as INLINEFORM5 , that make up the tweet. INLINEFORM6 is not dependent on the hypothesis, and can thus be ignored. Since we are using a bigram model, Equation EQREF15 can be written as: DISPLAYFORM0 
This is our purely linguistic baseline model.
The Bayesian approach allows us to easily integrate the contextual information into our models. INLINEFORM0 in Equation EQREF16 is the prior probability of a tweet having the sentiment INLINEFORM1 . The prior probability ( INLINEFORM2 ) can be calculated using the contextual information of the tweets. Therefore, INLINEFORM3 in equation EQREF16 is replaced by INLINEFORM4 , which is the probability of the hypothesis given the contextual information. INLINEFORM5 is the posterior probability of the following Bayesian equation: DISPLAYFORM0 
Where INLINEFORM0 is the set of contextual variables: INLINEFORM1 . INLINEFORM2 captures the probability that a tweet is positive or negative, given the state, hour of day, day of the week, month and author of the tweet. Here INLINEFORM3 is not dependent on the hypothesis, and thus can be ignored. Equation EQREF16 can therefore be rewritten to include the contextual information: DISPLAYFORM0 
Equation EQREF18 is our extended Bayesian model for integrating contextual information with more standard, word-based sentiment classification.
We considered five contextual categories: one spatial, three temporal and one authorial. Here is the list of the five categories:
We used our massive emoticon labelled dataset to calculate the average sentiment for all of these five categories. A tweet was given a score of INLINEFORM0 if it was labelled as negative and a score 1 if it was labelled as positive, so an average sentiment of 0 for a contextual category would mean that tweets in that category were evenly labelled as positive and negative.
All of the 18 million tweets in our dataset originate from the USA and are geo-tagged. Naturally, the tweets are not evenly distributed across the 50 states given the large variation between the population of each state. Figure FIGREF25 shows the percentage of tweets per state, sorted from smallest to largest. Not surprisingly, California has the highest number of tweets ( INLINEFORM0 ), and Wyoming has the lowest number of tweets ( INLINEFORM1 ).
Even the state with the lowest percentage of tweets has more than ten thousand tweets, which is enough to calculate a statistically significant average sentiment for that state. The sentiment for all states averaged across the tweets from the three years is shown in Figure FIGREF26 . Note that an average sentiment of INLINEFORM0 means that all tweets were labelled as positive, INLINEFORM1 means that all tweets were labelled as negative and INLINEFORM2 means that there was an even distribution of positive and negative tweets. The average sentiment of all the states leans more towards the positive side. This is expected given that INLINEFORM3 of the tweets in our dataset were labelled as positive.
It is interesting to note that even with the noisy dataset, our ranking of US states based on their Twitter sentiment correlates with the ranking of US states based on the well-being index calculated by Oswald and Wu BIBREF25 in their work on measuring well-being and life satisfaction across America. Their data is from the behavioral risk factor survey score (BRFSS), which is a survey of life satisfaction across the United States from INLINEFORM0 million citizens. Figure FIGREF27 shows this correlation ( INLINEFORM1 , INLINEFORM2 ).
We looked at three temporal variables: time of day, day of the week and month. All tweets are tagged with timestamp data, which we used to extract these three variables. Since all timestamps in the Twitter historical archives (and public API) are in the UTC time zone, we first converted the timestamp to the local time of the location where the tweet was sent from. We then calculated the sentiment for each day of week (figure FIGREF29 ), hour (figure FIGREF30 ) and month (figure FIGREF31 ), averaged across all 18 million tweets over three years. The 18 million tweets were divided evenly between each month, with INLINEFORM0 million tweets per month. The tweets were also more or less evenly divided between each day of week, with each day having somewhere between INLINEFORM1 and INLINEFORM2 of the tweets. Similarly, the tweets were almost evenly divided between each hour, with each having somewhere between INLINEFORM3 and INLINEFORM4 of the tweets.
Some of these results make intuitive sense. For example, the closer the day of week is to Friday and Saturday, the more positive the sentiment, with a drop on Sunday. As with spatial, the average sentiment of all the hours, days and months lean more towards the positive side.
The last contextual variable we looked at was authorial. People have different baseline attitudes, some are optimistic and positive, some are pessimistic and negative, and some are in between. This difference in personalities can manifest itself in the sentiment of tweets. We attempted to capture this difference by looking at the history of tweets made by users. The 18 million labelled tweets in our dataset come from INLINEFORM0 authors.
In order to calculate a statistically significant average sentiment for each author, we need our sample size to not be too small. However, a large number of the users in our dataset only tweeted once or twice during the three years. Figure FIGREF33 shows the number of users in bins of 50 tweets. (So the first bin corresponds to the number of users that have less than 50 tweets throughout the three year.) The number of users in the first few bins were so large that the graph needed to be logarithmic in order to be legible. We decided to calculate the prior sentiment for users with at least 50 tweets. This corresponded to less than INLINEFORM0 of the users ( INLINEFORM1 out of INLINEFORM2 total users). Note that these users are the most prolific authors in our dataset, as they account for INLINEFORM3 of all tweets in our dataset. The users with less than 50 posts had their prior set to INLINEFORM4 , not favouring positive or negative sentiment (this way it does not have an impact on the Bayesian model, allowing other contextual variables to set the prior).
As it is not feasible to show the prior average sentiment of all INLINEFORM0 users, we created 20 even sentiment bins, from INLINEFORM1 to INLINEFORM2 . We then plotted the number of users whose average sentiment falls into these bins (Figure FIGREF34 ). Similar to other variables, the positive end of the graph is much heavier than the negative end.
We used 5-fold cross validation to train and evaluate our baseline and contextual models, ensuring that the tweets in the training folds were not used in the calculation of any of the priors or in the training of the bigram models. Table TABREF35 shows the accuracy of our models. The contextual model outperformed the baseline model using any of the contextual variables by themselves, with state being the best performing and day of week the worst. The model that utilized all the contextual variables saw a INLINEFORM0 relative and INLINEFORM1 absolute improvement over the baseline bigram model.
Because of the great increase in the volume of data, distant supervised sentiment classifiers for Twitter tend to generally outperform more standard classifiers using human-labelled datasets. Therefore, it makes sense to compare the performance of our classifier to other distant supervised classifiers. Though not directly comparable, our contextual classifier outperforms the distant supervised Twitter sentiment classifier by Go et al BIBREF6 by more than INLINEFORM0 (absolute).
Table TABREF36 shows the precision, recall and F1 score of the positive and negative class for the full contextual classifier (Contextual-All).
Even though our contextual classifier was able to outperform the previous state-of-the-art, distant supervised sentiment classifier, it should be noted that our contextual classifier's performance is boosted significantly by spatial information extracted through geo-tags. However, only about one to two percent of tweets in the wild are geo-tagged. Therefore, we trained and evaluated our contextual model using all the variables except for state. The accuracy of this model was INLINEFORM0 , which is still significantly better than the performance of the purely linguistic classifier. Fortunately, all tweets are tagged with timestamps and author information, so all the other four contextual variables used in our model can be used for classifying the sentiment of any tweet.
Note that the prior probabilities that we calculated need to be recalculated and updated every once in a while to account for changes in the world. For example, a state might become more affluent, causing its citizens to become on average happier. This change could potentially have an effect on the average sentiment expressed by the citizens of that state on Twitter, which would make our priors obsolete.
Sentiment classification of tweets is an important area of research. Through classification and analysis of sentiments on Twitter, one can get an understanding of people's attitudes about particular topics. In this work, we utilized the power of distant supervision to collect millions of noisy labelled tweets from all over the USA, across three years. We used this dataset to create prior probabilities for the average sentiment of tweets in different spatial, temporal and authorial contexts. We then used a Bayesian approach to combine these priors with standard bigram language models. The resulting combined model was able to achieve an accuracy of INLINEFORM0 , outperforming the previous state-of-the-art distant supervised Twitter sentiment classifier by more than INLINEFORM1 .
In the future, we would like to explore additional contextual features that could be predictive of sentiment on Twitter. Specifically, we would like to incorporate the topic type of tweets into our model. The topic type characterizes the nature of the topics discussed in tweets (e.g., breaking news, sports, etc). There has already been extensive work done on topic categorization schemes for Twitter BIBREF26 , BIBREF27 , BIBREF28 which we can utilize for this task.
We would like to thank all the annotators for their efforts. We would also like to thank Brandon Roy for sharing his insights on Bayesian modelling. This work was supported by a generous grant from Twitter.

Figure and table captions:
Table 1: List of emoticons.
Figure 1: Percentage of tweets per state in the USA, sorted from lowest to highest.
Figure 2: Average sentiment of states in the USA, averaged across three years, from 2012 to 2014.
Figure 3: Ranking of US states based on Twitter sentiment vs. ranking of states based on their wellbeing index. r = 0.44, p < 0.005.
Figure 4: Average sentiment of different days of the week in the USA, averaged across three years, from 2012 to 2014.
Figure 5: Average sentiment of different hours of the day in the USA, averaged across three years, from 2012 to 2014.
Figure 6: Average sentiment of different months in the USA, averaged across three years, from 2012 to 2014.
Figure 7: Number of users (logarithmic) in bins of 50 tweets. The first bin corresponds to number of users that have less than 50 tweets throughout the three years and so on.
Figure 8: Number of users (with at least 50 tweets) per sentiment bins of 0.05, averaged across three years, from 2012 to 2014.
Table 3: Classifier accuracy, sorted from worst to best.
Table 4: Precision, recall and F1 score of the full contextual classifier (Contexual-All).

Questions about this paper:
What is the relationship between author and emotional valence?
What is the relationship between time and emotional valence?

Answers:
Among those who wrote more than 50 tweets, 16% of the authors have average sentiment within [0.95, 1.00], while only 1.5% of the authors have average sentiment within [-1.00, -0.95]

The closer the day of the week to Friday and Saturday, the more positive the sentiment; tweets made between 10 a.m. 12 noon are most positive, while those made around 3 a.m. and 20 p.m. are least positive; tweets made in April and May are most positive, while those made in August and September are least positive.