A Computational Approach to Automatic Prediction of Drunk Texting

Abstract: Alcohol abuse may lead to unsociable behavior such as crime, drunk driving, or privacy leaks. We introduce automatic drunk-texting prediction as the task of identifying whether a text was written when under the influence of alcohol. We experiment with tweets labeled using hashtags as distant supervision. Our classifiers use a set of N-gram and stylistic features to detect drunk tweets. Our observations present the first quantitative evidence that text contains signals that can be exploited to detect drunk-texting.

The ubiquity of communication devices has made social media highly accessible. The content on these media reflects a user's day-to-day activities. This includes content created under the influence of alcohol. In popular culture, this has been referred to as `drunk-texting'. In this paper, we introduce automatic `drunk-texting prediction' as a computational task. Given a tweet, the goal is to automatically identify if it was written by a drunk user. We refer to tweets written under the influence of alcohol as `drunk tweets', and the opposite as `sober tweets'.
A key challenge is to obtain an annotated dataset. We use hashtag-based supervision so that the authors of the tweets mention if they were drunk at the time of posting a tweet. We create three datasets by using different strategies that are related to the use of hashtags. We then present SVM-based classifiers that use N-gram and stylistic features such as capitalisation, spelling errors, etc. Through our experiments, we make subtle points related to: (a) the performance of our features, (b) how our approach compares against human ability to detect drunk-texting, (c) most discriminative stylistic features, and (d) an error analysis that points to future work. To the best of our knowledge, this is a first study that shows the feasibility of text-based analysis for drunk-texting prediction.
Past studies show the relation between alcohol abuse and unsociable behaviour such as aggression BIBREF0 , crime BIBREF1 , suicide attempts BIBREF2 , drunk driving BIBREF3 , and risky sexual behaviour BIBREF4 . suicide state that “those responsible for assessing cases of attempted suicide should be adept at detecting alcohol misuse”. Thus, a drunk-texting prediction system can be used to identify individuals susceptible to these behaviours, or for investigative purposes after an incident.
Drunk-texting may also cause regret. Mail Goggles prompts a user to solve math questions before sending an email on weekend evenings. Some Android applications avoid drunk-texting by blocking outgoing texts at the click of a button. However, to the best of our knowledge, these tools require a user command to begin blocking. An ongoing text-based analysis will be more helpful, especially since it offers a more natural setting by monitoring stream of social media text and not explicitly seeking user input. Thus, automatic drunk-texting prediction will improve systems aimed to avoid regrettable drunk-texting. To the best of our knowledge, ours is the first study that does a quantitative analysis, in terms of prediction of the drunk state by using textual clues.
Several studies have studied linguistic traits associated with emotion expression and mental health issues, suicidal nature, criminal status, etc. BIBREF5 , BIBREF6 . NLP techniques have been used in the past to address social safety and mental health issues BIBREF7 .
Drunk-texting prediction is the task of classifying a text as drunk or sober. For example, a tweet `Feeling buzzed. Can't remember how the evening went' must be predicted as `drunk', whereas, `Returned from work late today, the traffic was bad' must be predicted as `sober'. The challenges are:
We use hashtag-based supervision to create our datasets, similar to tasks like emotion classification BIBREF8 . The tweets are downloaded using Twitter API (https://dev.twitter.com/). We remove non-Unicode characters, and eliminate tweets that contain hyperlinks and also tweets that are shorter than 6 words in length. Finally, hashtags used to indicate drunk or sober tweets are removed so that they provide labels, but do not act as features. The dataset is available on request. As a result, we create three datasets, each using a different strategy for sober tweets, as follows:
The drunk tweets for Datasets 1 and 2 are the same. Figure FIGREF9 shows a word-cloud for these drunk tweets (with stop words and forms of the word `drunk' removed), created using WordItOut. The size of a word indicates its frequency. In addition to topical words such as `bar', `bottle' and `wine', the word-cloud shows sentiment words such as `love' or `damn', along with profane words.
Heuristics other than these hashtags could have been used for dataset creation. For example, timestamps were a good option to account for time at which a tweet was posted. However, this could not be used because user's local times was not available, since very few users had geolocation enabled.
The complete set of features is shown in Table TABREF7 . There are two sets of features: (a) N-gram features, and (b) Stylistic features. We use unigrams and bigrams as N-gram features- considering both presence and count.
Table TABREF7 shows the complete set of stylistic features of our prediction system. POS ratios are a set of features that record the proportion of each POS tag in the dataset (for example, the proportion of nouns/adjectives, etc.). The POS tags and named entity mentions are obtained from NLTK BIBREF9 . Discourse connectors are identified based on a manually created list. Spelling errors are identified using a spell checker by enchant. The repeated characters feature captures a situation in which a word contains a letter that is repeated three or more times, as in the case of happpy. Since drunk-texting is often associated with emotional expression, we also incorporate a set of sentiment-based features. These features include: count/presence of emoticons and sentiment ratio. Sentiment ratio is the proportion of positive and negative words in the tweet. To determine positive and negative words, we use the sentiment lexicon in mpqa. To identify a more refined set of words that correspond to the two classes, we also estimated 20 topics for the dataset by estimating an LDA model BIBREF10 . We then consider top 10 words per topic, for both classes. This results in 400 LDA-specific unigrams that are then used as features.
Using the two sets of features, we train SVM classifiers BIBREF11 . We show the five-fold cross-validation performance of our features on Datasets 1 and 2, in Section SECREF17 , and on Dataset H in Section SECREF21 . Section SECREF22 presents an error analysis. Accuracy, positive/negative precision and positive/negative recall are shown as A, PP/NP and PR/NR respectively. `Drunk' forms the positive class, while `Sober' forms the negative class.
Table TABREF14 shows the performance for five-fold cross-validation for Datasets 1 and 2. In case of Dataset 1, we observe that N-gram features achieve an accuracy of 85.5%. We see that our stylistic features alone exhibit degraded performance, with an accuracy of 75.6%, in the case of Dataset 1. Table TABREF16 shows top stylistic features, when trained on the two datasets. Spelling errors, POS ratios for nouns (POS_NOUN), length and sentiment ratios appear in both lists, in addition to LDA-based unigrams. However, negative recall reduces to a mere 3.2%. This degradation implies that our features capture a subset of drunk tweets and that there are properties of drunk tweets that may be more subtle. When both N-gram and stylistic features are used, there is negligible improvement. The accuracy for Dataset 2 increases from 77.9% to 78.1%. Precision/Recall metrics do not change significantly either. The best accuracy of our classifier is 78.1% for all features, and 75.6% for stylistic features. This shows that text-based clues can indeed be used for drunk-texting prediction.
Using held-out dataset H, we evaluate how our system performs in comparison to humans. Three annotators, A1-A3, mark each tweet in the Dataset H as drunk or sober. Table TABREF19 shows a moderate agreement between our annotators (for example, it is 0.42 for A1 and A2). Table TABREF20 compares our classifier with humans. Our human annotators perform the task with an average accuracy of 68.8%, while our classifier (with all features) trained on Dataset 2 reaches 64%. The classifier trained on Dataset 2 is better than which is trained on Dataset 1.
Some categories of errors that occur are:
Incorrect hashtag supervision: The tweet `Can't believe I lost my bag last night, literally had everything in! Thanks god the bar man found it' was marked with`#Drunk'. However, this tweet is not likely to be a drunk tweet, but describes a drunk episode in retrospective. Our classifier predicts it as sober.
Seemingly sober tweets: Human annotators as well as our classifier could not identify whether `Will you take her on a date? But really she does like you' was drunk, although the author of the tweet had marked it so. This example also highlights the difficulty of drunk-texting prediction.
Pragmatic difficulty: The tweet `National dress of Ireland is one's one vomit.. my family is lovely' was correctly identified by our human annotators as a drunk tweet. This tweet contains an element of humour and topic change, but our classifier could not capture it.
In this paper, we introduce automatic drunk-texting prediction as the task of predicting a tweet as drunk or sober. First, we justify the need for drunk-texting prediction as means of identifying risky social behavior arising out of alcohol abuse, and the need to build tools that avoid privacy leaks due to drunk-texting. We then highlight the challenges of drunk-texting prediction: one of the challenges is selection of negative examples (sober tweets). Using hashtag-based supervision, we create three datasets annotated with drunk or sober labels. We then present SVM-based classifiers which use two sets of features: N-gram and stylistic features. Our drunk prediction system obtains a best accuracy of 78.1%. We observe that our stylistic features add negligible value to N-gram features. We use our heldout dataset to compare how our system performs against human annotators. While human annotators achieve an accuracy of 68.8%, our system reaches reasonably close and performs with a best accuracy of 64%.
Our analysis of the task and experimental findings make a case for drunk-texting prediction as a useful and feasible NLP application.

Figure and table captions:
Figure 1: Word cloud for drunk tweets
Table 1: Our Feature Set for Drunk-texting Prediction
Table 2: Performance of our features on Datasets 1 and 2
Table 4: Cohen’s Kappa for three annotators (A1A3)
Table 3: Top stylistic features for Datasets 1 and 2 obtained using Chi-squared test-based ranking
Table 5: Performance of human evaluators and our classifiers (trained on all features), for Dataset-H as the test set

Questions about this paper:
What baseline model is used?
What stylistic features are used to detect drunk texts?

Answers:
Human evaluators
LDA unigrams (Presence/Count), POS Ratio, #Named Entity Mentions, #Discourse Connectors, Spelling errors, Repeated characters, Capitalization, Length, Emoticon (Presence/Count), Sentiment Ratio.