Introduction
In our online world, social media users tweet, post, and message an incredible number of times each day, and the interconnected, information-heavy nature of our lives makes stress more prominent and easily observable than ever before. With many platforms such as Twitter, Reddit, and Facebook, the scientific community has access to a massive amount of data to study the daily worries and stresses of people across the world.
Stress is a nearly universal phenomenon, and we have some evidence of its prevalence and recent increase. For example, the American Psychological Association (APA) has performed annual studies assessing stress in the United States since 2007 which demonstrate widespread experiences of chronic stress. Stress is a subjective experience whose effects and even definition can vary from person to person; as a baseline, the APA defines stress as a reaction to extant and future demands and pressures, which can be positive in moderation. Health and psychology researchers have extensively studied the connection between too much stress and physical and mental health BIBREF0, BIBREF1.
In this work, we present a corpus of social media text for detecting the presence of stress. We hope this corpus will facilitate the development of models for this problem, which has diverse applications in areas such as diagnosing physical and mental illness, gauging public mood and worries in politics and economics, and tracking the effects of disasters. Our contributions are as follows:
Dreaddit, a dataset of lengthy social media posts in five categories, each including stressful and non-stressful text and different ways of expressing stress, with a subset of the data annotated by human annotators;
Supervised models, both discrete and neural, for predicting stress, providing benchmarks to stimulate further work in the area; and
Analysis of the content of our dataset and the performance of our models, which provides insight into the problem of stress detection.
In the remainder of this paper, we will review relevant work, describe our dataset and its annotation, provide some analysis of the data and stress detection problem, present and discuss results of some supervised models on our dataset, and finally conclude with our summary and future work.
Related Work
Because of the subjective nature of stress, relevant research tends to focus on physical signals, such as cortisol levels in saliva BIBREF2, electroencephalogram (EEG) readings BIBREF3, or speech data BIBREF4. This work captures important aspects of the human reaction to stress, but has the disadvantage that hardware or physical presence is required. However, because of the aforementioned proliferation of stress on social media, we believe that stress can be observed and studied purely from text.
Other threads of research have also made this observation and generally use microblog data (e.g., Twitter). The most similar work to ours includes BIBREF5, who use Long Short-Term Memory Networks (LSTMs) to detect stress in speech and Twitter data; BIBREF6, who examine the Facebook and Twitter posts of users who score highly on a diagnostic stress questionnaire; and BIBREF7, who detect stress on microblogging websites using a Convolutional Neural Network (CNN) and factor graph model with a suite of discrete features. Our work is unique in that it uses data from Reddit, which is both typically longer and not typically as conducive to distant labeling as microblogs (which are labeled in the above work with hashtags or pattern matching, such as “I feel stressed”). The length of our posts will ultimately enable research into the causes of stress and will allow us to identify more implicit indicators. We also limit ourselves to text data and metadata (e.g., posting time, number of replies), whereas BIBREF5 also train on speech data and BIBREF7 include information from photos, neither of which is always available. Finally, we label individual parts of longer posts for acute stress using human annotators, while BIBREF6 label users themselves for chronic stress with the users' voluntary answers to a psychological questionnaire.
Researchers have used Reddit data to examine a variety of mental health conditions such as depression BIBREF8 and other clinical diagnoses such as general anxiety BIBREF9, but to our knowledge, our corpus is the first to focus on stress as a general experience, not only a clinical concept.
Dataset ::: Reddit Data
Reddit is a social media website where users post in topic-specific communities called subreddits, and other users comment and vote on these posts. The lengthy nature of these posts makes Reddit an ideal source of data for studying the nuances of phenomena like stress. To collect expressions of stress, we select categories of subreddits where members are likely to discuss stressful topics:
Interpersonal conflict: abuse and social domains. Posters in the abuse subreddits are largely survivors of an abusive relationship or situation sharing stories and support, while posters in the social subreddit post about any difficulty in a relationship (often but not exclusively romantic) and seek advice for how to handle the situation.
Mental illness: anxiety and Post-Traumatic Stress Disorder (PTSD) domains. Posters in these subreddits seek advice about coping with mental illness and its symptoms, share support and successes, seek diagnoses, and so on.
Financial need: financial domain. Posters in the financial subreddits generally seek financial or material help from other posters.
We include ten subreddits in the five domains of abuse, social, anxiety, PTSD, and financial, as detailed in tab:data-spread, and our analysis focuses on the domain level. Using the PRAW API, we scrape all available posts on these subreddits between January 1, 2017 and November 19, 2018; in total, 187,444 posts. As we will describe in sec:annotation, we assign binary stress labels to 3,553 segments of these posts to form a supervised and semi-supervised training set. An example segment is shown in fig:stress-example. Highlighted phrases are indicators that the writer is stressed: the writer mentions common physical symptoms (nausea), explicitly names fear and dread, and uses language indicating helplessness and help-seeking behavior.
The average length of a post in our dataset is 420 tokens, much longer than most microblog data (e.g., Twitter's character limit as of this writing is 280 characters). While we label segments that are about 100 tokens long, we still have much additional data from the author on which to draw. We feel this is important because, while our goal in this paper is to predict stress, having longer posts will ultimately allow more detailed study of the causes and effects of stress.
In tab:data-examples, we provide examples of labeled segments from the various domains in our dataset. The samples are fairly typical; the dataset contains mostly first-person narrative accounts of personal experiences and requests for assistance or advice. Our data displays a range of topics, language, and agreement levels among annotators, and we provide only a few examples. Lengthier examples are available in the appendix.
Dataset ::: Data Annotation
We annotate a subset of the data using Amazon Mechanical Turk in order to begin exploring the characteristics of stress. We partition the posts into contiguous five-sentence chunks for labeling; we wish to annotate segments of the posts because we are ultimately interested in what parts of the post depict stress, but we find through manual inspection that some amount of context is important. Our posts, however, are quite long, and it would be difficult for annotators to read and annotate entire posts. This type of data will allow us in the future not only to classify the presence of stress, but also to locate its expressions in the text, even if they are diffused throughout the post.
We set up an annotation task in which English-speaking Mechanical Turk Workers are asked to label five randomly selected text segments (of five sentences each) after taking a qualification test; Workers are allowed to select “Stress”, “Not Stress”, or “Can't Tell” for each segment. In our instructions, we define stress as follows: “The Oxford English Dictionary defines stress as `a state of mental or emotional strain or tension resulting from adverse or demanding circumstances'. This means that stress results from someone being uncertain that they can handle some threatening situation. We are interested in cases where that someone also feels negatively about it (sometimes we can find an event stressful, but also find it exciting and positive, like a first date or an interview).”. We specifically ask Workers to decide whether the author is expressing both stress and a negative attitude about it, not whether the situation itself seems stressful. Our full instructions are available in the appendix.
We submit 4,000 segments, sampled equally from each domain and uniformly within domains, to Mechanical Turk to be annotated by at least five Workers each and include in each batch one of 50 “check questions” which have been previously verified by two in-house annotators. After removing annotations which failed the check questions, and data points for which at least half of the annotators selected “Can't Tell”, we are left with 3,553 labeled data points from 2,929 different posts. We take the annotators' majority vote as the label for each segment and record the percentage of annotators who agreed. The resulting dataset is nearly balanced, with 52.3% of the data (1,857 instances) labeled stressful.
Our agreement on all labeled data is $\kappa =0.47$, using Fleiss's Kappa BIBREF10, considered “moderate agreement” by BIBREF11. We observe that annotators achieved perfect agreement on 39% of the data, and for another 32% the majority was 3/5 or less. This suggests that our data displays significant variation in how stress is expressed, which we explore in the next section.
Data Analysis
While all our data has the same genre and personal narrative style, we find distinctions among domains with which classification systems must contend in order to perform well, and distinctions between stressful and non-stressful data which may be useful when developing such systems. Posters in each subreddit express stress, but we expect that their different functions and stressors lead to differences in how they do so in each subreddit, domain, and broad category.
By domain. We examine the vocabulary patterns of each domain on our training data only, not including unlabeled data so that we may extend our analysis to the label level. First, we use the word categories from the Linguistic Inquiry and Word Count (LIWC) BIBREF12, a lexicon-based tool that gives scores for psychologically relevant categories such as sadness or cognitive processes, as a proxy for topic prevalence and expression variety. We calculate both the percentage of tokens per domain which are included in a specific LIWC word list, and the percentage of words in a specific LIWC word list that appear in each domain (“coverage” of the domain).
Results of the analysis are highlighted in tab:domain-liwc. We first note that variety of expression depends on domain and topic; for example, the variety in the expression of negative emotions is particularly low in the financial domain (with 1.54% of words being negative emotion (“negemo”) words and only 31% of “negemo” words used). We also see clear topic shifts among domains: the interpersonal domains contain roughly 1.5 times as many social words, proportionally, as the others; and domains are stratified by their coverage of the anxiety word list (with the most in the mental illness domains and the least in the financial domain).
We also examine the overall lexical diversity of each domain by calculating Yule's I measure BIBREF13. fig:domain-yule shows the lexical diversity of our data, both for all words in the vocabulary and for only words in LIWC's “negemo” word list. Yule's I measure reflects the repetitiveness of the data (as opposed to the broader coverage measured by our LIWC analysis). We notice exceptionally low lexical diversity for the mental illness domains, which we believe is due to the structured, clinical language surrounding mental illnesses. For example, posters in these domains discuss topics such as symptoms, medical care, and diagnoses (fig:stress-example, tab:data-examples). When we restrict our analysis to negative emotion words, this pattern persists only for anxiety; the PTSD domain has comparatively little lexical variety, but what it does have contributes to its variety of expression for negative emotions.
By label. We perform similar analyses on data labeled stressful or non-stressful by a majority of annotators. We confirm some common results in the mental health literature, including that stressful data uses more first-person pronouns (perhaps reflecting increased self-focus) and that non-stressful data uses more social words (perhaps reflecting a better social support network).
Additionally, we calculate measures of syntactic complexity, including the percentage of words that are conjunctions, average number of tokens per labeled segment, average number of clauses per sentence, Flesch-Kincaid Grade Level BIBREF14, and Automated Readability Index BIBREF15. These scores are comparable for all splits of our data; however, as shown in tab:label-complexity, we do see non-significant but persistent differences between stressful and non-stressful data, with stressful data being generally longer and more complex but also rated simpler by readability indices. These findings are intriguing and can be explored in future work.
By agreement. Finally, we examine the differences among annotator agreement levels. We find an inverse relationship between the lexical variety and the proportion of annotators who agree, as shown in fig:agreement-diversity. While the amount of data and lexical variety seem to be related, Yule's I measure controls for length, so we believe that this trend reflects a difference in the type of data that encourages high or low agreement.
Methods
In order to train supervised models, we group the labeled segments by post and randomly select 10% of the posts ($\approx $ 10% of the labeled segments) to form a test set. This ensures that while there is a reasonable distribution of labels and domains in the train and test set, the two do not explicitly share any of the same content. This results in a total of 2,838 train data points (51.6% labeled stressful) and 715 test data points (52.4% labeled stressful). Because our data is relatively small, we train our traditional supervised models with 10-fold cross-validation; for our neural models, we break off a further random 10% of the training data for validation and average the predictions of 10 randomly-initialized trained models.
In addition to the words of the posts (both as bag-of-n-grams and distributed word embeddings), we include features in three categories:
Lexical features. Average, maximum, and minimum scores for pleasantness, activation, and imagery from the Dictionary of Affect in Language (DAL) BIBREF16; the full suite of 93 LIWC features; and sentiment calculated using the Pattern sentiment library BIBREF17.
Syntactic features. Part-of-speech unigrams and bigrams, the Flesch-Kincaid Grade Level, and the Automated Readability Index.
Social media features. The UTC timestamp of the post; the ratio of upvotes to downvotes on the post, where an upvote roughly corresponds to a reaction of “like” and a downvote to “dislike” (upvote ratio); the net score of the post (karma) (calculated by Reddit, $n_\text{upvotes} - n_\text{downvotes}$); and the total number of comments in the entire thread under the post.
Methods ::: Supervised Models
We first experiment with a suite of non-neural models, including Support Vector Machines (SVMs), logistic regression, Naïve Bayes, Perceptron, and decision trees. We tune the parameters for these models using grid search and 10-fold cross-validation, and obtain results for different combinations of input and features.
For input representation, we experiment with bag-of-n-grams (for $n \in \lbrace 1..3\rbrace $), Google News pre-trained Word2Vec embeddings (300-dimensional) BIBREF18, Word2Vec embeddings trained on our large unlabeled corpus (300-dimensional, to match), and BERT embeddings trained on our unlabeled corpus (768-dimensional, the top-level [CLS] embedding) BIBREF19. We experiment with subsets of the above features, including separating the features by category (lexical, syntactic, social) and by magnitude of the Pearson correlation coefficient ($r$) with the training labels. Finally, we stratify the training data by annotator agreement, including separate experiments on only data for which all annotators agreed, data for which at least 4/5 annotators agreed, and so on.
We finally experiment with neural models, although our dataset is relatively small. We train both a two-layer bidirectional Gated Recurrent Neural Network (GRNN) BIBREF20 and Convolutional Neural Network (CNN) (as designed in BIBREF21) with parallel filters of size 2 and 3, as these have been shown to be effective in the literature on emotion detection in text (e.g., BIBREF22, BIBREF23). Because neural models require large amounts of data, we do not cull the data by annotator agreement for these experiments and use all the labeled data we have. We experiment with training embeddings with random initialization as well as initializing with our domain-specific Word2Vec embeddings, and we also concatenate the best feature set from our non-neural experiments onto the representations after the recurrent and convolutional/pooling layers respectively.
Finally, we apply BERT directly to our task, fine-tuning the pretrained BERT-base on our classification task for three epochs (as performed in BIBREF19 when applying BERT to any task). Our parameter settings for our various models are available in the appendix.
Results and Discussion
We present our results in tab:supervised-results. Our best model is a logistic regression classifier with Word2Vec embeddings trained on our unlabeled corpus, high-correlation features ($\ge $ 0.4 absolute Pearson's $r$), and high-agreement data (at least 4/5 annotators agreed); this model achieves an F-score of 79.8 on our test set, a significant improvement over the majority baseline, the n-gram baseline, and the pre-trained embedding model, (all by the approximate randomization test, $p < 0.01$). The high-correlation features used by this model are LIWC's clout, tone, and “I” pronoun features, and we investigate the use of these features in the other model types. Particularly, we apply different architectures (GRNN and CNN) and different input representations (pretrained Word2Vec, domain-specific BERT).
We find that our logistic regression classifier described above achieves comparable performance to BERT-base (approximate randomization test, $p > 0.5$) with the added benefits of increased interpretability and less intensive training. Additionally, domain-specific word embeddings trained on our unlabeled corpus (Word2Vec, BERT) significantly outperform n-grams or pretrained embeddings, as expected, signaling the importance of domain knowledge in this problem.
We note that our basic deep learning models do not perform as well as our traditional supervised models or BERT, although they consistently, significantly outperform the majority baseline. We believe this is due to a serious lack of data; our labeled dataset is orders of magnitude smaller than neural models typically require to perform well. We expect that neural models can make good use of our large unlabeled dataset, which we plan to explore in future work. We believe that the superior performance of the pretrained BERT-base model (which uses no additional features) on our dataset supports this hypothesis as well.
In tab:data-and-feat-comparison, we examine the impact of different feature sets and levels of annotator agreement on our logistic regressor with domain-specific Word2Vec embeddings and find consistent patterns supporting this model. First, we see a tradeoff between data size and data quality, where lower-agreement data (which can be seen as lower-quality) results in worse performance, but the larger 80% agreement data consistently outperforms the smaller perfect agreement data. Additionally, LIWC features consistently perform well while syntactic features consistently do not, and we see a trend towards the quality of features over their quantity; those with the highest Pearson correlation with the train set (which all happen to be LIWC features) outperform sets with lower correlations, which in turn outperform the set of all features. This suggests that stress detection is a highly lexical problem, and in particular, resources developed with psychological applications in mind, like LIWC, are very helpful.
Finally, we perform an error analysis of the two best-performing models. Although the dataset is nearly balanced, both BERT-base and our best logistic regression model greatly overclassify stress, as shown in tab:confusion-matrices, and they broadly overlap but do differ in their predictions (disagreeing with one another on approximately 100 instances).
We note that the examples misclassified by both models are often, though not always, ones with low annotator agreement (with the average percent agreement for misclassified examples being 0.55 for BERT and 0.61 for logistic regression). Both models seem to have trouble with less explicit expressions of stress, framing negative experiences in a positive or retrospective way, and stories where another person aside from the poster is the focus; these types of errors are difficult to capture with the features we used (primarily lexical), and further work should be aware of them. We include some examples of these errors in tab:error-analysis-paper, and further illustrative examples are available in the appendix.
Conclusion and Future Work
In this paper, we present a new dataset, Dreaddit, for stress classification in social media, and find the current baseline at 80% F-score on the binary stress classification problem. We believe this dataset has the potential to spur development of sophisticated, interpretable models of psychological stress. Analysis of our data and our models shows that stress detection is a highly lexical problem benefitting from domain knowledge, but we note there is still room for improvement, especially in incorporating the framing and intentions of the writer. We intend for our future work to use this dataset to contextualize stress and offer explanations using the content features of the text. Additional interesting problems applicable to this dataset include the development of effective distant labeling schemes, which is a significant first step to developing a quantitative model of stress.
Acknowledgements
We would like to thank Fei-Tzin Lee, Christopher Hidey, Diana Abagyan, and our anonymous reviewers for their insightful comments during the writing of this paper. This research was funded in part by a Presidential Fellowship from the Fu Foundation School of Engineering and Applied Science at Columbia University.
Data Samples
We include several full posts (with identifying information removed and whitespace collapsed) in fig:data-appendix-1,fig:data-appendix-2,fig:data-appendix-3,fig:data-appendix-4. Posts are otherwise reproduced exactly as obtained (with spelling errors, etc.). The selected examples are deliberately of a reasonable but fairly typical length for readability and space concerns; recall that our average post length is 420 tokens, longer for interpersonal subreddits and shorter for other subreddits.
Full Annotation Guidelines
We provide our annotation instructions in full in fig:annotation. Mechanical Turk Workers were given these instructions and examples followed by five text segments (one of which was one of our 50 check questions) and allowed to select “Stress”, “Not Stress', or “Can't Tell” for each. Workers were given one hour to complete the HIT and paid $0.12 for each HIT where they correctly answered the check question, with a limit of 30 total submissions per Worker.
Parameter Settings
We tune our traditional supervised models' parameters using grid search, all as implemented in Python's scikit-learn library BIBREF25. Our best model uses unbalanced class weights, L2 penalty, and a constant term C=10, with other parameters at their default values. All cross-validation runs were initialized with the same random seed for comparability and reproducibility.
We train each of our neural models with the Adam optimizer BIBREF24 for up to ten epochs with early stopping measured on the validation set. We apply a dropout rate of 0.5 during training in the recurrent layers and after the convolutional layers. We set our hidden sizes (i.e., the output of the recurrent and pooling layers) as well as our batch size to 128, and tune our learning rate to $5\cdot 10^{-4}$; we set these parameters relatively small to try to work with our small data. We also experiment with scheduling the learning rate on plateau of the validation loss, and with pre-training the models on a much larger sentiment dataset, the Stanford Sentiment Treebank BIBREF26, to help combat the problem of small data, but this does not improve the performance of our neural networks.
Error Analysis Examples
As a supplement to our error analysis discussion in sec:results, we provide additional examples of test data points which one or both of our best models (BERT-base or our best logistic regressor with embeddings trained on our unlabeled corpus and high-correlation discrete features) failed to classify correctly in tab:error-analysis-appendix.

Question: Is the dataset balanced across categories?
Answer: Yes

Question: What supervised methods are used?
Answer: Support Vector Machines (SVMs), logistic regression, Naïve Bayes, Perceptron, and decision trees, a two-layer bidirectional Gated Recurrent Neural Network (GRNN) BIBREF20 and Convolutional Neural Network (CNN) (as designed in BIBREF21)

Question: What labels are in the dataset?
Answer: binary label of stress or not stress

Question: What categories does the dataset come from?
Answer: abuse, social, anxiety, PTSD, and financial