SlowGuess/NAACL_2021_OCR huggingface dataset:
Elsbeth Turcan<sup>1</sup> and Smaranda Muresan<sup>1,2</sup> and Kathleen McKeown<sup>1</sup>  
Department of Computer Science, Columbia University<sup>1</sup>  
Data Science Institute, Columbia University<sup>2</sup>  
{eturcan, smara, kathy}@cs.columbia.edu
The problem of detecting psychological stress in online posts, and more broadly, of detecting people in distress or in need of help, is a sensitive application for which the ability to interpret models is vital. Here, we present work exploring the use of a semantically related task, emotion detection, for equally competent but more explainable and human-like psychological stress detection as compared to a black-box model. In particular, we explore the use of multi-task learning as well as emotion-based language model fine-tuning. With our emotion-infused models, we see comparable results to state-of-the-art BERT. Our analysis of the words used for prediction show that our emotion-infused models mirror psychological components of stress.
As crises have begun to multiply worldwide, including the COVID-19 pandemic and the resulting economic downturn, psychological stress has risen dramatically<sup>1</sup>. The problem of detecting psychological stress, and more broadly, of detecting people in distress and in need of help, is a sensitive application; therefore, the ability to interpret the results, in order to understand why, is vital. The consequences of blindly trusting a black-box model and mislabeling users' stress levels could be serious in a deployed application such as a therapeutic chatbot, where some users may not receive the immediate help they need. Furthermore, models that make decisions based on psychology theory about factors that impact stress will be easier for humans to understand, and their mistakes will be more obvious. Researchers have recently begun to study psychological stress, but in this work, we propose a new focus on examining the information our models use to make decisions and finding ways to incorporate psychological factors, like emotion, into them.
To approach the problem of stress detection, which has much less labeled data than many popular classification tasks, we first note that stress has been shown to interact with emotion (Lazarus, 2006; Thoern et al., 2016; Levenson, 2019), a task that has far more publicly available labeled data. For example, individuals who are stressed are likely to express emotions such as fear, sadness, or anger and unlikely to express emotions such as happiness.
Traditional multi-task learning would normally be helpful in this situation, but there are no currently available datasets labeled with both stress and emotion. Even if there were, it would be beneficial to incorporate external information without re-labeling new datasets for each new combination of useful tasks. Here, we present work exploring how to use semantically related tasks—here, emotion detection—to create emotion-infused models capable of equally competent, but explainable, psychological stress detection as compared to a black-box model. In particular, we explore the use of multi-task learning as well as emotion-infused language model fine-tuning, two existing frameworks which we examine through the lens of interpretability. Our code for this work is available at github.com/eturcan/emotion-infused.
Our contributions in this work are as follows: (i) consideration of factors suggested by psychological theory in deep learning methods for predicting stress, with a focus on emotion; (ii) an exploration of three different approaches to emotion-infused models, with experimental results showing comparable results to the state-of-the-art in all cases; and (iii) a framework for interpreting our models to show the impact of emotion and other factors in our models.
Researchers who use natural language approaches for stress detection often rely on external resources such as diagnostic questionnaires (e.g., Guntuku
<https://www.apa.org/news/press/releases/stress/2020/report-october>
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2895-2909 June 6-11, 2021. ©2021 Association for Computational Linguistics
et al. (2018)) or techniques like pattern matching (patterns such as "I am stressed", e.g., Winata et al. (2018); Lin et al. (2017)) to assign labels. Much of the work that has been done on psychological stress detection focuses either on establishing baseline models with little advancement in computational modeling, or on using external information about the text (e.g., author, time of posting, number of replies), which is usually, but not always available and may differ in meaning or importance across platforms and domains.
There has also been a substantial amount of work on detecting related mental health concerns such as anxiety (e.g., Shen and Rudzicz (2017); Gruda and Hasan (2019); Jiang et al. (2020)), but these are distinct from the generalized experience of stress.
The most similar work to ours is Turcan and McKeown (2019), our prior work publishing a dataset of psychological stress collected from the social media website Reddit and labeled by crowd workers, and presenting baselines with several basic non-neural and BERT-based models on this data. We use this dataset in our current work; however, we focus on exploring interpretable frameworks for this sensitive task and connecting the stress detection task concretely with emotion detection.
The models we propose in this work rely on two types of enhancements to the neural representation learned by models like BERT: multi-task learning and pre-training or fine-tuning. Multi-task learning is an increasingly popular framework in which some parameters in a model are shared between or used to inform multiple different tasks. Hard parameter sharing (Caruana, 1993), the variant we employ, uses some set of parameters as a shared base representation and then allows each task to have some private parameters on top and perform their own separate predictions. Multi-task learning has been successfully applied to many domains across NLP (Sun et al., 2019; Kiperwasser and Ballesteros, 2018; Liu et al., 2019); we are especially interested in instances where it has improved semantic and emotion-related tasks, such as Xu et al. (2018), who perform emotion detection with a suite of secondary semantic tasks including personality classification.
Pre-training and fine-tuning are another type of transfer learning where multiple tasks are trained in sequence rather than at the same time. Pre-trained language models are perhaps the most widely used example, where a large neural language model can
Dataset	Size
Dreaddit	3,553
GoEmotionsA,E,S	58K
GoEmotionsFSJ	4,136
Vent	1.6M
Table 1: The datasets we use in this work and their relative sizes (in terms of total number of data points).
be fine-tuned for many different tasks (Devlin et al., 2019). Additionally, continuing to pre-train the language model itself on language from the target domain has been shown to improve performance (Howard and Ruder, 2018; Chakrabarty et al., 2019; Gururangan et al., 2020) (also note Chronopoulou et al. (2019), who perform this task at the same time as the target task, in a form of multi-task learning). This methodology has been successfully extended to other domains, in which a model is first fine-tuned on some large, broadly useful task and then further fine-tuned for a smaller target task (e.g., Felbo et al. (2017), who first fine-tuned on emoji detection and then fine-tuned on target semantic tasks including emotion and sentiment detection).
It should be noted that the psychological stress is much better studied in settings where researchers have access to some physiological signals (e.g., Zuo et al. (2012); Allen et al. (2014); Al-Shargie et al. (2016); Kumar et al. (2020); Jaiswal et al. (2020)). This work is not as relevant to our task, since we have only text data available when detecting stress from online posts.
A comparison of all the datasets we use in this work can be seen in Table 1. The primary dataset we use for this work is Dreaddit (Turcan and McKeown, 2019), a dataset of 3,553 segments of Reddit posts from various support communities where the authors believe posters are likely to express stress. The stress detection problem as expressed in this dataset is a binary classification problem, with crowdsourced annotations aggregated as the majority vote from five annotators for each data point. We note that this paper frames the stress classification problem in terms of the author and the time-i.e., a post is labeled stressful only if the poster themselves is currently expressing stress.
Because this dataset is small for training a deep learning model, we also experiment with larger datasets to provide auxiliary information. We se
lect the GoEmotions dataset (Demszky et al., 2020), which consists of 58,009 Reddit comments labeled by crowd workers with one or more of 27 emotions (or Neutral), for its larger size and genre similarity to Dreaddit. In this paper, we refer to the dataset in this form as GoEmotions $_{all}$  or GoEmotions $_{A}$ . The authors also published two relabelings of this dataset, achieved by agglomerative clustering: one where labels are clustered together into the Ekman 6 basic emotions (anger, disgust, fear, joy, sadness, surprise, neutral) (Ekman, 1992) (GoEmotions $_{Ekman/E}$ ), and one into simple polarity (positive, negative, ambiguous, neutral) (GoEmotions $_{sentiment/S}$ ). We run our experiments with each version of this dataset.
We also explore the use of another social media website, Vent. Vent is a platform more similar to Twitter orTumblr than Reddit, where users post vents of any length and tag them as they like, and other users react to them or post comments. The benefit of Vent for this purpose is that posters self-identify some emotion they are feeling from a large list of pre-made emotions. The data we use is collected by Malko et al.  $(2021)^{2}$ . We select Vent data that has been labeled with fear or sadness, which we hypothesize to be related to stress, as well as joy, for a contrast. We note that this dataset is strictly single-class, whereas GoEmotions may have more than one emotion label per data point. In all, there are 1.6M vents in our dataset, much larger than Dreaddit or GoEmotions; we randomly sample this data in a stratified manner to create a training, development, and test set with an  $80/10/10$  ratio. To examine the effects of domain similarity, we also select a subset of GoEmotions with the corresponding emotion labels – we subsample the existing "all" dataset to select only data points originally labeled with fear, joy, or sadness, for a final set of 4,136 data points (3,342 of which are the train set). We call this subset GoEmotions $_{FSJ}$ , and we compare it against Vent to see whether genre similarity or data size is more important in this multitask setting.
We experiment with three types of emotion-infused models; that is, we present three different ways to incorporate emotion information into our stress detection models, divided into multi-task learning and fine-tuning.
4.1 Alternating Multi-Task Models
Our first multi-task models, which we refer to as Multi $^{Alt}$ , are simply two single-task models sharing the same base BERT representation layers. The models are alternating in that we train them with two datasets with two different sets of labels-i.e., we train the stress task with the Dreadit data and the emotion task with the GoEmotions or Vent data. We refer to the variants with a subscript, i.e., Multi $^{Alt}$  (i.e., GoEmotions with all emotions), Multi $^{Alt}$  (i.e., the Ekman GoEmotions relabeling), Multi $^{Alt}$  (i.e., the Vent data), etc. The Multi $^{Alt}$  models can be seen in Figure 1a. One loss step for these models consists of only one dataset and task, so they are trained with the negative log-likelihood (NLL) loss for single-label tasks (Dreadit, Vent, GoEmotions $_{FSJ}$ ) and the binary cross-entropy (BCE) loss for multi-label tasks (GoEmotions $_{A,E,S}$ ).
4.2 Classical Multi-Task Models
We also experiment with a multi-task learning setup where we perform the two tasks at the same time on the same input data. We call this architecture Multi. However, because the Dreaddit data is labeled only with stress, we first separately train BERT models on the various versions of GoEmotions and use them to predict emotion labels for Dreaddit. We then take these emotion labels to be "silver data" and train on them alongside stress. The Multi model can be seen in Figure 1b. Since stress detection is our main task in this work, we focus on this task where we have gold labels for stress, but note that it will be interesting in future work to experiment with other task settings, such as whether stress detection can improve emotion classification. In these models, the losses of the stress task and the emotion task are summed together for each batch using a tunable weight parameter, i.e.,  $\mathfrak{L} = \lambda \mathfrak{L}_{\mathrm{stress}} + (1 - \lambda)\mathfrak{L}_{\mathrm{emotion}}$
4.3 Fine-Tuning Models
We experiment with models in which we first endow the BERT representation with knowledge of the emotion task by fine-tuning and then apply it to stress detection (as in Phang et al. (2018)). We perform a sequential version of the MultiAlt models, in which we fine-tune a pre-trained BERT language model on another task, and then extract the language model parameters to initialize a BERT model that we continue to fine-tune
2Due to license and ethics policy restrictions, we currently do not make this data publicly available.
Figure 1: The emotion-informed architectures we use in our experiments.
on Dreadit. We denote these models as, e.g., Fine-Tune $_{GoEmotions_A \rightarrow Dreadit}$  for a model that was first trained on GoEmotions $_{all}$  and then on Dreadit (for space, we will abbreviate Fine-Tune as FT). These fine-tuning models can be seen in Figure 1c. These models are trained with the NLL and BCE losses as in the Multi $^{Alt}$  models.
We present a re-implementation of the same BERT-based fine-tuning model used in Turcan and McKeown (2019), where this model performed best on Dreaddit. We report this as an average of 3 runs with distinct random seeds, and our results are, on average, lower than the single model reported, but with high variance. Because of this, we assume that the previously reported performance is from the high end of this variance and use our average score as our baseline in this work. This model is a pre-trained BERT language model (released as bert-base-uncased by Wolf et al. (2019); we use this same pre-trained language model as the basis for all our models) followed by a dropout layer and a dense classification layer. We also report a recurrent neural network (RNN) model, which uses either a long short-term memory network (LSTM) (Hochreiter and Schmidhuber, 1997) or a gated recurrent unit (GRU) (Cho et al., 2014) in place of the transformer from BERT and is otherwise the
same. These models are trained with the NLL and BCE losses as with the Multi $^{Alt}$  models.
We train all of our models with minibatch gradient descent using the Adam optimizer (Kingma and Ba, 2015) with a batch size of 16, given GPU space constraints. We perform gradient clipping to 1.0 to prevent exploding gradients. When training any model, we perform early stopping based on the F1 score on the Dreadit development set and select the model parameters from the epoch that achieved the best development score for our final evaluated model.
5.3 Hyperparameter Tuning
We tune hyperparameters for all our models using Bayesian Optimization from the Python library  $\mathbf{ax}^3$ . All models train the initial learning rate of the Adam optimizer and the dropout probability before the final classification layer; the Multi models also tune the loss weight parameter  $\lambda$ , and we also note that the RNN model tunes additional parameters such as the type of RNN, hidden dimension, etc. For all models, we tune parameters based on the F1 score on the Dreadit development set; we train an ensemble of three models with three different, fixed random seeds and average their performance for a given parameter setting. We report the mean and standard deviation of three models, with three
<sup>3</sup>https://github.com/facebook/Ax
Model	Binary F1	Accuracy
RNN	67.58 ± 1.22	68.86 ± 1.10
BERT	78.88 ± 1.09	79.11 ± 1.32
MultiGEEA	79.02 ± 0.35	79.72 ± 0.69
MultiGEE	80.24 ± 1.39	81.07 ± 1.13
MultiGES	79.46 ± 1.05	79.86 ± 0.50
MultiGEFSJ	79.17 ± 0.61	78.69 ± 1.86
MultiVent	80.34 ± 1.39	79.67 ± 2.03
MultiDRS	78.97 ± 0.24	78.55 ± 0.07
MultiDrFSJ	78.90 ± 0.59	78.55 ± 0.07
Table 2: Results of our multitask models. The best result under each metric is bolded. GE is GoEmotions.
Model	Binary F1	Accuracy
BERT	78.88 ± 1.09	79.11 ± 1.32
FTGEA→Dr	76.40 ± 0.50	76.83 ± 0.40
FTGEE→Dr	79.44 ± 0.29	79.53 ± 0.46
FTGES→Dr	79.75 ± 0.52	80.61 ± 0.40
FTGEFSJ→Dr	80.25 ± 0.24	80.98 ± 0.20
different random seeds, trained with the best hyperparameters. More details about hyperparameter tuning can be found in the appendix.
We report the results of our multi-task models in Table 2 $^{4}$ . In general, our MultiAlt models perform similarly, and outperform the Multi models; we assume this is due to the introduction of noise in labeling the silver emotion data. Of these models, MultiAlt performs best. With regards to GoEmotions, the 28-way classification of GoEmotions $_A$  naturally leads to lower numerical performance than the tasks with smaller numbers of classes, and we expect that GoEmotions $_S$  may group too many distinctly labeled emotions together under the same emotion labels; it seems GoEmotions $_E$  is the happy medium for this model. We also note that the MultiAlt and MultiGoEmotions $_E$  models perform equally well, which indicates that the genre mismatch is not an issue for this problem, or that Vent has a similar enough genre to Reddit that it does not affect the results. Somewhat surprisingly, MultiGoEmotions $_{FSJ}$  does not do as well as
Table 3: Results of our fine-tuning models. The best result under each metric is bolded. GE is GoEmotions, and Dr is Dreaddit.
Dataset	Macro F1
GoEmotionsA	48.98
GoEmotionsE	62.16
GoEmotionsS	69.65
GoEmotionsFSJ	91.87
Table 4: Performance of our fine-tuning BERT models on the different GoEmotions labelings and datasets.
Multi $_{Vent}$ ; however, the GoEmotions data is much smaller than Vent, especially when subsampled to select specific emotions.
We further report the results of our finetuning models in Table 3. Because we expect that genre similarity should play a larger role when the secondary task can offer no direct training signal during the primary task finetuning, we evaluate on GoEmotions here and not Vent. Here, we observe that our best model, Fine-TuneGoEmotions  $FSJ \rightarrow$  Dreaddit, scores at least one standard deviation above BERT. We see higher increases in performance for the simpler classification problems in GoEmotions  $S$  and GoEmotions  $FSJ$  and worsened performance for GoEmotions  $A$ , suggesting that in the sequential paradigm, more complex tasks are not able to interact appropriately with the main task and instead interfere.
We also report the performance of the fine-tuning BERT models we trained on GoEmotions in order to label Dreaddit with emotion in Table 4; these results track well with the fine-tuning results reported by Demszky et al. (2020). Because these models are intermediates used for labeling, we report the F1 scores of the single model we actually used for labeling, although we tuned their parameters with an average of 3 different instances as with all other models. Many-way classification problems have much more opportunity for error and noise in an already-noisy process of labeling unlabeled data, so we use only the two best-performing GoEmotions models, which are those trained on the fewest-label datasets, GoEmotions $_S$  and GoEmotions $_{FSJ}$ , for our Multi models.
Overall, the inclusion of emotion information results in modest improvements, even though not statistically significant, as compared to BERT. However, our true goal in this work is to analyze the explainability of all of these models, to which we turn next.
<sup>4</sup>We did compute statistical significance by calculating the majority vote of each of the models' 3 runs and using the approximate randomization test, but no model is significantly different from BERT.
GoEmoA	GoEmoE	GoEmoS	GoEmoFSJ*
Dreadit (gold stress + pred. emotion)	0.3396	0.2554	0.0565	0.3207
GoEmotions (gold emotion + pred. stress)	0.1274	0.2668	0.2786	0.4115
Table 5: Correlations of the gold labels for each dataset with labels predicted by the other task's classifier in a Multi $^{Alt}$  model. GoEmotions $_{FSJ}$  (abbreviated for space as GoEmo $_{FSJ}$ ) is starred because its emotion data is not multi-label and therefore the correlation ratio  $\eta$  is used instead of the coefficient of determination  $R^2$  (which is used for the other, multilabel GoEmotions variants).
GoEmotionsS	GoEmotionsFSJ
neutral	negative	ambiguous	positive	fear	sadness	joy
Dreaddit	-0.3960	0.6128	-0.0106	-0.2759	0.9697	0.7113	0.1386
GoEmotions	-0.1021	0.4866	0.0751	-0.3323	0.9545	0.8921	0.0235
Table 6: Per-class scores of emotion and stress for Dreadlit (with gold stress and predicted emotion) and GoEmotions (with gold emotion and predicted stress). For GoEmotionsS, these numbers are the Pearson correlation r of each individual emotion label with the stress labels; for GoEmotions  $F_{SJ}$ , these are the average stress label assigned to data points in each emotion category, where 0 is non-stress and 1 is stress.
We perform three different analyses to probe our trained models and discover what information they learn to use. For our Multi $^{Alt}$  models, we investigate the usefulness of the emotion prediction layers in explaining stress classifications, and for all models, we use Local Interpretable Model-agnostic Explanations (LIME) (Ribeiro et al., 2016) to show that our emotion-infused models rely on meaningfully different types of words than BERT in order to make their predictions.
6.1 Multi-task Knowledge
We perform an analysis of our Multi $^{Alt}$  models to see what information they learn about emotion $^5$ . We take the development sets of each of the datasets (Dreaddit and GoEmotions) and predict their labels under the other task (i.e., emotion for Dreaddit and vice-versa). We report the correlation of these predicted labels with the gold labels in Table  $5^6$ . In this case, the GoEmotions $_{FSJ}$  variant is a single-label three-way classification problem, so we report the correlation ratio  $\eta$  (Fisher, 1938). The other GoEmotions variants are multi-label, so we report the coefficient of determination  $R^2$  (Cohen et al., 2015). We further present breakdowns of the correlations per emotion category for the polarity and
FSJ subsets of GoEmotions in Table 6 and include the All and Ekman sets as well as the Vent data in the appendix.
We observe that our multi-task models generally learn a moderate correlation between the stress labels and the emotion labels; they learn that negative emotions like fear and sadness are linked to stress and neutral or positive emotions are linked to non-stress, which makes intuitive sense. These emotion predictions can help explain the stress classifier's predictions; imagine, for example, showing a patient or clinician that the patient's social media shows a strong pattern of fear and anger as a more detailed explanation for places a stress classifier detects stress. From a machine learning perspective, this correlation also suggests the potential for using emotion data as distantly-labeled stress data to supplement the small extant stress datasets.
6.2 LIWC Analysis
We also investigate the types of information each model is using to make its decisions. In this section, we use the Linguistic Inquiry and Word Count (LIWC) (Pennebaker et al., 2015), a hand-crafted lexicon which collects words belonging to psychologically meaningful categories like positive emotion and cognitive processes, to categorize the information our different models use to predict stress.