SlowGuess/NAACL_2021_OCR huggingface dataset:
We first analyze the unigrams our various models use to perform stress classification using LIME. LIME accepts an input from our development set, perturbs it in the bag-of-unigrams space, and runs one of our classifiers on each perturbation to calculate the importance of various unigrams; through
<sup>5</sup>We did perform an equivalent analysis on the Multi models, which shows similar trends, but as Multi<sup>Alt</sup> shows better performance, we omit it for space.
<sup>6</sup>We also note the possibility that different combinations of emotions are relevant to stress; however, not enough of our data is labeled with multiple emotion labels (4% of Dreadlit's silver labels from GoEmotions<sub>S</sub>, 9% of GoEmotions<sub>E</sub>) to test this hypothesis in this work.
LIWC	BERT	MultiAltGEF	MultiVent	MultiDrFSJ	FTGEFSJ→Dr
Affective Processes	19%	22%	19%	16%	22%
Positive Emotion	8%	10%	9%	9%	12%
Anger	31%	40%	30%	25%	31%
Cognitive Processes	16%	17%	17%	17%	17%
Certainty	8%	13%	12%	16%	11%
Perceptual Processes	17%	15%	14%	14%	15%
Biological Processes	15%	19%	17%	16%	17%
Achievement	17%	19%	19%	13%	17%
Table 7: A comparison of how often several of our models rely on words from several LIWC categories to make their decisions, according to LIME. These numbers represent the percentage of available LIWC words each model selected in the top 10 LIME explanations for the entire development set. Dr is Dreadit, and GE is GoEmotions.
this process, we acquire the 10 unigrams with the highest magnitude output by LIME for each development example and consider them "explanations". We thus have 2,760 individual unigram explanations for the entire development set to analyze.
We then use the word lists from LIWC 2015's 72 psychological categories to see what types of words each classifier tends to use to make decisions of stress vs. non-stress. An abbreviated list of results showing our best models from each category is shown in Table  $7^{7}$ . We observe small but consistent effects suggesting that, in comparison to the basic BERT model, our emotion-enhanced models broadly learn to use the following information:
Affective information. Most emotion-infused models except for Multi learn to use affective information, which includes both positive and negative emotion words, more often. We see the largest increase in anger, one of the emotions we had identified as relevant to stress, for Multi $^{Alt}$ GoEmotions $_{E}$ , which makes intuitive sense because anger is one of the Ekman six basic emotions and thus, is explicitly predicted by this model.
Cognitive processes. All models show some increase in using words related to cognitive processes as compared to BERT; however, its subcategory Certainty, which includes words about absoluteness such as never, obvious, and clearly, shows larger changes. For example, MultiDreadditFSJ uses Certainty twice as often as BERT. These cognitive words seem to target the mental aspects of stress. Rumination and a focus on absoluteness are known signs of anxiety disorders, an extreme form of chronic stress (Nolen-Hoeksema et al., 2008; Miranda and Mennin, 2007).
Additional differences. We observe other,
smaller patterns among LIWC usage for these models. For example, the Multi $^{Alt}$  models use the most achievement-oriented words (although most models show modest increases), suggesting that this information, which includes words about success and failure, is relevant to emotion and to stress. This makes sense, since failing to achieve (e.g., failing a class) can be a major stressor. We also see larger proportions of biological process words used by all emotion-infused models. We suggest this is because Dreaddit includes posts taken from Reddit communities about anxiety and PTSD, where posters are likely to describe their physical and mental symptoms while seeking help.
6.3 Salient Words
We then investigate the data itself for highly significant words using the measure of relative salience proposed by Mohammad (2012),  $\text{RelativeSalience}(w|T_1, T_2) = \frac{f_1}{N_1} - \frac{f_2}{N_2}$ . That is, it measures the importance of a token  $w$  in two different corpora  $T_1, T_2$  by subtracting their two relative frequencies (where  $f_1, f_2$  are the counts of token  $w$  in each corpus and  $N_1, N_2$  are the total tokens in each corpus). We compute this measure for all words in the Dreadit training data, taking our two corpora to be the subsets labeled stress and not-stress. We take the top 200 unigrams for each label (stress as opposed to non-stress and vice-versa) and provide some examples in Table 8 with the full list of words available in the appendix. We examine the words and divide them into related groups in order to understand what types of information should theoretically be most important to classifying the data. For example, we see that different sets of function words are actually among the most important for both classes, with words like conjunctions typically appearing more indicative of stress
7 More detail on the full table is available in the appendix.
Category	Example Words
Stress	Function Words	and, but, how, like, no, not, or, where, why
Negative Sentiment	awful, bad, cry, fear, hate, stress, stupid
Helplessness	alone, can’t, nothing, nowhere, trying
Non-Stress	Function Words	a, for, if, some, the, was, who, will, would
Positive Sentiment	amazing, best, good, great, hope, nice
Support	email, helped, support, thank, together, we
Table 8: Some examples of words identified by relative salience on the Dreadit training data as indicative of stress or non-stress. We group the words by hand into semantically meaningful categories for ease of understanding.
Label	BERT	MultiAltGEF	MultiVent	MultiDrFSJ	FTGEFSJ→Dr
Stress	33%	36%	32%	32%	33%
Non-Stress	15%	15%	19%	18%	17%
Table 9: A comparison of how often several of our models rely on words identified as salient for stress or non-stress to make their decisions, according to LIME. These numbers represent the percentage of available relative salience words each model selected in the top 10 LIME explanations. Dreadtit is Dr, and GoEmotions is GE.
(which echoes Turcan and McKeown (2019)'s finding that stressful data is typically longer with more clauses), while non-stress includes words expressing future-thinking like if, will, and would. We also naturally find negative words for stress and positive words for non-stress, as well as a dichotomy of isolation and helplessness for stress vs. support and community for non-stress which is supported by psychological literature (Grant et al., 2009).
We then look at the intersection between relative salience and LIME explanations, counting how many LIME explanations are highly salient words for stress or non-stress; abbreviated results are shown in Table 9 and the full table is available in the appendix. We see that our emotion-infused models learn to rely more often on words identified as indicative of non-stress, the minority class, instead of stress, the majority class.
We note that the presented models do sometimes make some new errors when incorporating emotional information, and that while these methods successfully incorporate such information with no feature crafting, some further innovation may be needed in order to use this information optimally. For example, we reproduce an example from our development set, with profanity censored:
And everyone was passive aggressive. The manager tried to peg down my salary multiple times like a  $f^{**}$ king haggler at a market. Anyway, I decided to go get some antidepressants and the bottle fell out of my pocket, a coworker noticed and reported
it to my boss. Who smiled and asked if there was anything I'd like to tell her. The passive aggressive  $s^{**}t$  really got to me, and then I realized that I was being illegally paid.
The annotators for Dreaddit label this post not stress, presumably because there is not enough context for how the poster feels about this story presently, and the poster conveys more anger than anything else. The LIME explanations for the BERT model, which labels this correctly, include some profanity, but largely focus on function words. However, all four of our Multi $^{Alt}$ GoEmotions models misclassify this example as stressed and rely on words like aggressive (from passive aggressive) and the profanity to do so. Meanwhile, the emotion classifiers of our Multi $^{Alt}$ GoEmotions models are misled by words like smiled and label this example joy or positive. This is a difficult example; without noticing that the event happened in the past, it is easy to assume the poster is presently stressed. We believe examples like this require some grounding—for example, an understanding of what passive aggressive means and some representation of the timeline involved, that language models simply cannot express in the traditional classification setup.
We also reproduce an anonymized example where our emotion-infused models improve upon BERT:
She comes crying to me and formulates a plan to break up. She talks to  $<\text {name}>$  about their issues and her will to leave him wilts. She stays with him. Rinse and repeat, except it gets worse over time. How can I break the cycle, or help her break the
cycle?
BERT misclassifies this example, where the author is stressed about a friend's situation, as non-stressful, relying on words like break and help, while our Multi<sup>Alt</sup>GoEmotions models successfully use the word crying to predict stress. We notice that crying or worse is the highest-ranked explanation for most of our emotion-infused models. These results are promising for the development of models that focus on information that humans consider intuitive.
In this work, we present a suite of emotion-enhanced models that incorporate emotional information in various ways to enhance the task of binary stress prediction. All three types of our models achieve comparable performance to a state-of-the-art fine-tuning BERT baseline, and, more importantly, we show that they result in more explainable models. We also introduce a new framework for model interpretation using LIME and show that our emotion-enhanced multi-task models offer a new dimension of interpretability by using the predictions of auxiliary tasks to explain the primary task. In our future work, we hope to expand these analyses to tasks in other domains and devise model architectures that can make more direct use of multitask learning to make and explain their predictions.
Our intended use of stress detection is to help those in distress. We envision systems such as therapeutic chatbots or assistants that can understand users' emotions and identify those in need so that a person can intervene. We would urge any user of stress detection technology to carefully control who may use the system.
Currently, the presented models may fail in two ways: they may either misclassify stress, or they may use the wrong information to make their predictions. Obviously, there is some potential harm to a person who is truly in need if a system based on this work fails to detect them, and it is possible that a person who is not truly in need may be irritated or offended if someone reaches out to them because of a mistake. In terms of explanations, we note that previous work has shown that focusing on incorrect rationales can unfairly target some groups of people (Zhong et al., 2019), although in this work we see that function words truly differ across the
stressed and non-stressed populations and we do not observe any language that we know to be representative of minority groups in our explanations.
We emphasize our intention that emotional systems such as this be used responsibly, with a human in the loop—for example, a guidance counselor who can look at the predicted labels and offered explanations for their students' stress levels and decide whether or not they seem sensible.
We note that because most of our data was collected from Reddit, a website with a known overall demographic skew (towards young, white, American men), our conclusions about what stress looks like and how to detect it cannot necessarily be applied to broader groups of people. We also note that we have no way to determine the demographic information of the specific posters in any of our datasets and whether they differ from the overall Reddit statistics. We hope that we, and other researchers, can find ways to consider the specific ways in which minority groups express stress as well.
We thank our reviewers, as well as the members of our Natural Language Processing research group at Columbia University, for their insightful and constructive comments.
Fares Al-Shargie, Masashi Kiguchi, Nasreen Badruddin, Sarat C. Dass, and Ahmad Fadzil Mohammad Hani. 2016. Mental stress assessment using simultaneous measurement of eeg and fnirs. Biomedical Optics Express, 7(10):3882-3898.
Andrew P. Allen, Paul J. Kennedy, John F. Cryan, Timothy G. Dinan, and Gerard Clarke. 2014. Biological and psychological markers of stress in humans: Focus on the trier social stress test. *Neuroscience & Biobehavioral Reviews*, 38:94–124.
Richard Caruana. 1993. Multitask learning: A knowledge-based source of inductive bias. In Proceedings of the Tenth International Conference on Machine Learning, pages 41-48. Morgan Kaufmann.
Tuhin Chakrabarty, Christopher Hidey, and Kathy McKeown. 2019. IMHO fine-tuning improves claim detection. In Proceedings of the 2019 Conference of the North American Chapter of the Association for
<sup>8</sup>https://social.techjunkie.com/demographics-redisgit
Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 558-563, Minneapolis, Minnesota. Association for Computational Linguistics.
Kyunghyun Cho, Bart van Merrienboer, Caglar Güliquehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. CoRR, abs/1406.1078.
Alexandra Chronopoulou, Christos Baziotis, and Alexandros Potamianos. 2019. An embarrassingly simple approach for transfer learning from pretrained language models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2089-2095, Minneapolis, Minnesota. Association for Computational Linguistics.
Jacob Cohen, Patricia Cohen, Stephen G. West, and Leona S. Aiken. 2015. Applied multiple regression/correlation analysis for the behavioral sciences, 3rd edition. Routledge.
Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan S. Cowen, Gaurav Nemade, and Su-jith Ravi. 2020. GoEmotions: A dataset of fine-grained emotions. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 4040-4054. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171-4186. Association for Computational Linguistics.
Paul Ekman. 1992. Are there basic emotions? Psychological Review, 99(5):550-553.
Bjarke Felbo, Alan Mislove, Anders Søgaard, Iyad Rahwan, and Sune Lehmann. 2017. Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 1615-1625. Association for Computational Linguistics.
Ronald A. Fisher. 1938. Statistical methods for research workers.
Nina Grant, Mark Hamer, and Andrew Steptoe. 2009. Social Isolation and Stress-related Cardiovascular, Lipid, and Cortisol Responses. Annals of Behavioral Medicine, 37(1):29-37.
Dritjon Gruda and Souleiman Hasan. 2019. Feeling anxious? perceiving anxiety in tweets using machine learning. CoRR, abs/1909.06959.
Sharath Chandra Guntuku, Anneke Buffone, Kokil Jaidka, Johannes C. Eichstaedt, and Lyle H. Ungar. 2018. Understanding and measuring psychological stress using social media. CoRR, abs/1811.07430.
Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don't stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 8342-8360. Association for Computational Linguistics.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735-1780.
Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328-339, Melbourne, Australia. Association for Computational Linguistics.
Mimansa Jaiswal, Cristian-Paul Bara, Yuanhang Luo, Mihai Burzo, Rada Mihalcea, and Emily Mower Provost. 2020. MuSE: a multimodal dataset of stressed emotion. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 1499-1510, Marseille, France. European Language Resources Association.
Zhengping Jiang, Sarah Ita Levitan, Jonathan Zomick, and Julia Hirschberg. 2020. Detection of mental health from Reddit via deep contextualized representations. In Proceedings of the 11th International Workshop on Health Text Mining and Information Analysis, pages 147-156, Online. Association for Computational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Eliyahu Kiperwasser and Miguel Ballesteros. 2018. Scheduled multi-task learning: From syntax to translation. Trans. Assoc. Comput. Linguistics, 6:225-240.
Satish Kumar, A S M Iftekhar, Michael Goebel, Tom Bullock, Mary H. MacLean, Michael B. Miller, Tyler Santander, Barry Giesbrecht, Scott T. Grafton, and B. S. Manjunath. 2020. Stressnet: Detecting stress in thermal videos.
Richard S. Lazarus. 2006. Stress and emotion: a new synthesis, 1st edition. Springer Publishing Company.
Robert W. Levenson. 2019. Stress and illness: A role for specific emotions. Psychosomatic Medicine, 81(8):720-730.
Huijie Lin, Jia Jia, Jiezhong Qiu, Yongfeng Zhang, Guangyao Shen, Lexing Xie, Jie Tang, Ling Feng, and Tat-Seng Chua. 2017. Detecting stress based on social interactions in social networks. IEEE Transactions on Knowledge and Data Engineering, 29(09):1820-1833.
Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019. Multi-task deep neural networks for natural language understanding. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4487-4496. Association for Computational Linguistics.
Anton Malko, Cecile Paris, Andreas Duenser, Mervi Kangas, Diego Mollá, Ross Sparks, and Stephen Wan. 2021. Expressing and reacting to emotions in a specialised online community. Technical report, CSIRO.
Regina Miranda and Douglas S. Mennin. 2007. Depression, generalized anxiety disorder, and certainty in pessimistic predictions about the future. Cognitive Therapy and Research, pages 71-82.
Saif M. Mohammad. 2012. From once upon a time to happily ever after: Tracking emotions in mail and books. Decis. Support Syst., 53(4):730-741.
Susan Nolen-Hoeksema, Blair E. Wisco, and Sonja Lyubomirsky. 2008. Rethinking rumination. Perspectives on Psychological Science, (5):400-424.
James Pennebaker, Ryan L. Boyd, Kayla Jordan, and Kate Blackburn. 2015. The development and psychometric properties of liwc2015.
Jason Phang, Thibault Févry, and Samuel R. Bowman. 2018. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. CoRR, abs/1811.01088.
Marco Túlio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. "Why should I trust you?": Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016, pages 1135-1144. ACM.
Judy Hanwen Shen and Frank Rudzicz. 2017. Detecting anxiety through Reddit. In Proceedings of the Fourth Workshop on Computational Linguistics and Clinical Psychology — From Linguistic Signal to Clinical Reality, pages 58–65, Vancouver, BC. Association for Computational Linguistics.
Yu Sun, Shuohuan Wang, Yu-Kun Li, Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang. 2019. ERNIE 2.0: A continual pre-training framework for language understanding. CoRR, abs/1907.12412.
Hanna A. Thoern, Marcus Grueschow, Ulrike Ehlert, Christian C. Ruff, and Brigit Kleim. 2016. Attentional bias towards positive emotion predicts stress resilience. PLoS ONE, 11(3).
Elsbeth Turcan and Kathleen McKeown. 2019. Dreaddit: A reddit dataset for stress analysis in social media. In Proceedings of the Tenth International Workshop on Health Text Mining and Information Analysis LOUHI@EMNLP 2019, Hong Kong, November 3, 2019, pages 97-107. Association for Computational Linguistics.
Genta Indra Winata, Onno Pepijn Kampman, and Pascale Fung. 2018. Attention-based LSTM for psychological stress detection from spoken language using distant supervision. CoRR, abs/1805.12307.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2019. Huggingface's transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771.
Peng Xu, Andrea Madotto, Chien-Sheng Wu, Ji Ho Park, and Pascale Fung. 2018. Emo2vec: Learning generalized emotion representation by multitask training. In Proceedings of the 9th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, WASSA@EMNLP 2018, Brussels, Belgium, October 31, 2018, pages 292-298. Association for Computational Linguistics.
Ruiqi Zhong, Steven Shao, and Kathleen R. McKeown. 2019. Fine-grained sentiment analysis with faithful attention. CoRR, abs/1908.06870.
Xin Zuo, Tian Li, and Pascale Fung. 2012. A multilingual natural stress emotion database. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC-2012), pages 1174–1178, Istanbul, Turkey. European Language Resources Association (ELRA).
Name	Type	Range
learning rate	continuous	[10-6, 10-3]
Pdropout)	continuous	[0, 1]
λ	continuous	[0, 0.9]
embedding dim.	integer	[32, 256]
hidden dim.	integer	[32, 512]
nlayers	integer	[1, 3]
RNN	categorical	{LSTM, GRU}
We report the contents of the NAACL 2021 Reproducibility Checklist that apply to our work.
A.1 Training and Tuning
Our MultiAlt, MultiDreadittS, and MultiDreadittFSJ models were trained on one Tesla V100 GPU with one CPU. All other models were trained on one Nvidia P100 GPU with one CPU.
Hyperparameter tuning was done the same way for every model, with Bayesian optimization as implemented by  $ax$ , with the F1 score on the Dreaddit development set as the criterion to optimize. MultiDreadittS and MultiDreadittFSJ were given 35 trials for time constraints; all other models were given 50 trials. All models were trained with a patience of 5 epochs and a tolerance of 0.0001 for dev set improvement, and allowed to run for a maximum of 20 epochs. All models tuned the initial learning rate and the dropout probability, with the Multi models also tuning the lambda weight parameter between their two task losses. Additionally, the RNN model was initialized from scratch and additionally tuned the embedding dimension, hidden dimension, number of layers, and type of RNN. Our parameter ranges are shown in Table 10.