SlowGuess/ACL_2020_OCR huggingface dataset:
For hypothesis H1, we evaluate measures of social skill in responses to different categories of prompts. While research in psychology uses several visual, linguistic and paralinguistic indicators of social skills, in this paper we focus on two indicators that are measurable in our data: average response length in tokens and the rate of spoken-language fillers and discourse markers usage. The first measure - response length - can be seen as a basic measure of taciturnity. The second measure - usage of fillers and discourse markers - can be used as proxy for conversational skills, since speakers use these terms to manage conversations (Fox Tree, 2010). Christenfeld (1995) and Lake et al. (2011) also find that discourse marker usage correlates with social skill. Following is the list of fillers and discourse markers: "um", "uh", "you know", "well", "oh", "so", "I mean", and "like".
Table 2 shows the values of these measures for social skill for responses to backchannels and starters categories. We found that both measures were significantly lower for responses to starters-category prompts for depressed participants as opposed to not-depressed participants  $(p < 0.05)$ . However, the measures showed no significant difference between depressed and not-depressed individuals for responses to categories
4Note that this prompt was not removed in Section 3.2 since by itself, the prompt's presence is not predictive of depression (without considering the response).
<sup>5</sup>We compute this measure as the ratio of discourse marker and filler occurrences to number of tokens, averaged over responses.
representing backchannels ("mmm," "mm," "awesome," and "nice"). Note that a conversation usually begins with prompts from the starters category and thereafter backchannels are used to encourage the speaker to continue speaking (Goodwin, 1986). Given this, our results suggest that depressed individuals in the given population indeed initially demonstrate poorer social skills than not-depressed individuals, but the effect levels off as the interviewer encourages them to keep speaking using backchannels. Given this, our results suggest that depressed individuals in the given population indeed initially demonstrate poorer social skills than not depressed individuals, but the effect stops being visible as the conversation continues, either because the depressed individuals become more comfortable talking or because the interviewers' encouragement through backchannels elicits more contributions.
Hypotheses H2 and H3 - regarding qualified language and self-focus, respectively - involve semantic qualities of depressed language. To explore these hypotheses, we use a reverse engineering approach to determine salient words for depression in responses to each prompt category.
We describe this reverse engineering approach as follows: since the aggregated representation of an individual's responses in a category  $(\bar{\mathbf{R}}_i^k$  computed in Equation 2) resides in the same vector space as individual word embeddings, we can identify words in our corpus that produce the strongest (positive) signal for depression in various categories. We refer to these as signal words. Signal words are ranked not by their frequency in the dataset, but by their predictive potential - the strength of association between the word's semantic representation and a given category. We evaluate hypotheses H2 and H3 by observing semantic similarities between these signal words and the language themes identified by the hypotheses. Selections from the top 10 signal words for depression associated with categories corresponding to starters, specific backchannels, and generic backchannels are shown in Figure 3.
Figure 3 shows hypothesis H2 is supported by
Starters	Generic Backchannels	Specific Backchannels
"When was the last time you...?"	"mhm"	"nice"
"How have you been feeling lately?"	"mm"	"awesome"
Correlation with depression	nah	theoretical	mission
ah	mechanical	invaluable
eh	plausible	wished
huh	indirect	secure
wow	variable	accomplished
sold	moms	California
drinks	we	bars
explorer	do	restaurants
communications	neighborhood	mild
veterinary	kids	hair
Figure 3: Signal words associated with language in depressed individuals. Columns represent various types of prompts (Starters, Generic Backchannels and Specific Backchannels). The bottom half shows ranked lists of signal words from the responses. Blue words are strongly indicative and red words are least indicative of depression.
signal words in responses to generic backchannels; words such as "theoretical" and "plausible" constitute qualified language, and in the context of generic backchannels, the proposed model identifies them as predictive of depression. Similarly, hypothesis H3 is also supported in responses to generic backchannels. The model identifies words related to community ("kids," "neighborhood," "we") as strong negative signals for depression, supporting that depressed language reflects detachment from community.
However, the model only focuses on these semantic themes in responses to generic backchannel categories. As we found in our evaluation of hypothesis H1, the model localizes cues for depression to specific contexts. Signal words for depression in responses to the starters category are more reflective of our findings for hypothesis H1: the model focuses on short, low-semantic-content words that could indicate social skill deficit. For example, Figure 3 shows we identified "wow" as a signal word for the starters category. In one example from the corpus, a depressed participant uses "wow" to express uncomfort with an emotional question: the interviewer asks, "Tell me about the last time you were really happy," and the interviewee responds, "wow (laughter) um."
For responses to specific backchannels, strong signal words reflect themes of goals and desires
$^6$  A word's signal strength is computed for a given category  $k$  by taking the dot product of the word's embedding with the weights in the decision layer corresponding to category  $k$ . Large positive numbers correspond to positive predictions and vice versa. Since the Decision Layer is a dot product with all response aggregations, it is intuitive to compute prediction strength for a group of categories by adding together prediction strengths from individual groups.
("wished," "mission," "accomplished"). Psychologists have observed a correlation between depression and goal commitment and pursuit (Vergara and Roberts, 2011; Klossek, 2015), and our finding indicates that depressed individuals discuss goal-related themes as response to specific backchannels.
Overall, our model's design not only helps in reducing its opacity but also informs psycholinguistic analysis, making it more useful as part of an informed decision-making process. Our analysis indicates that even though research has shown strong correlation between depression and various interpersonal factors such as social skills, self-focus and usage of qualified language, clinical support tools should focus on these factors in light of conversational cues.
4.7 Sources of Error
In this section, we analyze major sources of error. We apply a similar reverse engineering method as in Section 4.6. For prompts in each category, we consider corresponding responses that result in strong incorrect signals (false positive or false negative) based on the category's weights in the decision layer. We focus on the categories with the most significance presence in the dataset: the categories corresponding to starters, the "mhm" backchannel, and the prompt "Have you been diagnosed with depression?".
For the starters category, false positive-signal responses tend to contain a high presence of fillers and discourse markers ("uh," "huh," "post mm traumatic stress uh no uh uh," "hmm"). It is possible that because the model learned to focus on short, low-semantic-content responses, it incorrectly correlates presence of fillers and discourse markers with depression. For the "mhm" category, we identified several false negatives, in which the responses included concrete words like "uh nice environment", "I love the landscape", and "I love the waters". Since the "mhm" category focuses on vague, qualified language to predict depression (see Figure 3), the presence of concrete words in these responses could have misled the model. For the "Have you been diagnosed with depression?" category, the misclassified interviews contained short responses to this prompt like "so," "never," "yes," "yeah," and "no," as well as statements containing the word "depression." For this category, the model seems to incorrectly correlate short re
sponses and direct mentions of depression with the depressed class.
Much work exists at the intersection of natural language processing (NLP), psycholinguistics, and clinical psychology. For example, exploring correlations between counselor-patient interaction dynamics and counseling outcomes (Althoff et al., 2016); studying linguistic development of mental healthcare counsellors (Zhang et al., 2019); identifying differences in how people disclose mental illnesses across gender and culture (De Choudhury et al., 2017); predicting a variety of mental health conditions from social media posts (Sekulic and Strube, 2019; De Choudhury et al., 2013a; Guntuku et al., 2019; Coppersmith et al., 2014); and analyzing well-being (Smith et al., 2016) and distress (Buechel et al., 2018).
Specifically, many researchers have used NLP methods for identifying depression (Morales et al., 2017). They focus on for predicting depression from Twitter posts (Resnik et al., 2015; De Choudhury et al., 2013b; Jamil et al., 2017), Facebook updates (Schwartz et al., 2014), student essays (Resnik et al., 2013), etc.
Previous works have also focused on predicting depression severity from screening interview data (Yang et al., 2016; Sun et al., 2017; Pampouchidou et al., 2016). Unlike ours, these approaches rely on audio, visual, and text input.
More recent approaches are based on deep learning. Yang et al. (2017) propose a CNN-based model leveraging jointly trained paragraph vectorizations, Al Hanai et al. (2018) propose an LSTM-based model fusing audio features with Doc2Vec representations of response text, Makiuchi et al. (2019) combine LSTM and CNN components, and Mallol-Ragolta et al. (2019) propose a model that uses a hierarchical attention mechanism. However, these approaches are more opaque and difficult to interpret.
Other approaches are similar to ours in the sense that they utilize the structure provided by interview prompts. Al Hanai et al. (2018) and Gong and Poellabauer (2017) propose models that extract separate sets of features for responses to each unique prompt in their corpus. However, these approaches require manually identifying unique prompts. Our model can instead automatically learn new, task-specific categorization of prompts.
Lubis et al. (2018) perform a K-means clustering of prompt to assign prompts to latent dialogue act categories. These are used as features in a neural dialogue system. Our approach expands upon this idea of incorporating a separate unsupervised clustering step by allowing the learning goal to influence the clustering. Our approach is also related to that of Chaturvedi et al. (2014) in that it automatically categorizes various parts of the conversation. However, they use domain-specific handcrafted features and discrete latent variables for this categorization. Our approach instead can leverage the neural architecture to automatically identify features useful for this categorization.
To the best of our knowledge, our approach is the first deep learning approach that jointly categorizes prompts to learn context-dependent patterns in responses.
This paper addressed the problem of identifying depression from interview transcripts. The proposed model analyzes the participant's responses in light of various categories of prompts provided by the interviewer. The model jointly learns these prompt categories while identifying depression. We show that the model outperforms competitive baselines and we use the prompt categorization to investigate various psycholinguistic hypotheses.
Depression prediction is a difficult task which requires especially trained experts to conduct interviews and do their detailed analysis (Lakhan et al., 2010). While the absolute performance of our model is low for immediate practical deployment, it improves upon existing methods and at the same time, unlike modern methods, provides insight about the model's workflow. For example, our findings show how language of depressed individuals changes when interviewers use backchannels to encourage continued speech. We hope that this combination will encourage the research community to make more progress in this direction. Future work can further investigate temporal patterns in how language used by depressed people evolves over the course of an interaction.
Tuka Al Hanai, Mohammad Ghassemi, and James Glass. 2018. Detecting Depression with Audio/Text Sequence Modeling of Interviews. In *Interspeech* 2018, 19th Annual Conference of the International
Speech Communication Association, Hyderabad, India, 2-6 September 2018, pages 1716-1720.
Tim Althoff, Kevin Clark, and Jure Leskovec. 2016. Large-scale Analysis of Counseling Conversations: An Application of Natural Language Processing to Mental Health. Transactions of the Association for Computational Linguistics, 4:463-476.
Nancy J. C. Andreasen. 1976. Linguistic Analysis of Speech in Affective Disorders. Archives of General Psychiatry, 33(11):1361.
Janet B. Bavelas, Linda Coates, and Trudy Johnson. 2000. Listeners as co-narrators. Journal of Personality and Social Psychology, 79(6):941-952.
Sven Buechel, Anneke Buffone, Barry Slaff, Lyle Ungar, and João Sedoc. 2018. Modeling Empathy and Distress in Reaction to News Stories. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 4758-4765.
Snigdha Chaturvedi, Dan Goldwasser, and Hal Daume III. 2014. Predicting instructor's intervention in MOOC forums. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1501-1511, Baltimore, Maryland. Association for Computational Linguistics.
Nicholas Christenfeld. 1995. Does it hurt to say um? Journal of Nonverbal Behavior, 19:171-186.
Glen Coppersmith, Mark Dredze, and Craig Harman. 2014. Quantifying Mental Health Signals in Twitter. In Proceedings of the Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality, pages 51-60, Baltimore, Maryland, USA. Association for Computational Linguistics.
Munmun De Choudhury, Scott Counts, and Eric Horvitz. 2013a. Predicting postpartum changes in emotion and behavior via social media. In 2013 ACM SIGCHI Conference on Human Factors in Computing Systems, CHI '13, Paris, France, April 27 - May 2, 2013, pages 3267-3276.
Munmun De Choudhury, Michael Gamon, Scott Counts, and Eric Horvitz. 2013b. Predicting Depression via Social Media. In Proceedings of the Seventh International Conference on Weblogs and Social Media, ICwsm 2013, Cambridge, Massachusetts, USA, July 8-11, 2013.
Munmun De Choudhury, Sanket S. Sharma, Tomaz Logar, Wouter Eekhout, and René Clausen Nielsen. 2017. Gender and Cross-Cultural Differences in Social Media Disclosures of Mental Illness. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing - CSCW '17, pages 353-369, Portland, Oregon, USA. ACM Press.
David DeVault, Ron Artstein, Grace Benn, Teresa Dey, Edward Fast, Alesia Gainer, Kallirroi Georgila, Jonathan Gratch, Arno Hartholt, Margaux Lhommet, Gale M. Lucas, Stacy Marsella, Fabrizio Morbini, Angela Nazarian, Stefan Scherer, Giota Stratou, Apar Suri, David R. Traum, Rachel Wood, Yuyu Xu, Albert A. Rizzo, and Louis-Philippe Morency. 2014. SimSensei Kiosk: A Virtual Human Interviewer for Healthcare Decision Support. In International conference on Autonomous Agents and Multi-Agent Systems, AAMAS '14, Paris, France, May 5-9, 2014, pages 1061-1068.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171-4186.
Jean E. Fox Tree. 2010. Discourse Markers across Speakers and Settings. Language and Linguistics Compass, 4(5):269-281.
Yuan Gong and Christian Poellabauer. 2017. Topic Modeling Based Multi-modal Depression Detection. In Proceedings of the 7th Annual Workshop on Audio/Visual Emotion Challenge - AVEC '17, pages 69-76, Mountain View, California, USA. ACM Press.
Charles Goodwin. 1986. Between and within: Alternative sequential treatments of continuers and assessments. Human Studies, 9(2-3):205-217.
Yves Grandvalet and Yoshua Bengio. 2005. Semi-supervised Learning by Entropy Minimization. page 8.
Jonathan Gratch, Ron Artstein, Gale Lucas, Giota Strou, Stefan Scherer, Angela Nazarian, Rachel Wood, Jill Boberg, David DeVault, Stacy Marsella, David Traum, Skip Rizzo, and Louis-Philippe Morency. 2014. The Distress Analysis Interview Corpus of human and computer interviews. In Proceedings of the Ninth International Conference on Language Resources and Evaluation, LREC 2014, Reykjavik, Iceland, May 26-31, 2014, pages 3123-3128.
Sharath Chandra Guntuku, Daniel Preotiuc-Pietro, Johannes C. Eichstaedt, and Lyle H. Ungar. 2019. What Twitter Profile and Posted Images Reveal about Depression and Anxiety. In Proceedings of the Thirteenth International Conference on Web and Social Media, ICWSM 2019, Munich, Germany, June 11-14, 2019, pages 236-246.
John Heritage and Jeffrey Robinson. 2006. The Structure of Patients' Presenting Concerns: Physicians' Opening Questions. Health communication, 19:89-102.
Zunaira Jamil, Diana Inkpen, Prasadith Buddhitha, and Kenton White. 2017. Monitoring Tweets for Depression to Detect At-risk Users. In Proceedings of the Fourth Workshop on Computational Linguistics and Clinical Psychology — From Linguistic Signal to Clinical Reality, pages 32-40, Vancouver, BC. Association for Computational Linguistics.
Ulrike Klossek. 2015. The Role of Goals and Goal Orientation as Predisposing Factors for Depression. Ph.D. thesis, University of Exeter.
Johanna K. Lake, Karin R. Humphreys, and Shannon Cardy. 2011. Listener vs. speaker-oriented aspects of speech: Studying the disfluencies of individuals with autism spectrum disorders. Psychonomic Bulletin & Review, 18(1):135-140.
Shaheen E Lakhan, Karen Vieira, and Elissa Hamlat. 2010. Biomarkers in psychiatry: drawbacks and potential for misuse. International Archives of Medicine, 3(1):1.
Nurul Lubis, Sakriani Sakti, Koichiro Yoshino, and Satoshi Nakamura. 2018. Unsupervised Counselor Dialogue Clustering for Positive Emotion Elicitation in Neural Dialogue System. In Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue, pages 161-170, Melbourne, Australia. Association for Computational Linguistics.
Mariana Rodrigues Makiuchi, Tifani Warnita, Kuniaki Uto, and Koichi Shinoda. 2019. Multimodal Fusion of BERT-CNN and Gated CNN Representations for Depression Detection. In Proceedings of the 9th International on Audio/Visual Emotion Challenge and Workshop - AVEC '19, pages 55-63, Nice, France. ACM Press.
Adria Mallol-Ragolta, Ziping Zhao, Lukas Stappen, Nicholas Cummins, and Björn W. Schuller. 2019. A Hierarchical Attention Network-Based Approach for Depression Detection from Transcribed Clinical Interviews. In *Interspeech* 2019, pages 221-225. ISCA.
Margaret Mccallister, Beth Matarasso, Barbara Dixon, and C Shepperd. 2004. Conversation starters: re-examining and reconstructing first encounters within the therapeutic relationship. Journal of Psychiatric and Mental Health Nursing, 11.
Michelle Morales, Stefan Scherer, and Rivka Levitan. 2017. A Cross-modal Review of Indicators for Depression Detection Systems. In Proceedings of the Fourth Workshop on Computational Linguistics and Clinical Psychology — From Linguistic Signal to Clinical Reality, pages 1–12, Vancouver, BC. Association for Computational Linguistics.
Anastasia Pampouchidou, Kostas Marias, Fan Yang, Manolis Tsiknakis, Olympia Simantiraki, Amir Fazlollahi, Matthew Pediaditis, Dimitris Manousos, Alexandros Roniotis, Georgios Giannakakis, Fabrice Meriaudeau, and Panagiotis Simos. 2016. Depression Assessment by Fusing High and Low Level
Features from Audio, Video, and Text. In Proceedings of the 6th International Workshop on Audio/Visual Emotion Challenge - AVEC '16, pages 27-34, Amsterdam, The Netherlands. ACM Press.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global Vectors for Word Representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar; A meeting of SIGDAT, a Special Interest Group of the ACL, pages 1532-1543.
Philip Resnik, William Armstrong, Leonardo Claudino, Thang Nguyen, Viet-An Nguyen, and Jordan Boyd-Graber. 2015. Beyond LDA: Exploring Supervised Topic Modeling for Depression-Related Language in Twitter. In Proceedings of the 2nd Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality, pages 99–107, Denver, Colorado. Association for Computational Linguistics.
Philip Resnik, Anderson Garron, and Rebecca Resnik. 2013. Using Topic Modeling to Improve Prediction of Neuroticism and Depression in College Students. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1348-1353, Seattle, Washington, USA. Association for Computational Linguistics.
Stephanie Rude, Eva-Maria Gortner, and James Pennebaker. 2004. Language use of depressed and depression-vulnerable college students. Cognition & Emotion, 18(8):1121-1133.
H. Andrew Schwartz, Johannes Eichstaedt, Margaret L. Kern, Gregory Park, Maarten Sap, David Stillwell, Michal Kosinski, and Lyle Ungar. 2014. Towards Assessing Changes in Degree of Depression through Facebook. In Proceedings of the Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality, pages 118-125, Baltimore, Maryland, USA. Association for Computational Linguistics.
Chris Segrin. 1990. A meta-analytic review of social skill deficits in depression. Communication Monographs, 57(4):292-308.
Ivan Sekulic and Michael Strube. 2019. Adapting Deep Learning Methods for Mental Health Prediction on Social Media. In Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019), pages 322-327, Hong Kong, China. Association for Computational Linguistics.
Laura Smith, Salvatore Giorgi, Rishi Solanki, Johannes Eichstaedt, H. Andrew Schwartz, Muhammad Abdul-Mageed, Anneke Buffone, and Lyle Ungar. 2016. Does 'well-being' translate on Twitter? In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2042-2047, Austin, Texas. Association for Computational Linguistics.