SlowGuess/NAACL_2021_OCR huggingface dataset:
Meri Coleman and TL Liau. 1975. A computer readability formula designed for machine scoring. Journal of Applied Psychology, 60(2):283-284.
Dwight Connely. 1976. Some effects of general anxiety and situational stress upon lexical diversity, speaking rate, speaking time, and evaluations of a speaker.
Glen Coppersmith, Mark Dredze, and Craig Harman. 2014. Quantifying mental health signals in Twitter. In Proceedings of the Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality, pages 51-60, Baltimore, Maryland, USA. Association for Computational Linguistics.
Manuelle Cottin, Andrew JJ MacIntosh, Akiko Kato, Akinori Takahashi, Marion Debin, Thierry Raclot, and Yan Ropert-Coudert. 2014. Corticosterone administration leads to a transient alteration of foraging behaviour and complexity in a diving seabird. Marine Ecology Progress Series, 496:249-262.
Michael Covington and Joe McFall. 2010. Cutting the gordian knot: The moving-average type-token ratio (mattr). Journal of Quantitative Linguistics, 17:94-100.
Edgar Dale and Jeanne Chall. 1948. A formula for predicting readability. Educational Research Bulletin, 27:11-20.
Egdar Dale and Jeanne Chall. 1995. Readability revisited: The new Dale-Chall readability formula. Brookline Books, Cambridge, UK.
Karthik Dinakar, Jackie Chen, Henry Lieberman, Rosalind Picard, and Robert Filbin. 2015. Mixed-initiative real-time topic modeling & visualization for crisis counseling. In Proceedings of the 20th international conference on intelligent user interfaces, pages 417-426.
Marcel Fafchamps and Julien Labonne. 2016. Using split samples to improve inference about causal effects. Working Paper 21842, National Bureau of Economic Research.
Andrzej Galecki and Tomasz Burzykowski. 2013. Linear Mixed-Effects Models Using R: A Step-by-Step Approach. Springer Publishing Company, Incorporated.
Amy L Goldberger. 1997. Fractal variability versus pathologic periodicity: complexity loss and stereotypy in disease. Perspectives in Biology and Medicine, 40(4).
Ary L Goldberger, C-K Peng, and Lewis A Lipsitz. 2002. What is physiologic complexity and how does it change with aging and disease? Neurobiology of aging, 23(1):23-26.
Matthew Honnibal and Ines Montani. 2017. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. https://spacy.io/.
Marijkhe Howeler. 1972. Diversity of word usage as a stress indicator in an interview situation. Journal of Psycholinguistic Research, 1(3):243-248.
Christine Howes, Matthew Purver, and Rose McCabe. 2014. Linguistic indicators of severity and progress in online text-based therapy for depression. In Proceedings of the Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality, pages 7-16, Baltimore, Maryland, USA. Association for Computational Linguistics.
Thomas D Hull, Philippa Connolly, Kush Mahan, and Katie Yang. 2018. The treatment effectiveness of asynchronous text therapy for depression and anxiety: A longitudinal cohort study. *Talkspace Research*.
J. Peter Kincaid, Robert P. Fishburne, Richard Lawrence Rogers, and Brad S. Chissom. 1975. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel.
Victor Kuperman, Hans Stadthagen-Gonzalez, and Marc Brysbaert. 2012. Age-of-acquisition ratings for 30,000 English words. Behavior Research Methods, 44:978-990.
Fei-Tzin Lee, Derrick Hull, Jacob Levine, Bonnie Ray, and Kathy McKeown. 2019. Identifying therapist conversational actions across diverse psychotherapeutic approaches. In Proceedings of the Sixth Workshop on Computational Linguistics and Clinical Psychology, pages 12-23, Minneapolis, Minnesota. Association for Computational Linguistics.
OP Malhotra and S Roy Salam. 1991. Cyclic oscillations of leucocyte counts in chronic myeloid leukaemia. Postgraduate medical journal, 67(783):87-89.
Philip McCarthy and Scott Jarvis. 2010. Mtdld, vocdd, and hd-d: A validation study of sophisticated approaches to lexical diversity assessment. Behavior research methods, 42:381-92.
Philip M McCarthy. 2005. An assessment of the range and usefulness of lexical diversity measures and the potential of the measure of textual, lexical diversity (MTLD). Ph.D. thesis, The University of Memphis.
Philip M McCarthy and Scott Jarvis. 2007. vocab: A theoretical and empirical evaluation. Language Testing, 24(4):459-488.
Gerard Mckee, D.D. Malvern, and Brian Richards. 2000. Measuring vocabulary diversity using dedicated software. *Literary and Linguistic Computing*, 15:323-337.
Jones G Parker, Jesse D Marshall, Biafra Ahanonu, Yu-Wei Wu, Tony Hyun Kim, Benjamin F Grewe, Yanping Zhang, Jin Zhong Li, Jun B Ding, Michael D Ehlers, et al. 2018. Diametric neural ensemble dynamics in parkinsonian and dyskinetic states. Nature, 557(7704):177-182.
James W Pennebaker, Matthias R Mehl, and Kate G Niederhoffer. 2003. Psychological aspects of natural language. use: our words, our selves. Annual Reviews of Psychology, 54.
Verónica Pérez-Rosas, Xuetong Sun, Christy Li, Yuchen Wang, Kenneth Resnicow, and Rada Mihalcea. 2018. Analyzing the quality of counseling conversations: the tell-tale signs of high-quality counseling. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Verónica Pérez-Rosas, Xinyi Wu, Kenneth Resnicow, and Rada Mihalcea. 2019. What makes a good counselor? learning to distinguish between high-quality and low-quality counseling conversations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 926-935, Florence, Italy. Association for Computational Linguistics.
John P Pestian, Pawel Matykiewicz, Michelle Linn-Gust, Brett South, Ozlem Uzuner, Jan Wiebe, K Brettonnel Cohen, John Hurdle, and Christopher Brew. 2012. Sentiment analysis of suicide notes: A shared task. Biomedical informatics insights, 5:BII-S9042.
Robert L Spitzer, Kurt Kroenke, Janet BW Williams, and Bernd Lowe. 2006. A brief measure for assessing generalized anxiety disorder. JAMA Internal Medicine, 166(10).
Talkspace. 2020. Talkspace - #1 rated online therapy, 1 million+ users. https://www-talkspace.com/, accessed Oct 1, 2020.
Yulia Tsvetkov, Manaal Faruqui, Wang Ling, Brian MacWhinney, and Chris Dyer. 2016. Learning the curriculum with Bayesian optimization for task-specific word representation learning. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 130-139, Berlin, Germany. Association for Computational Linguistics.
Regina JJM Van den Eijnden, Gert-Jan Meerkerk, Ad A Vermulst, Renske Spijkerman, and Rutger CME Engels. 2008. Online communication, compulsive internet use, and psychosocial well-being among adolescents: A longitudinal study. Developmental psychology, 44(3):655.
Robert S Wachal and Ottfried Spreen. 1973. Some measures of lexical diversity in aphasic and normal language performance. Language and Speech, 16(2):169-181.
Justine Zhang, Robert Filbin, Christine Morrison, Jaclyn Weiser, and Cristian Danescu-Niculescu-Mizil. 2019. Finding your voice: The linguistic development of mental health counselors. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 936-947, Florence, Italy. Association for Computational Linguistics.
$\S 8.1$  defines and explains our linguistic complexity measures in further detail. To supplement  $\S 4, \S 8.2$  details our linear mixed model.
8.1 Definitions of Complexity Measures
Here, we describe in detail the linguistic complexity measures we used, which span lexical diversity (♀), syntactic simplicity (♂), readability (♀), and prototypicality (♂).
1. Type-Token Ratio (TTR): #types / #tokens. Because TTR decreases for longer texts, we use the moving average type-token ratio (MATTR) (Covington and McFall, 2010)—for a given sequence of tokens, we slide a window of size  $W = 50$  over all tokens with a stride of  $s = 1$ , compute lexical richness for each of the windows, and output the average.
2.  $\bullet$  HD-D (vocd-D): McCarthy and Jarvis (2007) found that output  $\mathcal{D}$  of vocd-D (Mckee et al., 2000), which estimates the fit of TTRs for text samples of different length, is merely a complex approximation ( $R = 0.971$ ) of a hypergeometric distribution, which they use in an index called HD-D. $^{1}$  HD-D measures the mean contribution that each type makes to the TTR of all possible combinations of a sample of size 35-50, and higher HD-D indicates greater lexical diversity
3. Measure of Textual Lexical Diversity (MTLD): this more complicated measure of lexical diversity measures the mean length of word strings that maintain a criterion level of lexical variation. See McCarthy (2005) for details.
4. Parse tree depth: dependency parse tree depth using spaCy (Honnibal and Montani, 2017).
5. Sentence length: number of words in a sentence.
6. Dale-Chall readability score (Dale and Chall, 1948, 1995): DCRS = 0.1579(DWR  $\cdot$  100) + 0.0496WPS, where DWR is the ratio of difficult words² and WPS is the average words per sen
tence. Texts with higher DCRS are supposed to be more challenging to read.
7. Coleman-Liau index (Coleman and Liau, 1975):  $\mathrm{CLI} = 0.0588L - 0.296S - 15.8$ , where  $L$  is the average number of letters per 100 words and  $S$  is the average number of sentences per 100 words. CLI aims to approximate the U.S. grade level thought necessary to comprehend the text.
8. Flesch-Kincaid grade level (Kincaid et al., 1975):  $\mathrm{FKGL} = 0.39\mathrm{WPS} - 11.8\mathrm{SPW} - 15.59$ , where WPS is the average words per sentence and SPW is the average syllables per word. Higher scores indicate material that is more challenging to read.
9. Age of acquisition (AoA): the average AoA of words was extracted from a database of crowdsourced ratings of over 30 thousand words (Kupperman et al., 2012). For instance, potty has an AoA of 2.28, and blasphemous has an AoA of 11.25.
10. Concreteness: averaged word-level concreteness ratings on the scale from 1-5 (1 is most abstract, and 5 is most concrete) for 40 thousand English lemmas (Brysbaert et al., 2014). For instance, spirituality is rated 1.07, and scarf is rated 4.97.
11. Syllable count: average number of syllables per word, as computed using pyphen: https://pyphen.org/
12. Talkativeness: number of alphanumeric tokens for either client or therapist in a conversation (all messages in one week period before a survey).
8.2 Linear Mixed Model Analysis
As the anxiety of clients can correlate with many variables, we use a linear mixed model (Galecki and Burzykowski, 2013) (sometimes called multilevel or hierarchical models), which is a regression model that accounts for both fixed effects (variation that is explained by independent variables of interest) and random effects (variation that is not explained by independent variables of interest). In this subsection, we show the expressions for the linear mixed models we use in §4. For each linguistic complexity measure  $m$ , our fixed effects include client linguistic complexity  $\mathcal{C}_C^m$ , therapist linguistic complexity  $\mathcal{C}_T^m$ , time (weeks in therapy)  $t$ , client
1See https://textinspector.com/help/ lexical-diversity/ for McCarthy's recommendation on vocd-D vs HD-D. 2Words not on a list of 3,000 familiar words at https: //www.readabilityformulas.com/articles/ dale-chall-readability-word-list.php
age  $a_{C}$ , client gender  $g_{C}$ , and client education  $e_{C}$  and our random effects include clients with respect to time  $(1 + t|C)$ , and therapists  $(1|T)$ .
For computing correlation between client anxiety and a linguistic complexity variable of interest  $\mathcal{C}' \in \{\mathcal{C}_C^m, \mathcal{C}_T^m, (\mathcal{C}_C^m - \mathcal{C}_T^m), |\mathcal{C}_C^m - \mathcal{C}_T^m|\}$  For computing correlations between linguistic complexity and client anxiety (Figure 1), we use
$$
\begin{array}{c} \text {a n x i e t y} \sim \mathcal {C} ^ {\prime} + t + a _ {C} + g _ {C} + e _ {C} \\ + (1 + t | C) + (1 | T). \end{array} \tag {1}
$$