SlowGuess/ACL_2020_OCR huggingface dataset:
Forwards- and backwards-ranges (Figure 3D). We take the forwards-range  $\overrightarrow{\sigma_w}$  of  $w$  to be the average cosine distance from each vector in  $\overrightarrow{U_w}$  to the center point  $\overrightarrow{u_w}$ . Likewise, we take  $\overleftarrow{\sigma_w}$  as the average distance from each vector in  $\overleftarrow{U_w}$  to  $\overleftarrow{u_w}$ .
Phrasing-level orientation (Figure 3E). Importantly, since we've computed the forwards- and backwards-ranges  $\overrightarrow{\sigma_w}$  and  $\overleftarrow{\sigma_w}$  using distances in the same space  $\mathbb{T}$ , their values are comparable. We then compute the orientation of  $w$  as their difference:  $\Omega_w = \overleftarrow{\sigma_w} -\overrightarrow{\sigma_w}$ .
Utterance-level orientation. To compute the orientation of an utterance  $c_{i}$ , we first compute the orientation of each sentence in  $c_{i}$  as the tfidf weighted average  $\Omega_w$  of its constituent phrasings. Note that a multi-sentence utterance can orient in both directions—e.g., a counselor could concatenate  $c_{2}$  and  $c_{1}$  from Figure 1 in a single utterance, addressing the texter's previous utterance before moving ahead. To model this heterogeneity, we consider both the minimum and maximum sentence-orientations in an utterance:  $\Omega^{\mathrm{min}}$  captures the extent to which the utterance looks backwards, while  $\Omega^{\mathrm{max}}$  captures the extent to which it aims to advance forwards.
5 Application to Counseling Data
We apply our method to characterize messages from crisis counselors on the platform. We compute the orientations of the phrasings they use, represented as dependency-parse arcs. We use a training set of 351,935 texter and counselor messages each, from a random sample of conversations omitted in subsequent analyses. Table 1 shows representative phrasings and sentences of different orientations. Around two-thirds of phrasings and sentences have  $\Omega < 0$ , echoing the importance of addressing the texer's previous remarks.
In what follows, we analyze counselor behaviors in terms of orientation, and illustrate how the measure can be useful for examining conversations. We start by validating our method via two complementary approaches. In a subset of sentences manually annotated with the counseling
strategies they exhibit, we show that orientation meaningfully reflects these strategies (Section 5.1). At a larger scale, we show that the orientation of utterances over the course of a conversation aligns with domain knowledge about counseling conversation structure (Section 5.2). We also find that other measures for characterizing utterances are not as rich as orientation in capturing counseling strategies and conversation structure (Section 5.3). Finally, we show that a counselor's orientation in a conversation is tied to indicators of their effectiveness in helping the texter (Section 5.4).
5.1 Validation: Counseling Strategies
Even though it is computed without the guidance of any annotations, we expect orientation to meaningfully reflect strategies for advancing or addressing that crisis counselors are taught. The first author hand-labeled 400 randomly-selected sentences with a set of pre-defined strategies derived from techniques highlighted in the training material. We note example sentences in Table 1 which exemplify each strategy, and provide more extensive descriptions in the appendix.
Figure 4A depicts the distributions of orientations across each label, sorted from most backwards- to most forwards-oriented. We find that the relative orientation of different strategies corroborates their intent as described in the literature. Statements reflecting or affirming what the texer has said to check understanding or convey empathy (characterized by phrasings like totally normal) tend to be backwards-oriented; statements prompting the texer to advance towards problem-solving (e.g., [what] has helped) are more forwards-oriented. Exploratory queries for more information on what the texer has already said (e.g., happened to make) tend to have middling orientation (around 0). The standard deviation of orientations over messages within most of the labels is significantly lower than across labels (bootstrapped  $p < .05$ , solid circles), showing that orientation yields interpretable groupings of messages in terms of important counseling strategies.
The measure also offers complementary information. For instance, we find sentences that aren't accounted for by pre-defined labels, but still map to interpretable orientations, such as backwards-oriented examples assuaging texter concerns about the platform being a safe space to self-disclose.
$^{5}$ Further implementation details are listed in the appendix.  $^{6}$ Example sentences are derived from real sentences in the data, and modified for readability. The examples were chosen to reflect common situations in the data, and were vetted by the platform to ensure the privacy of counselors and texters.
Orientation	Example phrasings	Example sentences
Backwards-oriented (bottom 25%)	sounds frustrating, totally normal, great ways, on [your] plate, be overwhelming, sometimes feel frightening, on top [of] been struggling, feeling alone	You have a lot of things on your plate, between family and financial problems. [reflection]It's totally normal to feel lonely when you have no one to talk to. [reflection]Those are great ways to improve the relationship. [affirmation]
(middle 25%)	happened [to] make, mean [when you] say, is that, you recognized, source of the moment, are brave	Has anything happened to make you anxious? [exploration]It's good you recognized the need to reach out. [affirmation]Can you tell me what you mean when you say you're giving up? [risk assessment]
Forwards-oriented (top 25%)	plan for, confided [to] anyone, usually do, has helped, been talking, best support have considered, any activities	Can you think of anything that has helped when you've been stressed before? [problem solving]I want to be the best support for you today. [problem solving]We've been talking for a while now, how do you feel? [closing]
Table 1: Example phrasings and sentences with labeled strategies from crisis counselors' messages, at varying orientations: backwards-oriented (from the bottom  $25\%$  of  $\Omega$ ), middle, and forwards-oriented (from top  $25\%$ ).
Figure 4: Validating the orientation measure and comparing to alternatives. A Leftmost: Mean  $\Omega$  per counseling strategy label (vertical line denotes  $\Omega = 0$ ). Next three: same for other measures. B: Mean  $\Omega^{\mathrm{max}}$  and  $\Omega^{\mathrm{min}}$  per segment for risk-assessed (orange) and non-risk-assessed (black) conversations. Both: Solid circles indicate statistically significant differences (Wilcoxon  $p < 0.01$ , comparing within-counselor).
5.2 Validation: Conversation Structure
We also show that orientation tracks with the structure of crisis counseling conversations as described in the training material. Following Althoff et al. (2016), we divide each conversation with at least ten counselor messages into five equal-sized segments and average  $\Omega^{\mathrm{max}}$  and  $\Omega^{\mathrm{min}}$  over messages in each segment.
Figure 4B (black lines) shows that over the course of a conversation, messages tend to get more forwards-oriented (higher  $\Omega^{\mathrm{max}}$  and  $\Omega^{\mathrm{min}}$ ). This matches a standard conversation structure
taught in the training: addressing the texter's existing problems before advancing towards problem-solving. While this correspondence holds in aggregate, orientation also captures complementary information to advancement through stages—e.g., while problem-solving, counselors may still address and affirm a texter's ideas (Table 1, row 3).
We also consider a subset of conversations where we expect a different trajectory: for potentially suicidal texters, the training directs counselors to immediately start a process of risk assessment in which actively prompting the texter to disclose their level of suicidal ideation takes precedence over other objectives. As such, we expect more forwards-oriented messages at the starts of conversations involving such texters. Indeed, in the  $30\%$  of conversations which are risk-assessed, we find significantly larger  $\Omega^{\mathrm{max}}$  in the first segment (Figure 4B, orange line; Wilcoxon  $p < 0.01$  in the first stage, comparing within-counselor).  $\Omega^{\mathrm{min}}$  is smaller at each stage, suggesting that counselors balance actively prompting these critical disclosures with addressing them.
5.3 Alternative Operationalizations
We compare orientation to other methods for capturing a counselor's balancing decisions:
Naive distance. We consider the naive approach in Section 4, taking a difference in cosine distances between tfidf representations of a message and its reply, and a message and its predecessor.
Backwards-range. We consider just the message's backwards-range. For each sentence we take tfidf weighted averages of component  $\overleftarrow{\sigma_w}$  and take minimum  $\overleftarrow{\sigma}$  for each message.
<sup>7</sup>We get qualitatively similar results with maximum  $\vec{\sigma}$ .
Question-asking. We consider whether the message has a question. This was used in Walker and Whittaker (1990) as a signal of taking control, which could be construed as forwards-oriented.
Within-label standard deviations of each alternative measure are generally not significantly smaller than across-label (Figure 4A), indicating that these measures are poorer reflections of the counseling strategies. Label rankings under the measures are also arguably less intuitive. For instance, reflection statements have relatively large (naive) cosine distance from their predecessors. Indeed, the training encourages counselors to process rather than simply restate the texer's words.
These measures also track with the conversation's progress differently—notably, none of them distinguish the initial dynamics of risk-assessed conversations as reflected in  $\Omega^{\mathrm{max}}$  (see appendix).
5.4 Relation to Conversational Effectiveness
Past work on counseling has extensively discussed the virtues of addressing a client's situation (Rogers, 1957; Hill and Nakayama, 2000). Some studies also suggest that accounting for both addressing and advancing is important, such that effective counselors manage to mix backwards- and forwards-oriented actions (Mishara et al., 2007).
We use orientation to examine how these strategies are tied to conversational effectiveness in crisis counseling at a larger scale, using our measures to provide a unified view of advancing and addressing. To derive simple conversation-level measures, we average  $\Omega^{\mathrm{max}}$  and  $\Omega^{\mathrm{min}}$  over each counselor message in a conversation.
Adjudicating counseling conversation quality is known to be difficult (Tracey et al., 2014). As a starting point, we relate our conversation-level measures to two complementary indicators of a conversation's effectiveness:8
Perceived helpfulness. We consider responses from a post-conversation survey asking the texter whether the conversation was helpful, following Althoff et al. (2016). Out of the  $26\%$  of conversations with a response,  $89\%$  were rated as helpful. Conversation length. We consider a conversation's length as a simple indicator of the pace of its progress: short conversations may rush the texer, while prolonged conversations could suggest
Figure 5: Relation between orientation and conversational effectiveness. A: Mean  $\Omega^{\mathrm{min}}$  and  $\Omega^{\mathrm{max}}$  in conversations rated as helpful (green) or unhelpful (grey) (macroaveraged per conversation). Differences in both measures are significant (Mann Whitney U test  $p < 0.001$ ). B, C: Mean  $\Omega^{\mathrm{min}}$  and  $\Omega^{\mathrm{max}}$  of conversations with varying lengths (in # of messages). Both plots: Error bars show  $95\%$  bootstrapped confidence intervals.
stalling and could even demoralize the counselor (Landphair and Preddy, 2012).<sup>10</sup>
Figure 5A compares  $\Omega^{\mathrm{min}}$  and  $\Omega^{\mathrm{max}}$  in conversations rated as helpful and unhelpful by tex- ters. Both measures are significantly smaller in conversations perceived as helpful, suggesting that texers have a better impression of relatively backwards-oriented interactions where the coun-selor is inclined towards addressing their situation. As such, this result echoes past findings relating addressing to effectiveness.
Figure 5B compares  $\Omega^{\mathrm{min}}$  in conversations of varying lengths, showing that  $\Omega^{\mathrm{min}}$  increases with length, such that counselors exhibit less propensity for addressing in longer conversations. Anecdotal observations cited in interviews with the platform's staff suggest one interpretation: conversations in which a texter feels their concerns were not satisfactorily addressed may be prolonged when they circle back to revisit these concerns.
Figure 5C relates  $\Omega^{\mathrm{max}}$  to conversation length. We find that  $\Omega^{\mathrm{max}}$  is smaller in the lengthiest conversations, suggesting that such prolonged in
<sup>10</sup>As the training material notes, conversation length and texter perception may signal complementary or even conflicting information about a texter's experience of a conversation and its effectiveness: "Some texters resist the end of the conversation. They ruminate [...] causing the conversation to drag on without any progress."
<sup>8</sup>We perform all subsequent analyses on a subset of 234,433 conversations, detailed in the appendix.
<sup>9</sup>We note that this indicator is limited by important factors such as the selection bias in respondents.
teractions may be stalled by a weaker impulse to advance forwards. Extremely short conversations have smaller  $\Omega^{\mathrm{max}}$  as well, such that premature endings may also reflect issues in advancing. As such, we add credence to the previously posited benefits of mixing addressing and advancing: forwards-oriented actions may be tied to making progress, while a weaker propensity to advance may signal a suboptimal pace.
Counselor-level analysis. These findings could reflect various confounds—for instance, a counselor's choice of orientation may have no bearing on the rating they receive from a particularly difficult texer. To address this, we compute similar correspondences between orientation and our effectiveness indicators at the level of counselors rather than conversations; this analysis is detailed in the appendix. Our conversation-level results are replicated under these controls.
In this work, we sought to examine a key balance in crisis counseling conversations between advancing forwards and addressing what has already been said. Realizing this balance is one of the many challenges that crisis counselors must manage, and modeling the actions they take in light of such challenges could point to policies to better support them. For instance, our method could assist human supervisors in monitoring the progress of ongoing conversations to detect instances of rushing or stalling, or enable larger-scale analyses of conversational behaviors to inform how counselors are trained. The unsupervised approach we propose could circumvent difficulties in getting large-scale annotations of such sensitive content.
Future work could bolster the measure's usefulness in several ways. Technical improvements like richer utterance representations could improve the measure's fidelity; more sophisticated analyses could better capture the dynamic ways in which the balance of objectives is negotiated across many turns. The preliminary explorations in Section 5.4 could also be extended to gauge the causal effects of counselors' behaviors (Kazdin, 2007).
We expect balancing problems to recur in conversational settings beyond crisis counseling, such as court proceedings, interviews, debates and other mental health contexts like long-term therapy. In these settings, individuals also make potentially
consequential choices that span the backwards-forwards orientation axis, such as addressing previous arguments (Tan et al., 2016; Zhang et al., 2016) or asking leading questions (Leech, 2002). Our measure is designed to be broadly applicable, requiring no domain-specific annotations; we provide exploratory output on justice utterances from the Supreme Court's oral arguments in the appendix and release code implementing our approach at http://convokit.cornell.edu to encourage experiments in other domains. However, the method's efficacy in the present setting is likely boosted by the relative uniformity of crisis counseling conversations; and future work could aim to better accommodate settings with less structure and more linguistic variability. With such improvements, it would be interesting to study other domains where interlocutors are faced with conversational challenges.
We thank Jonathan P. Chang, Caleb Chiam, Liye Fu, Dan Jurafsky, Jack Hessel, and Lillian Lee for helpful conversations, and the anonymous reviewers for their thoughtful comments. We also thank Ana Smith for collecting and processing the Supreme Court oral argument transcripts we used in the supplementary material. This research, and the counseling service examined herein, would not have been possible without Crisis Text Line. We are particularly grateful to Robert Filbin, Christine Morrison, and Jaclyn Weiser for their valuable insights into the experiences of counselors and for their help with using the data. The research has been supported in part by NSF CAREER Award IIS1750615 and a Microsoft Research PhD Fellowship. The collaboration with Crisis Text Line was supported by the Robert Wood Johnson Foundation; the views expressed here do not necessarily reflect the views of the foundation.
Tim Althoff, Kevin Clark, and Jure Leskovec. 2016. Large-scale Analysis of Counseling Conversations: An Application of Natural Language Processing to Mental Health. Transactions of the Association for Computational Linguistics.
David C. Atkins, Mark Steyvers, Zac E. Imel, and Padhraic Smyth. 2014. Scaling up the evaluation of psychotherapy: Evaluating motivational interviewing fidelity via statistical text classification. Implementation Science.
Graham D. Bodie, Andrea J. Vickery, Kaitlin Cannava, and Susanne M. Jones. 2015. The Role of "Active Listening" in Informal Helping Conversations: Impact on Perceptions of Listener Helpfulness, Sensitivity, and Supportiveness and Discloser Emotional Improvement. Western Journal of Communication.
David Bracewell, Marc Tomlinson, and Hui Wang. 2012. Identification of Social Acts in Dialogue. In Proceedings of COLING.
Dogan Can, David C. Atkins, and Shrikanth S. Narayanan. 2015. A dialog act tagging approach to behavioral coding: A case study of addiction counseling conversations. In Proceedings of INTER-SPEECH.
Jie Cao, Michael Tanana, Zac Imel, Eric Poitras, David Atkins, and Vivek Srikumar. 2019. Observing Dialogue in Therapy: Categorizing and Forecasting Behavioral Codes. In Proceedings of ACL.
Jonathan P. Chang, Caleb Chiam, Liye Fu, Andrew Wang, Justine Zhang, and Cristian Danescu-Niculescu-Mizil. 2020. ConvoKit: A Toolkit for the Analysis of Conversations. In Proceedings of SIG-DIAL.
Mark G Core and James F Allen. 1997. Coding Dialogs with the DAMSL Annotation Scheme. AAAI fall symposium on communicative action in humans and machines.
Jacques Gaume, Nicolas Bertholet, Mohamed Faouzi, Gerhard Gmel, and Jean-Bernard Daeppen. 2010. Counselor motivational interviewing skills and young adult change talk articulation during brief motivational interventions. Journal of Substance Abuse Treatment.
Barbara J. Grosz, Aravind K. Joshi, and Scott Weinstein. 1995. Centering: A Framework for Modeling the Local Coherence of Discourse. Computational Linguistics.
Clara E. Hill and Emilie Y. Nakayama. 2000. Client-centered therapy: Where has it been and where is it going? A comment on Hathaway (1948). Journal of Clinical Psychology.
Jon Houck. 2008. Motivational Interviewing Skill Code (MISC) 2.1.
Neil P. Jones, Alison A. Papadakis, Caitlin M. Hogan, and Timothy J. Strauman. 2009. Over and over again: Rumination, reflection, and promotion goal failure and their interactive effects on depressive symptoms. Behaviour Research and Therapy.
Alan E. Kazdin. 2007. Mediators and mechanisms of change in psychotherapy research. Annual Review of Clinical Psychology.
Juliette Landphair and Teri Preddy. 2012. More than talk: Co-Rumination among college students. *About Campus*.
Fei-Tzin Lee, Derrick Hull, Jacob Levine, Bonnie Ray, and Kathy McKeown. 2019. Identifying therapist conversational actions across diverse psychotherapeutic approaches. In Proceedings of the Sixth Workshop on Computational Linguistics and Clinical Psychology.
Beth L. Leech. 2002. Asking Questions: Techniques for Semistructured Interviews. Political Science & Politics.
William C. Mann and Sandra A. Thompson. 1988. Rhetorical Structure Theory: Toward a functional theory of text organization. Text - Interdisciplinary Journal for the Study of Discourse.
Brian L. Mishara, François Chagnon, Marc S. Daigle, Bogdan Balan, Sylvaine Raymond, Isabelle Marcoux, Cecile Bardon, Julie K. Campbell, and Alan D. Berman. 2007. Which helper behaviors and intervention styles are related to better short-term outcomes in telephone crisis intervention? Results from a Silent Monitoring Study of Calls to the U.S. 1-800-SUICIDE Network. Suicide & life-threatening behavior.
Johanna D. Moore and Cecile Paris. 1993. Planning Text for Advisory Dialogues: Capturing Intentional and Rhetorical Information. Computational Linguistics.
Viet-An Nguyen, Jordan Boyd-Graber, Philip Resnik, Deborah A. Cai, Jennifer E. Midberry, and Yuanxin Wang. 2014. Modeling topic control to detect influence in conversations using nonparametric topic models. Machine Learning.
Susan Nolen-Hoeksema, Blair E. Wisco, and Sonja Lyubomirsky. 2008. Rethinking Rumination. Perspectives on Psychological Science.
Sungjoon Park, Donghyun Kim, and Alice Oh. 2019. Conversation Model Fine-Tuning for Classifying Client Utterances in Counseling Dialogues. In Proceedings of NAACL.
Verónica Pérez-Rosas, Rada Mihalcea, Kenneth Resnicow, Satinder Singh, and Lawrence An. 2017. Understanding and Predicting Empathic Behavior in Counseling Therapy. In Proceedings of ACL.
Verónica Pérez-Rosas, Xuetong Sun, Christy Li, Yuchen Wang, Kenneth Resnicow, and Rada Mihalcea. 2018. Analyzing the Quality of Counseling Conversations: The Tell-Tale Signs of High-quality Counseling. In Proceedings of LREC.
Anthony R. Pisani, Nitya Kanuri, Bob Filbin, Carlos Gallo, Madelyn Gould, Lisa S. Lehmann, Robert Levine, John E. Marcotte, Brian Pascal, David Rousseau, Shairi Turner, Shirley Yen, and Megan L. Ranney. 2019. Protecting User Privacy and Rights in Academic Data-Sharing Partnerships: Principles From a Pilot Program at Crisis Text Line. Journal of Medical Internet Research.