SlowGuess/ACL_2020_OCR huggingface dataset:
Justine Zhang
jz727@cornell.edu
Cristian Danescu-Niculescu-Mizil
cristian@cs.cornell.edu
Throughout a conversation, participants make choices that can orient the flow of the interaction. Such choices are particularly salient in the consequential domain of crisis counseling, where a difficulty for counselors is balancing between two key objectives: advancing the conversation towards a resolution, and empathetically addressing the crisis situation.
In this work, we develop an unsupervised methodology to quantify how counselors manage this balance. Our main intuition is that if an utterance can only receive a narrow range of appropriate replies, then its likely aim is to advance the conversation forwards, towards a target within that range. Likewise, an utterance that can only appropriately follow a narrow range of possible utterances is likely aimed backwards at addressing a specific situation within that range. By applying this intuition, we can map each utterance to a continuous orientation axis that captures the degree to which it is intended to direct the flow of the conversation forwards or backwards.
This unsupervised method allows us to characterize counselor behaviors in a large dataset of crisis counseling conversations, where we show that known counseling strategies intuitively align with this axis. We also illustrate how our measure can be indicative of a conversation's progress, as well as its effectiveness.
Participants in a conversation constantly shape the flow of the interaction through their choices. In psychological crisis counseling conversations, where counselors support individuals in mental distress, these choices arise in uniquely complex and high-stakes circumstances, and are reflected in rich conversational dynamics (Sacks, 1992). As such, counseling is a valuable context for computationally modeling conversational behavior (Atkins
Figure 1: Two possible exchanges in a counseling conversation, illustrating key objectives that a counselor must balance:  $c_{1}$  aims to advance the conversation towards a discussion of possible confidants;  $c_{2}$  aims to address the emotion underlying the preceding utterance.
et al., 2014; Althoff et al., 2016; Pérez-Rosas et al., 2018; Zhang et al., 2019). Modeling the conversational choices of counselors in this endeavor is an important step towards better supporting them.
Counselors are driven by several objectives that serve the broader goal of helping the individual in distress; two key objectives are exemplified in Figure 1. The counselor must advance a conversation towards a calmer state where the individual is better equipped to cope with their situation (Mishara et al., 2007; Sandoval et al., 2009): in  $c_{1}$ , the counselor prompts the individual to brainstorm options for social support. The counselor must also empathetically address what was already said, "coming to an empathic understanding" of the individual (Rogers, 1957; Hill and Nakayama, 2000): in  $c_{2}$ , the counselor validates feelings that the individual has just shared.
Balancing both objectives is often challenging, and overshooting in one direction can be detrimental to the conversation. A counselor who leans too much on advancing forwards could rush the conversation at the expense of establishing an empathetic connection; a counselor who leans too much backwards, on addressing what was already said, may fail to make any progress.
These examples are derived from material used to train counselors in our particular setting, detailed in Section 2.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5276-5289 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
In this work, we develop a method to examine counselor behaviors as they relate to this balancing challenge. We quantify the relative extent to which an utterance is aimed at advancing the conversation, versus addressing existing content. We thus map each utterance onto a continuous backwards-forwards axis which models the balance of these objectives, and refer to an utterance's position on this axis as its orientation.
At an intuitive level, our approach considers the range of content that is expected to follow or precede a particular utterance. For an utterance like  $c_{1}$  that aims to advance the conversation towards an intended target, we would expect a narrow range of appropriate replies, concentrated around that target (e.g., suggestions of possible confidants). We would likewise expect an utterance like  $c_{2}$  that aims to address a previously-discussed situation to only be an appropriate reply for a narrow range of possible utterances, concentrated around that specific type of situation (e.g., disclosures of negative feelings). Starting from this intuition, we develop an unsupervised method to quantify and compare these expected forwards and backwards ranges for any utterance, yielding our orientation measure.
Using this measure, we characterize counselor behaviors in a large collection of text-message conversations from a crisis counseling service, which we accessed in collaboration with the service and with the participants' consent. We show how orientation meaningfully distinguishes between key conversational strategies that counselors are taught during their training. We also show that our measure tracks a conversation's progress and can signal its effectiveness, highlighting the importance of balancing the advancing and addressing objectives, and laying the basis for future inquiries in establishing potential causal effects.
In summary, we develop an unsupervised methodology that captures how counselors balance the conversational objectives of advancing and addressing (Section 4), apply and validate it in a large dataset of counseling conversations (Section 5), and use it to investigate the relation between a counselor's conversational behavior and their effectiveness (Section 5.4). While our method is motivated by a salient challenge in counseling, we expect similar balancing problems to recur in other conversational settings where participants must carefully direct the flow of the interaction, such as court trials and debates (Section 6).
2 Setting: Counseling Conversations
We develop our method in the context of Crisis Text Line, a crisis counseling platform which provides a free 24/7 service for anyone in mental crisis—henceforth texters—to have one-on-one conversations via text message with affiliated counselors. We accessed a version of this collection, with over 1.5 million conversations, in collaboration with the platform and with the consent of the participants. The data was scrubbed of personally identifiable information by the platform. These conversations are quite substantive, averaging 25 messages with 29 and 24 words per counselor and texter message, respectively.
In each conversation, a crisis counselor's high-level goal is to guide the texter towards a calmer mental state. In service of this goal, all counselors first complete 30 hours of training provided by the platform, which draws on past literature in counseling to recommend best practices and conversational strategies. The first author also completed the training to gain familiarity with the domain.
While the platform offers guidance to counselors, their task is inevitably open-ended, given the emotional complexity of crisis situations. As such, the counselors are motivated by an explicit goal that structures the interaction, but they face a challenging flexibility in choosing how to act.
We now describe the conversational challenge of balancing between advancing the conversation forwards or addressing what was previously said. Our description of the challenge and our computational approach to studying it are informed by literature in counseling, on the platform's training material and on informal interviews with its staff.
A conversational balance. A crisis counselor must fulfill multiple objectives in their broader goal of helping a texter. One objective is guiding the texter through their initial distress to a calmer mental state (Mishara et al., 2007; Sandoval et al., 2009), as in Figure 1,  $c_{1}$ . Various strategies that aim to facilitate this advancing process are taught to counselors during training: for instance, a counselor may prompt a texter to identify a goal or cop
2The data can be accessed by applying at https:// www.crisistextline.org/data-philosophy/ data-fellows/. The extensive ethical and privacy considerations, and policies accordingly implemented by the platform, are detailed in Pisani et al. (2019).
ing mechanism (Rollnick and Miller, 1995). As such, they attempt to move the conversation forwards, towards its eventual resolution.
The counselor must also engage with the texter's concerns (Rogers, 1957; Hill and Nakayama, 2000), as in  $c_{2}$ , via strategies that empathetically address what the texter has already shared (Rollnick and Miller, 1995; Weger et al., 2010; Bodie et al., 2015). For instance, counselors are taught to reflect, i.e., reframe a texter's previous message to convey understanding, or draw on what was said to affirm the texter's positive qualities. In doing so, the counselor looks backwards in the conversation.
Past work has posited the benefits of mixing between strategies that aim at either objective (Mishara et al., 2007). However, as the training acknowledges, striking this balance is challenging. Overzealousy seeking to advance could cut short the process of establishing an empathetic connection. Conversely, focusing on the conversation's past may not help with eventual problem solving (Bodie et al., 2015), and risks stalling it. A texter may start to counterproductively rehash or ruminate on their concerns (Nolen-Hoeksema et al., 2008; Jones et al., 2009); indeed, prior psychological work has highlighted the thin line between productive reflection and rumination (Rose et al., 2007; Landphair and Preddy, 2012).
Orientation. To examine this balancing dynamic, we model the choices that counselors make at each turn in a conversation. Our approach is to derive a continuous axis spanned by advancing and addressing. We refer to an utterance's position on this axis, representing the relative extent to which it aims at either objective, as its orientation  $\Omega$ . We interpret a forwards-oriented utterance with positive  $\Omega$  as aiming to advance the conversation, and a backwards-oriented utterance with negative  $\Omega$  as aiming to address what was previously brought up. In the middle, the axis reflects the graded way in which a counselor can balance between aims—for instance, using something the texter has previously said to help motivate a problem-solving strategy.
Related characterizations. While we develop orientation to model a dynamic in counseling, we view it as a complement to other characterizations of conversational behaviors in varied settings.
Prior work has similarly considered how utterances relate to the preceding and subsequent discourse (Webber, 2001). Frameworks like centering theory (Grosz et al., 1995) aim at identify
ing referenced entities, while we aim to more abstractly model interlocutor choices. Past work has also examined how interlocutors mediate a conversation's trajectory through taking or ceding control (Walker and Whittaker, 1990) or shifting topic (Nguyen et al., 2014); Althoff et al. (2016) considers the rate at which counselors in our setting advance across stages of a conversation. While these actions can be construed as forwards-oriented, we focus more on the interplay between forwards- and backwards-oriented actions. A counselor's objectives may also cut across these concepts: for instance, the training stresses the need for empathetic reflecting across all stages and topics.
Orientation also complements prior work on dialogue acts, which consider various roles that utterances play in discourse (Mann and Thompson, 1988; Core and Allen, 1997; Ritter et al., 2010; Bracewell et al., 2012; Rosenthal and McKeown, 2015; Prabhakaran et al., 2018; Wang et al., 2019). In counseling settings, such approaches have highlighted strategies like reflection and question-asking (Houck, 2008; Gaume et al., 2010; Atkins et al., 2014; Can et al., 2015; Tanana et al., 2016; Pérez-Rosas et al., 2017, 2018; Park et al., 2019; Lee et al., 2019; Cao et al., 2019). Instead of modeling a particular taxonomy of actions, we model how counselors balance among the underlying objectives; we later relate orientation to these strategies (Section 5). Most of these approaches use annotations or predefined labeling schemes, while our method is unsupervised.
4 Measuring Orientation
We now describe our method to measure orientation, discussing our approach at a high level before elaborating on the particular operationalization. The code implementing our approach is distributed as part of the ConvoKit library (Chang et al., 2020), at http://convokit.cornell.edu.
4.1 High-Level Sketch
Orientation compares the extent to which an utterance aims to advance the conversation forwards with the extent to which it looks backwards. Thus, we must somehow quantify how the utterance relates to the subsequent and preceding interaction.
Naive attempt: direct comparison. As a natural starting point, we may opt for a similarity-based approach: an utterance that aims to address its preceding utterance, or predecessor, should be similar
Figure 2: Words representative of replies and predecessors for utterances with two example phrasings, as observed in training data. Top row: observed replies to utterances with  $w_{1}$  span a narrower range than observed predecessors (relative sizes of red and blue circles);  $w_{1}$  thus has smaller forwards-range  $\overrightarrow{\sigma_{w_1}}$  than backwards-range  $\overleftarrow{\delta_{w_1}}$  (i.e., it is forwards-oriented,  $\Omega_{w_1} > 0$ ). Bottom row: observed predecessors to utterances with  $w_{2}$  span a narrower range than replies;  $w_{2}$  thus has smaller  $\overleftarrow{\delta_{w_2}}$  than  $\overrightarrow{\sigma_{w_2}}$  (i.e., it is backwards-oriented  $\Omega_{w_2} < 0$ ).
to it; an utterance that aims to advance the conversation should be similar to the reply that it prompts. In practice, having to make these direct comparisons is limiting: an automated system could not characterize an utterance in an ongoing conversation by comparing it to a reply it has yet to receive.
This approach also has important conceptual faults. First, addressing preceding content in a conversation is different from recapitulating it. For instance, counselors are instructed to reframe rather than outright restate a texter's message, as in Figure 1,  $c_{2}$ . Likewise, counselors need not advance the conversation by declaring something for the texter to simply repeat; rather than giving specific recommendations, counselors are instructed to prompt the texters to come up with coping strategies on their own, as in  $c_{1}$ . Further, texters are not bound to the relatively formal linguistic style counselors must maintain, resulting in clear lexical differences. Measuring orientation is hence a distinct task from measuring similarity.
Second, an utterance's intent to advance need not actually be realized. A counselor's cues may be rebuffed or misunderstood (Schegloff, 1987; Thomas, 1983): a texter could respond to  $c_{1}$  by continuing to articulate their problem with  $t_{2}$ . Likewise, a counselor may intend to address a texter's concerns but misinterpret them. To model the balance in objectives that a counselor is aiming for, our characterization of an utterance cannot be contingent on its actual reply and predecessor.
Our approach: characterizing expectations. We instead consider the range of replies we might expect an utterance to receive, or the range of pre
deceessors that it might follow. Intuitively, an utterance with a narrow range of appropriate replies aims to direct the conversation towards a particular target, moreso than an utterance whose appropriate replies span a broader range.<sup>3</sup> Likewise, an utterance that is an appropriate reply to only a narrow range of possible predecessors aims to address a particular situation. We draw on unlabeled data of past conversations to form our expectations of these ranges, and build up our characterizations of utterances from their constituent phrasings, e.g., words or dependency-parse arcs.
The intuition for our approach is sketched in Figure 2. From our data, we observe that utterances containing confided to anyone generally elicited replies about potential confidants (e.g., sister, friend), while the replies that followed utterances with sounds frustrating span a broader, less well-defined range. As such, we have a stronger expectation of what a reply prompted by a new utterance with confided to anyone might contain than a reply to a new utterance with sounds frustrating. More generally, for each phrasing  $w$ , we quantify the strength of our expectations of its potential replies by measuring the range spanned by the replies it has already received in the data, which we refer to as its forwards-range  $\overrightarrow{\sigma_w}$ . We would say that confided to anyone has a smaller  $\overrightarrow{\sigma_w}$  than sounds frustrating, meaning that its observed replies were more narrowly concentrated; this is represented as the relative size of the red regions on the right side of Figure 2.
In the other direction, we observe in our data that sounds frustrating generally followed descriptions of frustrating situations (e.g., ignores, judges), while the range of predecessors to confided to anyone is broader. We thus have a stronger expectation of the types of situations that new utterances with sounds frustrating would respond to, compared to new utterances with confided to anyone. For a phrasing  $w$ , we quantify the strength of our expectations of its potential predecessors by measuring its backwards-range  $\overleftarrow{\sigma_w}$ , spanned by the predecessors we've observed. As such, sounds frustrating has a smaller  $\overleftarrow{\sigma_w}$  than confided to anyone, corresponding to the relative size of the blue regions on the left side of Figure 2.
3Consider leading versus open-ended questions. When people ask leading questions, they intend to direct the interaction towards specific answers they have in mind; when people ask open-ended questions, they are more open to what answers they receive and where the interaction is headed.
The relative strengths of our expectations in either direction then indicate the balance of objectives. If we have a stronger expectation of  $w$ 's replies than of its predecessors—i.e., smaller  $\overrightarrow{\sigma_w}$  than  $\overleftarrow{\sigma_w}$ —we would infer that utterances with  $w$  aim to advance the conversation towards a targeted reply more than they aim to address a particular situation. Conversely, if we have stronger expectations of  $w$ 's predecessors—i.e., smaller  $\overleftarrow{\sigma_w}$ —we would infer that utterances with  $w$  aim to address the preceding interaction, rather than trying to drive the conversation towards some target.
We thus measure orientation by comparing a phrasing's forwards- and backwards-range. The expectation-based approach allows us to circumvent the shortcomings of a direct comparison; we may interpret it as modeling a counselor's intent in advancing and addressing at each utterance (Moore and Paris, 1993; Zhang et al., 2017).
4.2 Operationalization
We now detail the steps of our method, which are outlined in Figure 3. Formally, our input consists of a set of utterances from counselors  $\{c_i\}$ , and a set of utterances from texters  $\{t_i\}$ , which we've observed in a dataset of conversations (Figure 3A). We note that each texter utterance can be a reply to, or a predecessor of, a counselor utterance (or both). We use this unlabeled "training data" to measure the forwards-range  $\overrightarrow{\sigma_w}$ , the backwards-range  $\overleftarrow{\sigma_w}$  (Figures 3B-D), and hence the orientation  $\Omega_w$  of each phrasing  $w$  used by counselors (Figure 3E). We then aggregate to an utterance-level measure.
For each counselor phrasing  $w$ , let  $\overrightarrow{T_w}$  denote the subset of texter utterances which are replies to counselor utterances containing  $w$  (Figure 3A). As described above, the forwards-range  $\overrightarrow{\sigma_w}$  quantifies the spread among elements of  $\overrightarrow{T_w}$ ; we measure this by deriving vector representations of these utterances  $\overrightarrow{U_w}$  (Figure 3B, detailed below), and then comparing each vector in  $\overrightarrow{U_w}$  to a central reference point  $\overrightarrow{u_w}$  (Figures 3C and 3D).4 Likewise,  $\overleftarrow{\underline{\sigma}_w}$  quantifies the similarity among elements of  $\overleftarrow{T_w}$ , the set of predecessors to counselor utterances with  $w$ ; we compute  $\overleftarrow{\underline{\sigma}_w}$  by comparing each corresponding vector in  $\overleftarrow{U_w}$  to a central point  $\overleftarrow{u_w}$ .
B. Derive vector representations of texter utterances  $\mathcal{X}^{\mathrm{SVD}}\approx USV^{T}$  where  $\mathcal{X}$  is a tfidf reweighted term-document matrix of all texter utterances
C. Derive central points
$$
\overrightarrow {u _ {w}} = \mathrm {w} _ {w} ^ {T} \overrightarrow {U _ {w}} S ^ {- 1}
$$
where  $\mathbf{w}_{\mathrm{w}}^{\mathrm{i}}$  is the tfidf weight of  $w$  in  $c_{i}$
Figure 3: Outline of steps to compute orientation  $\Omega_w$  of phrasing  $w$ , as described in Section 4.2. Panels A-D show the procedure for computing forwards-range  $\overrightarrow{\sigma_w}$ ; the procedure for backwards-range  $\overleftarrow{\sigma_w}$  is similar.
D. Compute
forwards-range
$$
\begin{array}{l} \overrightarrow {\tau_ {w}} = \underset {w \in \overrightarrow {U _ {w}}} {\operatorname {a v g}} (\bullet \xrightarrow {} \bullet) \\ \text {w h e r e} \bullet \xrightarrow {} \bullet \end{array}
$$
is the cosine distance between  $\vec{u_w}$  and  $u$
E. Compute orientation:
$\Omega_w = \overleftrightarrow{\sigma_w} - \overrightarrow{\sigma_w}$
Deriving vector representations (Figure 3B). To obtain vectors for each texter utterance, we construct  $\mathcal{X}$ , a tfidf reweighted term-document matrix where rows represent texter utterances and columns represent phrasings used by texers. To ensure that we go beyond lexical matches and capture conceptual classes (e.g., possible confidants, frustrating situations), we use singular value decomposition to get  $\mathcal{X} \approx USV^T$ . Each row of  $U$  is a vector representation  $u_i$  of utterance  $t_i$  in the induced low-dimensional space  $\mathbb{T}$ .  $\overrightarrow{U_w}$  then consists of the corresponding subset of rows of  $U$  (highlighted in Figure 3B).
Deriving central points (Figure 3C). For each  $w$ , we take the central point  $\overrightarrow{u_w}$  to be a weighted average of vectors in  $\overrightarrow{U_w}$ . Intuitively, a texter utterance  $t_i$  with vector  $u_i$  should have a larger contribution to  $\overrightarrow{u_w}$  if  $w$  is more prominent in the counselor utterance  $c_i$  that preceded it. We let  $\mathbf{w}_w^i$  denote the normalized tfidf weight of  $w$  in  $c_i$ , and use  $\mathbf{w}_w^i$  as the weight of the corresponding vector  $u_i$ . To properly map the resultant weighted sum  $\sum \mathbf{w}_w^i u_i$  into  $\mathbb{T}$ , we divide each dimension by the corresponding singular value in  $S$ . As such, if  $\mathbf{w}_w$  is a vector of weights  $\mathbf{w}_w^i$ , we can calculate the central point  $\overrightarrow{u_w^i}$
4Using a central reference point to calculate the forwardsrange, as opposed to directly computing pairwise similarities among replies in  $\overrightarrow{U_w}$  , allows us to account for the context of  $w$  in the utterances that prompted these replies (via tfidf weighting, as subsequently discussed).
of  $\overrightarrow{U_w}$  as  $\overrightarrow{u_w} = \mathbf{w}_w^T\overrightarrow{U_w} S^{-1}$ . In the other direction, we likewise compute  $\overleftarrow{u_w} = \mathbf{w}_w^T\overleftarrow{U_w} S^{-1}$ .