SlowGuess/ACL_2020_OCR huggingface dataset:
Vinodkumar Prabhakaran, Camilla Griffiths, Hang Su, Prateek Verma, Nelson Morgan, Jennifer L. Eberhardt, and Dan Jurafsky. 2018. Detecting Institutional Dialog Acts in Police Traffic Stops. Transactions of the Association for Computational Linguistics.
Alan Ritter, Colin Cherry, and Bill Dolan. 2010. Unsupervised Modeling of Twitter Conversations. In Proceedings of NAACL.
Carl R. Rogers. 1957. The necessary and sufficient conditions of therapeutic personality change. Journal of Consulting Psychology.
Stephen Rollnick and William R. Miller. 1995. What is Motivational Interviewing? Behavioural and Cognitive Psychotherapy.
Amanda J. Rose, Wendy Carlson, and Erika M. Waller. 2007. Prospective Associations of Co-Rumination with Friendship and Emotional Adjustment: Considering the Socioemotional Trade-Offs of Co-Rumination. Developmental Psychology.
Sara Rosenthal and Kathleen McKeown. 2015. I Couldn't Agree More: The Role of Conversational Structure in Agreement and Disagreement Detection in Online Discussions. In Proceedings of SIGDIAL.
Harvey Sacks. 1992. Lectures on Conversation. Blackwell.
Jonathan Sandoval, Amy Nicole Scott, and Irene Padilla. 2009. Crisis counseling: An overview. *Psychology in the Schools*.
Emanuel A. Schegloff. 1987. Some sources of misunderstanding in talk-in-interaction. Linguistics.
Chenhao Tan, Vlad Niculae, Cristian Danescu-Niculescu, and Lillian Lee. 2016. Winning Arguments: Interaction Dynamics and Persuasion Strategies in Good-faith Online Discussions. In Proceedings of WWW.
Michael Tanana, Kevin A. Hallgren, Zac E. Imel, David C. Atkins, and Vivek Srikumar. 2016. A Comparison of Natural Language Processing Methods for Automated Coding of Motivational Interviewing. Journal of Substance Abuse Treatment.
Jenny Thomas. 1983. Cross-Cultural Pragmatic Failure. Applied Linguistics, (2).
Terence J. G. Tracey, Bruce E. Wampold, James W. Lichtenberg, and Rodney K. Goodyear. 2014. Expertise in psychotherapy: An elusive goal? The American Psychologist.
Marilyn Walker and Steve Whittaker. 1990. Mixed Initiative in Dialogue: An Investigation into Discourse Segmentation. In Proceedings of ACL.
Xuewei Wang, Weiyan Shi, Richard Kim, Yoojung Oh, Sijia Yang, Jingwen Zhang, and Zhou Yu. 2019. Persuasion for Good: Towards a Personalized Persuasive Dialogue System for Social Good. In Proceedings of ACL.
Bonnie Lynn Webber. 2001. Computational Perspectives on Discourse and Dialog. In The Handbook of Discourse Analysis. John Wiley & Sons, Ltd.
Harry Weger, Gina R. Castle, and Melissa C. Emmett. 2010. Active Listening in Peer Interviews: The Influence of Message Paraphrasing on Perceptions of Listening Skill. International Journal of Listening.
Justine Zhang, Robert Filbin, Christine Morrison, Jaclyn Weiser, and Cristian Danescu-Niculescu-Mizil. 2019. Finding Your Voice: The Linguistic Development of Mental Health Counselors. In Proceedings of ACL.
Justine Zhang, Ravi Kumar, Sujith Ravi, and Cristian Danescu-Niculescu-Mizil. 2016. Conversational Flow in Oxford-style Debates. In Proceedings of NAACL.
Justine Zhang, Arthur Spirling, and Cristian Danescu-Niculescu-Mizil. 2017. Asking too Much? The Rhetorical Role of Questions in Political Discourse. In Proceedings of EMNLP.
A.1 Further Details About Methodology
Here, we provide further details on our methodology for measuring orientation, to supplement the description in Section 4.2 and aid reproducibility.
Our aim in the first part of our methodology is to measure the orientation of phrasings  $\Omega_w$ . We would like to ensure that our measure is not skewed by the relative frequencies of phrasings, and take two steps to this ends, which empirically produced more interpretable output. First, we scale rows of term-document matrix  $\mathcal{X}$  (corresponding to texter phrasings) to unit  $\ell_2$  norm before deriving their representation in  $\mathbb{T}$  via singular value decomposition. Second, we remove the first SVD dimension, i.e., first column of  $U$ , and renormalize each row, before proceeding.
A.2 Further Details About Application to Counseling Data
Here, we provide further details on how we applied our methodology to the dataset of counseling conversations in order to measure the orientation of counselor utterances, as described in Section 5. In particular, we list empirical choices
Figure 6: Mean naive distance, backwards-range  $(\overleftarrow{\sigma})$ , and  $\%$  of utterances with questions, per segment for risk-assessed (orange) and non-risk-assessed (black) conversations; solid circles indicate statistically significant differences (Wilcoxon  $p < 0.01$ , comparing conversation types within counselor).
made in extracting and then processing the training set of 351,935 texter and counselor messages used to measure phrasing orientations.
We randomly sampled  $20\%$  of counselors in the data; all conversations by these counselors were omitted in subsequent analyses. We merged consecutive messages from the same interlocutor. To mitigate potential noise in characterizing phrasings, we considered only counselor and texer message pairs in which each message has between 15 and 45 words. We extracted all messages from the conversations which met these constraints.
We represent counselor phrasings as dependency-parse arcs and texter messages as unigrams, reflecting the comparatively structured language of the counselors versus the texters (counselors are instructed to speak in grammatically well-formed sentences). We consider the 5000 most frequent phrasings for each role, and discard sentences without any such phrasings. Finally, we used 25 SVD dimensions to induce  $\mathbb{T}$
A.3 Full Listing of Counselor Action Labels
Table 2 lists each counseling action derived from the training material and used during the validation procedure (Section 5.1) to label sentences.
A.4 Orientation and Lexical Properties
Here, we supplement our discussion of simple lexical properties that could be used to characterize messages (Section 5.3), discussing how orientation reflects these properties and showing that orientation is not subsumed by them.
Backwards-range. As seen in their weak backwards-range (high, i.e., spread-out  $\left\langle \overline{\sigma}\right\rangle$ ), affirmations that highlight the texer's strengths can follow a variety of situations. However, the replies they prompt are yet more diffuse, emphasizing the need to compare both directions.
reflection (113)re-wording to show understanding and validate feelingsIt can be overwhelming to go through that on your own.
affirmation (60)pointing out the texter's positive qualities and actionsYou showed a lot of strength in reaching out to us.
exploration (44)prompting texters to expand on their situationIs this the first real fight you've had with your boyfriend?
problem solving (110)identifying the texter's goals and potential coping skillsWhat do you usually do to help you feel calmer?
closing (43)reviewing the conversation and transitioning to a closeI think you have a good plan to get some rest tonight.
risk assessment (19)assessing suicidal ideation or risk of self-harmDo you have access to the pills right now?
Table 2: Counseling strategies and representative examples derived from the training material. The number of sentences (out of 400) assigned to each label is shown in parentheses (11 were not labeled as any action).
Question-asking. We see that questions—which nominally prompt the texter for a response—are more forwards-oriented than non-questions;  $61\%$  of sentences with ‘?’ have  $\Omega > 0$  compared to  $21\%$  of sentences without. However, these numbers also show that explicitly-marked questions are inexact proxies of forwards-oriented sentences—as in Table 1, questions can address a past remark by prompting clarifications, while counselors can use non-questions to suggest an intent to advance stages (e.g., to transition to problem-solving).
A.5 Relating Alternate Measures to Conversation Progress
Figure 6 shows averages per conversation segment (i.e.,  $20\%$  of a conversation) for each alternative measure considered in Section 5.3. Comparing to the average  $\Omega^{\mathrm{max}}$  and  $\Omega^{\mathrm{min}}$  shown in Figure 4, we see that these measures track with the conversa
tion's progress differently, and none of them distinguish the initial dynamics of risk-assessed conversations as dramatically as reflected in  $\Omega^{\mathrm{max}}$ , e.g., simple counts of questions do not distinguish between questions geared towards risk-assessment versus more open-ended problem exploration.
A.6 Further Details About Data Used in Analyses
Here, we provide further details about the subset of data we used to analyze counselors' orientation behavior (Section 5.4). In particular, our aim was to characterize behavior in typical conversations rather than exceptional cases or those that reflected earlier versions of the training curriculum. As such, we only considered the 234,433 conversations which had least five counselor messages, were not risk-assessed or disconnected before completion, and were taken by counselors who joined the platform after January 2017.
A.7 Counselor-Level Analysis
Here, we provide further details about our procedure for analyzing counselor-level correspondences between orientation and effectiveness indicators, as alluded to in Section 5.4.
Recall that our conversation-level findings may be confounded by texter idiosyncracies: for instance, texters with particularly difficult situations might affect a counselor's behaviour, but may also be more likely to give bad ratings, independent of how the counselor behaves. Alternatively, an overly long conversation could arise because the counselor is less forwards-oriented, or because the texter is reluctant to make progress from the outset, making it hard for the counselor to attempt to prompt them forwards.
To separate a counselor's decisions from these situational effects, we take a counselor-level perspective. While counselors cannot selectively talk with different types of texters, they can exhibit cross-conversational inclinations for particular behaviors. We therefore relate these cross-conversational tendencies in orienting a conversation to a counselor's long-term propensity for receiving helpful ratings, or having long or short conversations. We proceed to describe our methodology for relating counselor tendencies to perceived helpfulness; an analogous procedure could be applied to conversation length as well.
We characterize a counselor's orienting behavior as the average  $\Omega^{\mathrm{max}}$  and  $\Omega^{\mathrm{min}}$  over the con
versations they take; we likewise take the proportion of their (rated) conversations which were perceived as helpful. We restrict our counselor level analyses to the 20th to 120th conversations of the 1495 counselors who have taken at least 120 conversations (ignoring their initial conversations when they are still acclimatizing to the platform).
To cleanly disentangle counselor tendency and conversational circumstance, we split each counselor's conversations into two interleaved subsets (i.e., first, third, fifth ... versus second, fourth ... conversations), measuring orientation on one subset and computing a counselor's propensity for helpful ratings on the other. Here, we draw an analogy to the machine learning paradigm of taking a train-test split: "training" counselor tendencies on one subset and "testing" their relation to rating on the other subset. In general, the directions of the effects we observe hold with stronger effects if we do not take this split.
Echoing conversation-level effects, counselors that tend to be less forwards-oriented and more backwards-oriented (those in the bottom thirds of  $\Omega^{\mathrm{max}}$  and  $\Omega^{\mathrm{min}}$  respectively) are more likely to be perceived as helpful; this contrast is stronger in terms of  $\Omega^{\mathrm{min}}$  (Cohen's  $d = 0.30$ ,  $p < 0.001$ ) than  $\Omega^{\mathrm{max}}$  ( $d = 0.13$ ,  $p < 0.05$ ), suggesting that a counselor's tendency for advancing weighs less on their perceived helpfulness than their tendency for addressing. Also in line with the conversation-level findings, counselors with smaller  $\Omega^{\mathrm{max}}$  tend to have longer conversations ( $d = 0.54$ ,  $p < 0.001$ ), as do counselors with larger  $\Omega^{\mathrm{min}}$  ( $d = 0.17$ )—here, a counselor's tendency for advancing is more related to their propensity for shorter conversations than their tendency for addressing.
We note that counselors on the platform cannot selectively take conversations with certain texters; rather, the platform automatically assigns incoming texters to a counselor. As such, the counselor-level effects we observe cannot be explained by counselor self-selection for particular situations.
A.8 Orientation in Multi-Sentence Utterances
Our motivation in characterizing utterances using the minimum and maximum sentence orientation was to reflect potential heterogeneities in utterances which could be both backwards- and forwards-oriented (consider a message where  $c_{2}$  and  $c_{1}$  from Figure 1 are concatenated). Examin
Orientation	Example phrasings	Example sentences
Less forwards-oriented (bottom 25%)	i understand, have been, part of, so you, sentence, talking about might, particular but the, give to	As I understand the facts [...] he had tried to kill the husband, shooting him twice in the head? (Scalia)You started out by talking about what the first jury knew, but [...] we aren’t reviewing that determination. (Roberts)I guess the problem is the list of absurdities that they point to, not the least of which is a dry dock. (Sotomayor)So you hedged, because it’s very hard to find the right sentence. (Breyer)
More forwards-oriented (top 25%)	hypothetical, would have, agree, difference [between], [do] you think, your position your argument, a question apply, was there	Suppose under this hypothetical [...] the judge doesn’t say aggravated murder when he submits it to the jury. (Kennedy)I just want to know your position on the second, the cart before the horse point. (Souter)Do you also agree [...] that if not properly administered there is some risk of excruciating pain? (Stevens)What’s the difference between pigment and color [...] ? (Ginsburg)
Table 3: Example phrasings and sentences from utterances of Supreme Court justices, identified in parentheses, which are less or more forwards-oriented (bottom and top  $25\%$  of  $\Omega$ ).
ing the  $64\%$  of counselor messages with multiple sentences, we see that  $52\%$  of these messages have  $\Omega^{\mathrm{min}} < 0$  and  $\Omega^{\mathrm{max}} > 0$ . Our method, which is able to account for this heterogeneity, thus points to one potential strategy for counselors to bridge between both objectives.
A.9 Application to Supreme Court Oral Arguments
Here, we include an exploratory study of how our approach could be adapted to analyze domains beyond crisis counseling conversations, as alluded to in Section 6. We apply the method to measure the orientation of utterances by Supreme Court justices during oral arguments, when they engage in exchanges with lawyers (so justices and lawyers play the roles of counselor and texter, respectively, in our method). We used transcripts of 668 cases, taken from the Oyez project (https://www.oyez.org/), averaging 120 justice utterances per case.[11]
Oral arguments contain more linguistic and topical heterogeneity than counseling conversations, since they cover a wide variety of cases, and because the language used by each justice is more differentiated. In addition, the dataset is much smaller. As such, this represents a more challenging setting than the counseling context, requiring changes to the precise procedure used to measure orientation, and pointing to the need for further technical improvements, discussed in Section 6.
Nonetheless, our present methodology is able to produce interpretable output. Table 3 shows representative phrasings and (paraphrased) sentences with different orientations. In contrast to the coun
seling domain,  $70\%$  of phrasings and  $93\%$  of sentences have  $\Omega > 0$ , perhaps reflecting the particular power dynamic in the Supreme Court, where justices are tasked with scrutinizing the arguments made by lawyers. We find that highly forwards-oriented phrasings and utterances tend to reflect justices pressing on the lawyers to address a point (e.g., do you agree, what's the difference between); the least forwards-oriented phrases involve the justice rehashing and reframing (not always in complimentary terms) a lawyer's prior utterances (e.g., so you [...] [as] i understand).
We used a training set of 15,862 justice and lawyer messages, where each utterance had between 10 and 100 words. Both lawyer and justice utterances were represented as dependency-parse arcs. Empirically, we found that the methodology was sensitive to idiosyncracies of particular cases and justices. To minimize this effect, we restricted the size of the justice's vocabulary by only considering the 398 justice phrasings which occurred in at least 200 utterances.
11The data used can be found at http://analysmith. com/research/scotus/data.