SlowGuess/NAACL_2021_OCR huggingface dataset:
Elisa Ferracane<sup>1</sup>, Greg Durrett<sup>2</sup>, Junyi Jessy Li<sup>1</sup>, and Katrin Erk<sup>1</sup>
$^{1}$ Department of Linguistics
$^{2}$ Department of Computer Science
elisa@ferracane.com, gdurrett@cs.utexas.edu
jessy@austin.utexas.edu, katrin.erk@mail.utexas.edu
Discourse signals are often implicit, leaving it up to the interpreter to draw the required inferences. At the same time, discourse is embedded in a social context, meaning that interpreters apply their own assumptions and beliefs when resolving these inferences, leading to multiple, valid interpretations. However, current discourse data and frameworks ignore the social aspect, expecting only a single ground truth. We present the first discourse dataset with multiple and subjective interpretations of English conversation in the form of perceived conversation acts and intents. We carefully analyze our dataset and create computational models to (1) confirm our hypothesis that taking into account the bias of the interpreters leads to better predictions of the interpretations, (2) and show disagreements are nuanced and require a deeper understanding of the different contextual factors. We share our dataset and code at http://github.com/elisaF/subjective_discourse.
Discourse, like many uses of language, has inherent ambiguity, meaning it can have multiple, valid interpretations. Much work has focused on characterizing these "genuine disagreements" (Asher and Lascarides, 2003; Das et al., 2017; Poesio et al., 2019; Webber et al., 2019) and incorporating their uncertainty through concurrent labels (Rohde et al., 2018) and underspecified structures (Hanneforth et al., 2003). However, prior work does not examine the subjectivity of discourse: how you resolve an ambiguity by applying your personal beliefs and preferences.
Our work focuses on subjectivity in question-answer conversations, in particular how ambiguities of responses are resolved into subjective assessments of the conversation act, a speech act in conversation (Traum and Hinkelman, 1992), and the communicative intent, the intention underly
Figure 1: Conflicting interpretations of conversation acts + intents for witness responses in a U.S. congressional testimony.
ing the act (Cohen and Perrault, 1979). We choose conversation acts (or more broadly, dialogue acts) as a challenge to the view that dialog act classification may be an "easy" task that has never been approached from a subjective perspective. Moreover, they are a good fit for our question-answering setting and are intuitive for naive annotators to understand. Our data consists of witness testimonials in U.S. congressional hearings. In Figure 1, annotators give conflicting assessments of responses given by the witness Mark Zuckerberg (CEO of Facebook) who is being questioned by Congressman Eliot Engel.
To make sense of our setting that has speakers (witness, politicians) and observers (annotators), we are inspired by the game-theoretic view of conversation in Asher and Paul (2018). The players (witness, politicians) make certain discourse moves in order to influence a third party, who is the judge of the game (the annotator). Importantly, the judge makes biased evaluations about the type of the player (e.g., sincere vs. deceptive), which leads to differing interpretations of the same response.
In our example, the two annotators are the biased judges with differing judgments on what type of player Zuckerberg is: the first assumes sincere and the second deceptive. For Zuckerberg's first response, the conversation act is interpreted un
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1626-1644 June 6-11, 2021. ©2021 Association for Computational Linguistics
ambiguously: both annotators agree he is signaling he can't answer the question. The intent, however, is ambiguous, where the cynical annotator interprets the clarification question as lying in order to stall vs. being honest. The second response yields both diverging conversation acts and intents: the first judge interprets the conversation act as an answer with the intent to provide a direct response, whereas the second judge perceives the conversation act as a shift to answer a different question with the intent to dodge the original, unfavorable question. We detail our full label set in Section 3.2.
We create the first discourse dataset with multiple, valid labels that are subjective. They do not hold concurrently and vary depending on the annotator; we collect annotator sentiments towards the conversants as a rough proxy for annotator bias. We further elicit annotator explanations for a window into their rationalization. A careful annotation protocol and qualification process ensure high quality crowd-sourced annotators with a strong understanding of the task. Our dataset contains 6k judgments over 1k question-response pairs, with disagreements in  $53.5\%$  of the data. However, unlike our prior example, disagreements are not often trivially attributable to differing sentiments. Uncooperative moves are sometimes warranted, regardless of annotator sentiment. Interpretation of a response is further influenced by its question. A qualitative analysis of annotator explanations reveals strikingly different uses of subjective language across diverging interpretations.
Identifying all the possible interpretations of a response is a useful way of analyzing discourse in a realistic setting with multiple observers, and could aid in uncovering sociolinguistic aspects relevant to variations in discourse comprehension. With these goals in mind, we propose the task of predicting the complete set of annotator labels for a given response. We find a transformer-based model outperforms other neural and linear models. We confirm our assumption that incorporating the context of the judge helps the model make better predictions, but still leaves room for improvement.
In summary, the task together with the dataset present a valuable opportunity to understand perceptions of discourse in a non-cooperative environment. More broadly, we show the need and value for considering the subjectivity of NLP tasks. Our work introduces a framework for identifying, elic
iting, and analyzing these subjective elements, to enable application for other tasks.
Asher and Paul (2016) apply their game-theoretic view of non-cooperative conversations to discourse moves in Segmented Discourse Representation Theory (Asher and Lascarides, 2003). Our work is applied instead to conversation acts and their communicative intents, which are more amenable to untrained annotators. Conversation acts are speech acts specific to conversation that can encompass entire turns in a conversation (Traum and Hinkelman, 1992). Speech act theory describes performative actions, i.e., how we can do things with words (Austin, 1962; Searle, 1969), but fails to account for how the act is perceived by an observer (the annotator in our scenario). Subsequent work in planning extends the theory to incorporate the cognitive context of an observer that includes the perceived communicative intent underlying a speech act (Cohen and Perrault, 1979; Pollack, 1986).
Speech act theory originally did not consider insincere speakers, but later work recognized that even in non-cooperative settings, conversants adhere to the conventions of dialogue, or discourse obligations, such as responding to a question (Traum and Allen, 1994; Potts, 2008). For this reason, we explicitly separate judgments on conversation acts (that usually fulfill a specific obligation) from communicative intents, which can be perceived as deceptive (or sincere).
Prior work examines how writer intentions are often misaligned with reader perceptions (Chang et al., 2020), which further motivates our focus on the reader (our annotator). While our work focuses on subjectivity, ambiguity is studied in many NLP tasks, including Natural Language Inference (Pavlick and Kwiatkowski, 2019; Nie et al., 2020), evaluation of NLG (Schoch et al., 2020), a recent SemEval 2021 shared task, as well as several discourse tasks (Asher and Lascarides, 2003; Versley, 2011; Webber and Joshi, 2012; Das et al., 2017; Poesio et al., 2019; Webber et al., 2019). Only one study strives to understand how these ambiguities are resolved: Scholman (2019) shows different interpretations of ambiguous coherence relations can be attributable to different cognitive biases. However, our work focuses more generally on subjec
1https://sites.google.com/view/ semeval2021-task12/home
tivity rather than cognitive processes.
Related NLP tasks include dialog act classification, intent detection, deception detection and argumentation, though we importantly note these predict only a single interpretation. Dialog acts are similar to conversation acts that apply at the utterance level. Classification models typically combine representations of linguistic units (word, utterance, conversation-level) (Chen et al., 2018). In our work, we employ a hierarchical model to account for the levels in our label taxonomy. Intent detection is traditionally applied to human-computer scenarios for task-specific goals such as booking a flight. Our conversation data is not task-oriented, and we thus define our intents more closely aligned with beliefs in the sincerity of the speaker. Detection of deception is, unlike many other NLP tasks, challenging even for humans (Ott et al., 2011). Most datasets consist of instructed lies (where participants are told to lie). Our work contains naturally-occurring deception where we include not just lying but other more covert mechanisms such as being deliberately vague or evasive (Clementson, 2018), both frequent in political discourse (Bull, 2008).
Argumentation mining analyzes non-cooperative conversations, but typically requires expert annotators. Recent work decomposes the task into intuitive questions for crowdsourcing (Miller et al., 2019), inspiring our annotation schemes that assume little to no training. Closer to our setting is argument persuasiveness, where Durmus and Cardie (2018) find prior beliefs of the audience play a strong role in their ability to be persuaded, which further motivates our focus on the annotator's bias.
We create the first dataset with multiple, subjective interpretations of discourse (summarized in Table 1). Recalling our example in Figure 1, we focus on responses to questions: the conversation act, how the response is perceived to address the question (such as Zuckerberg saying he cant_answer); and the communicative intent, the sincere or deceptive intent behind choosing that form of response (such as one annotator believing the intent was honest). As our source of data, we choose the question-answer portions of U.S. congressional hearings (all in English) for several reasons: they contain political and societal controversy identifiable by crowdsourced workers, they have a strong signal of ambiguity as to the form and intent of
item	#sents/ turn	#toks/ turn	total sents	total toks	total spkrs
question	4.1	81.5	4096	82582	91
response	2.6	47.0	2634	48831	20
Table 1: Statistics of our 20 U.S. congressional hearings.
the response, and the data is plentiful.2 A dataset statement is in Appendix D.
3.1 Dataset creation
Congressional hearings are held by committees to gather information about specific issues before legislating policies. Hearings usually include testimonies and interviews of witnesses. We focus on hearings that interview a single witness and that exceed a certain length ( $>100$  turns) as a signal of argumentative discourse. To ensure a variety of topics and political leanings are included, we sample a roughly equal number of hearings from 4 Congresses (113th-116th) that span the years 2013-2019, for a total of 20 hearings. For each hearing, we identify a question as a turn in conversation containing a question posed by a politician that is immediately followed by a turn in conversation from the witness, which is the response. We thus extract the first 50 question-response pairs from each hearing. Each data point consists of a question followed by a response. Table 1 summarizes the dataset statistics.
3.2 Dataset annotation
We collect labels through the Amazon Mechanical Turk crowdsourcing platform. In the task, we ask a series of nested questions feasible for untrained annotators (from which we derive question response labels), then elicit annotator sentiment. Each HIT consists of five question-response pairs in sequential order from the same hearing; we group them to preserve continuity of the conversation while not overloading the annotator. We collect 7 judgments for each HIT.<sup>3</sup> Screenshots of the task and the introductory example with all annotations are in Appendix A.
3.2.1 Annotations
For each question-response pair we collect three pieces of information: the question label, the re
${}^{2}$  Transcripts lack intonation and gestures, and thus a certain amount of information is lost from the original discourse.
3During our pilot, we experimented with increasing the number of judgments (up to 11) but found the number of chosen labels remains stable. We thus scaled back to 7.
(1) Q: How much of the financing was the Export-Import Bank responsible for?
(2) Q: If you were properly backing up information required under the Federal Records Act, which would include the information she deleted from the server, you'd have had all of those emails in your backup, wouldn't you?
(3) Q: So you're not willing to say that it doesn't meet due process requirements at this point?
R: We financed about $3 billion.
R: All emails are not official records under any official records act.
R: Well, what I'd like to do is look at the procedures in place.
Table 2: Different question forms lead to different responses: (1) an information-seeking question leads to a direct answer. (2) A loaded question with a presupposition and tag question leads to an indirect answer because the responder rejects the presupposition. (3) A declarative question where the questioner commits to an unfavorable view of the responder leads to an indirect answer.
sponse label, and an explanation. At the end of each HIT, we collect two pieces of information: the annotator's sentiment towards the questioners, and sentiment towards the witness.<sup>4</sup>
Question We collect judgments on the question as it can influence the response. For example, an objective, information-seeking question lends itself to a direct answer (Table 2 example (1)). A loaded question with presuppositions can instead result in an indirect answer when rejecting these presuppositions (Walton, 2003; Groenendijk and Stokhof, 1984), as in example (2) of Table 2. Leading questions, often asked as declarative or tag questions, are conducive to a particular answer (Bolinger, 1957) and signal the questioner is making a commitment to that underlying proposition. A pragmatic listener, such as our annotator, is inclined to believe the questioner has reliable knowledge to make this commitment (Gunlogson, 2008). Challenging the commitment leads to indirect answers as in example (3) of Table 2.
To elicit the question intent without requiring familiarity with the described linguistic concepts, we ask the annotator a series of intuitive questions to decide if the question is an attack on the witness, favoring the witness, or is neutral. We use a rule-based classifier to determine the question type (wh, polar, disjunctive, tag, declarative).
Response For judging the response, we combine conversation acts with communicative intents as in Figure 2, in the spirit of the compositional semantic framework of Govindarajan et al. (2019). The taxonomy is a result of a combination of expert involvement, data observation and user feedback.[5]
Figure 2: Hierarchical taxonomy of the perceived conversation act and intent for a response, forming the 6 response labels.
We next describe the taxonomy and its theoretical motivations.
In accordance with the discourse obligations of a conversation, a witness must respond in some form to a question (Traum and Allen, 1994). The function of the response is captured by the perceived conversation act, and is meant to be a more objective judgment (e.g., recognizing that Zuckerberg is using the 'can't answer' form of a response, regardless of whether you believe him). This conversation act constitutes the top layer of the taxonomy. The conversation acts include the standard answer and cant_answer. Inspired by work on answerhood (Ginzburg et al., 2019; de Marneffe et al., 2009; Groenendijk and Stokhof, 1984) and evasion in political discourse (Gabrielsen et al., 2017), we also include a more nuanced view of answering the question where giving a partial answer or answering a different question is labeled as shift.
The bottom layer of the taxonomy is the perceived intent underlying that conversation act, and is meant to be subjective. The intents hinge on whether the annotator believes the witness's conversation act is sincere or not. For answer, the annotator may believe the intent is to give a direct answer, or instead an overanswer with the in-
features in Plüss and Piwek (2016)), and researchers in the dialogue field to construct the initial taxonomy, then conducted internal pilots with linguists and non-linguists, and finally conducted several iterations of an external pilot with crowdworkers to further refine the taxonomy.
4We elicit sentiments at the end because we do not expect annotators to be familiar with the hearing or conversants. Future annotations could elicit sentiments at the beginning to capture strong a priori biases in high-profile hearings.
<sup>5</sup>We consulted with existing taxonomies (SWBD-DAMSL Jurafsky et al. (1997), MRDA Shriberg et al. (2004), DialogBank Bunt et al. (2018), evasive rhetorical strategies in Gabrielsen et al. (2017), dialogue acts paired with content
Figure 3: Distribution of response labels.
tent to sway the questioner (or even the public audience).<sup>6</sup> If shifting the question, the annotator may believe the responder is correcting the question (e.g., to reject a false presupposition) or is attempting to dodge the question. If the witness says they cant_answer, the annotator may believe the witness is honest or is lying.
The annotation task implements a series of nested questions that mimic the hierarchy of the label taxonomy, which we map to conversation act and intent labels. That is, we first ask how the witness responds to the question (conversation act), then what is the intent and combine these into a single response label.
Explanation We ask annotators for a free-form explanation of their choices in order to elicit higher quality labels (McDonnell et al., 2016) and for use in the qualifying task as explained later.
Sentiment At the end of the HIT, we ask the annotator to rate their sentiment towards the politicians and towards the witness on a 7-point scale (we later collapse these into 3 levels: negative, neutral, positive). These ratings provide a rough proxy for annotator bias.
3.2.2 Worker qualification
Because the task requires significant time and cognitive effort, we establish a qualification process. In the qualifying task, we include question-response pairs already explained in the instructions, and unambiguous cases as far as the conversation act (e.g., a response of 'Yes' can only be construed as an answer). The criteria for qualification are: correctly labeling the conversation act for the instruction examples and unambiguous cases, providing explanations coherent with the intent label, and response times not shorter than the reading time.
(a) all hearings
(b) Violations of Hatch Act
(c) Oversight of Dept. of Justice
Figure 4: Response label agreement (blue) and disagreement (between 2 or 3 labels in orange, green) for all hearings (a), a particular hearing with more (b) and less (c) disagreement.
Figure 5: Response label IAA on the disagreement subset for all annotators (left) increases when grouped by sentiment (right); dashed orange line is response IAA for entire dataset.
This rigorous process yielded high quality data from 68 annotators who were genuinely engaged with the task. On average, an annotator labeled 91 question-response pairs, with 4 superannotators who provided labels for half of the data. During post-processing, we consider a label valid if it receives more than one annotator vote. The annotated dataset consists of 1000 question-response pairs with 6,207 annotations (3-7 annotations per item) on the first 50 question-responses from each of 20 congressional hearings.
3.3 Annotated Dataset Analysis
Here, we explore the annotated dataset to confirm its validity, focusing on the response labels (Figure 3) and sentiment towards the witness. We then conduct a word association analysis that finds meaningful lexical cues for the conversation act, but not for the intent label.
Is there disagreement? One initial question with collecting data on multiple interpretations is whether crowdworkers have sufficiently different viewpoints. However, we do find there is sufficient disagreement: Figure 4 (a) shows annotators disagree about the response label (the combined conversation act + intent) on roughly half the data  $(53.5\%)$ , though this trend can vary considerably from one hearing to the next as shown in (b) and (c).
Is disagreement real or noise? To understand whether disagreements are genuine or noise, we
Overanswering with the intent to be helpful was included in our original taxonomy but then eliminated due to sparsity. This in addition to the requirements of  $>95\%$  approval rating,  $>500$  approved HITs, and living in the US for greater familiarity with the political issues.
Question	Response	Resp. Label	Anno.Sentiments
We used to be on the Small Business Committee, so we had definitions for what was a small business. What was yours?	What was my-say that again, sir?	cant_answer+ honest	[-1,-1,0,0,1,1]
And you spoke to their CEO immediately?	We shut down the app. We demanded-	shift+dodge	[-1,-1,0,0,1,1]
Table 3: All 6 annotators with different sentiments (negative: -1, neutral: 0, positive: 1) agree on the response label, even when it conflicts with their sentiment towards the witness (blue depicts sincere or positive, while red is deceptive or negative).
Response Label disagreement	%
ans+direct	vs.	shift+dodge	15.9
shift+correct	vs.	shift+dodge	8.2
ans+direct	vs.	ans+overans	5.6
cant_ans+honest	vs.	cant_ans+lying	4.9
ans+direct	vs.	shift+correct	3.9
ans+direct	vs.	shift+correct
vs.	shift+dodge	3.6
Table 4: Distribution of most frequent response disagreements; sincere intents in blue and deceptive in red.
examine the response label's inter-annotator agreement (IAA) and which labels are disagreed upon.
We do not expect high IAA for the response label as we are eliciting disagreement. Overall, IAA is 0.494 in Krippendorff's  $\alpha$  (considered 'moderate'; Artstein and Poesio (2008)), but importantly, we find higher agreement on the conversation act (0.652) compared to the intent (0.376). This finding confirms annotator understanding that the top-level label is more objective than the bottom-level one. We next group annotators with the same sentiments, expecting that when there is a disagreement, the same-sentiment groups will agree more with each other than with others. We partly confirm this intuition in Figure 5: grouping annotators by their sentiment increases agreement, but not by much. Sentiment is actually a more complicated signal, as we show in the following section.
Exploring annotator disagreements on the response label, we list the most frequent in Table 4. We find the disagreements often have opposing intents, but agree on the conversation act (e.g., shift+correct vs. shift+dodge). This result is encouraging, showing annotators have a shared understanding of the label definitions and further motivating our label taxonomy (Figure 2).
Is sentiment predictive of intent? We have pointed out how the annotator's sentiment towards the witness can help explain the label they choose. Is annotator sentiment then an easy predictor of the intent label or is it a more complicated signal? A correlation study shows they are in fact only weakly correlated (correlation ratio  $\eta = 0.34$  for coarse-