SlowGuess/NAACL_2021_OCR huggingface dataset:
grained sentiment). There are two reasons for this result: (1) responses may have an unambiguous interpretation regardless of annotator sentiment, and (2) annotator sentiment towards the witness typically fluctuates throughout the hearing.
The most common unambiguous response is answer+direct (58%). Direct answers often leave little room for interpretation (e.g., 'Yes, that is correct'). More interestingly, annotators sometimes choose an intent that conflicts with their sentiment towards the witness (in 10% of unambiguous items). We illustrate the two cases in Table 3. In the first case, even the annotators with a negative view of the witness choose a sincere intent label. Conversely, in the second case, even the annotators with a positive view of the witness choose a deceptive intent label. While these are small phenomena, they illustrate the nuances of signaling sincerity and how they interact with the annotator's sentiment towards the witness.
For the annotator's sentiment across a hearing, a simplifying assumption is that it remains constant (recall the sentiment is reported at the end of each HIT, and HITs are presented to annotators in almost the same order as the original hearing). In practice it does not:  $59\%$  of annotators that label more than one HIT change their sentiment. As one annotator explained, "When he [the witness] said that, I got a different attitude towards him."
Influence of question Earlier, we posited the question influences the response (Table 2). We find the question intent and type are weakly correlated with the response label. On a per-hearing basis, though, we observe stronger correlation for declarative question types in some hearings, partly confirming our hypothesis. We find qualitative evidence in explanations that annotators consider the question ("it was a terrible question to begin with").
Lexical cues for labels To understand whether the response labels have lexical cues, we follow Schuster et al. (2019) to analyze the local mu
Annotator with positive sentiment	Annotator with negative sentiment
R: Congresswoman, it might be useful to clarify what actually happened. A developer who is a researcher-	Resp. Label: shift+correct
Expl: Witness wants to clarify what happened.	Resp. Label: shift+dodge
Expl: Mr. Zuckerberg goes off on a tangent to “clarify” the situation.
R: We are working through the process. We have never said we would not provide those.	Resp. Label: ans+direct
Expl: Mr. Koskinen answers and does say factually that they never said they would not provide the emails.	Resp. Label: shift+dodge
Expl: Koskinen evades the question, by saying that he never said he wouldn’t provide the emails.
Table 5: Explanations from annotators with opposing interpretations quoting the same response text (underlined) with subjective language in blue (neutral, positive) and red (negative).
Response label	top LMI n-grams
ans+direct	‘yes’, ‘be correct’
ans+overanswer	‘think’, ‘think that’
shift+dodge	‘we’, ‘-’
shift+correct	‘-’, ‘be something’
cant_ans+lying	‘I’, ‘?’, ‘not’
cant_ans+honest	‘I’, ‘not’, ‘?’
Table 6: n-grams with highest LMI scores in each label.
tual information (LMI) between labels and the response text n-grams  $(n = 1,2,3)$ . Unlike PMI, LMI highlights high frequency words co-occurring with the label. The top-scoring n-grams in Table 6 show most labels have a meaningful cue (the lower scoring words are not informative as they tend to be hearing-specific with much lower frequencies). The ans+direct cues signal straight answers. Dashes for both shift indicate the witness was interrupted (recall these include partial answers). Both cant_answer labels have the same cues, which include negation (to indicate not being able to answer) and question mark for clarification questions. We thus expect these cues may help identify conversation acts, but not the intents.
In summary, our analysis of the dataset shows there is ample and genuine disagreement. Interestingly, these disagreements are only partly attributable to differences in annotator sentiment. Furthermore, sentiment often fluctuates across a hearing, and can be influenced by what is said during the hearing. The question labels are not a straightforward signal for the response labels, but can vary by hearing. Finally, we find evidence of lexical cues for the conversation act label, but not for the intent.
3.4 Qualitative Analysis of Explanations
The explanations are a rich source of data for understanding annotator interpretations, with evidence they are applying personal beliefs ('Bankers are generally evil') and experiences ('I have watched
hearings in congress'). We conduct a qualitative analysis to gain insight into the differing interpretations. Explanations are free-form, but annotators sometimes quote parts of the response. Interestingly, multiple annotators can quote the same text, yet arrive at opposite labels, as in Table 5. Studying these cases offers a window into what part of a discourse may trigger a subjective view, and how this view is expressed.
To this end, we examine the discourse and argumentative relations of the quoted text, and the linguistic devices used by the annotator to present the quote. We find the quoted text is often part of the response's supporting argument, serving as the background or motivation that underpins the main claim. The annotator's presentation of the quote differs drastically depending on their slant. Sincere labels use neutral or positive language ('state', 'say factually'), whereas deceptive labels use negative words and framing ('evades', 'goes off on a tangent'). Quotation marks in positive explanations become scare quotes in a negative one (first example in Table 5). On the negative side, we also find hedging ('claim') and metaphors ('skirting the meaning', 'dances around').
Our qualitative analysis shows annotators consider the side arguments underpinning the main claims, and employ rich linguistic devices to reflect their judgments.
We propose the task of predicting all possible interpretations of a response (i.e., all perceived conversation act+intent labels) with the goals of analyzing discourse in a realistic setting and understanding sociolinguistic factors contributing to variations in discourse perception. We frame this task as a multi-label classification setting where 6 binary classifiers predict the presence of each of the 6 labels. $^{8}$
<sup>8</sup>We experimented with a set-valued classifier that predicts the label set from all observed combinations (27-way multi-
We evaluate with macro-averaged F1 which gives equal weight to all classes, unlike micro-averaging which in our imbalanced data scenario (Figure 3) would primarily reflect the performance of the large classes.
We experiment with pretrained language models with the intuition that a general language understanding module can pick up on patterns in the response to distinguish between the classes.
Training We split the data into 5 cross-validation folds, stratified by congressional hearing (to preserve the differing response distributions as seen in Figure 3). We reserve one fold for hyperparameter tuning and use the remaining 4 folds for cross-validation at test time.
Baselines The ALL POSITIVE baseline predicts 1 for all labels. This baseline easily outperforms a majority baseline that predicts the most frequent label (answer+direct). LOG REGRESSION performs logistic regression with bag-of-words representations. CNN is a convolutional neural network as implemented in Adhikari et al. (2020). Other baselines performing lower than CNN are in Appendix C.
Pretrained We experiment with several pretrained language models, and find ROBERTA (Liu et al., 2019) performs the best on the held-out development fold. We use the implementation from Hugging Face. $^{10}$  We feed in the tokenized response text and truncate input to 512 word pieces (additional inputs used in the model variants we describe next are separated by the [SEP] token).
Hierarchical We use two classifiers to mimic the hierarchy of our taxonomy: the first classifier predicts the conversation act while the second predicts the complete label (conversation act+intent). We train the classifiers independently, and condition the second classifier on the ground truth of the first classifier during training, only placing a distribution over intents consistent with that conversation act. At test time, we use predictions from the first classifier instead of ground truth.
+Question Building on top of the hierarchical model, this model incorporates the context of the
question by including all interrogative sentences.[11]
+Annotator This model incorporates annotators' coarse-grained sentiment towards the witness (fed in as a space-separated sequence of numbers, where each number is mapped from {negative, neutral, positive} sentiment to  $\{-1,0,1\}$ ).
The pretrained models easily outperform the baselines as seen in Table 7, where ROBERTA performs best. We next report results on incorporating hierarchy and context. Macro-F1 is calculated over the pooled results of the 4 folds; statistical significance is measured with the paired bootstrap test (Efron and Tibshirani, 1994) and  $\alpha < 0.05$ .
Model	macro-F1 (var)
ALL POSITIVE	35.0
LOG REGRESSION	40.5	(6.9)
CNN	46.0	(7.2)
BERT (uncased)	55.6	(3.8)
ROBERTA	58.5	(3.7)
HIERARCHICAL	58.2	(3.9)
Table 7: Results on the held-out fold's dev set, averaged across three random restarts.
Adding hierarchy As seen in Table 8, incorporating an additional classifier to predict the top-level conversation helps, but not significantly. The per-class performance shows it mainly helps the less-represented conversation acts shift and cant_answer, with a better false negative rate for these classes. While the HIERARCHICAL model makes fewer errors of the kind intended to be corrected by the hierarchy as illustrated in Table 9 (by not predicting labels incompatible with the conversation act), the difference is very small. Jointly training these two classifiers with an adaptive learning curriculum may yield better results, which we leave for future work.
Adding context As shown in Table 8, adding the question in +QUESTION actually hurts performance, in particular by overpredicting the smaller classes ans+overans and cant_ans+honest. The lack of a benefit
<sup>11</sup>We employ this truncation method because questions can be very lengthy (Table 1) We obtain poorer results using other forms of question context, including the entire question text, or only the last question.
12We nevertheless choose to build on this model as the subsequent models incorporating context exhibit more stable and significant differences.
class classification), but found this didn't work well. See Appendix B for training details and hyperparameters. https://huggingface.co/transformers/
Model	macro-F1	top-level class F1	bottom-level class F1
answer	shift	cant_ans	answer +direct	answer +overans	shift +dodge	shift +correct	cant_ans +lying	cant_ans +honest
ROBERTA	56.9	88.3	69.9	72.9	87.8	26.1	63.2	42.3	51.3	70.4
HIERARCHICAL	57.6†	87.9	74.0	77.3	87.3	26.7	66.3	47.2	42.6	75.3
+QUESTION	56.4	87.9	76.0	75.3	87.6	23.8	65.5	45.6	43.8	72.0
+ ANNOTATOR	60.5*	87.5	75.4	78.2	87.3	33.6	67.6	46.3	54.4	73.6
Table 8: Macro and class-level F1 on the test sets for the top-level (conversation act) and bottom-level (conversation act+intent) classes. † indicates not stat. sig. better vs. ROBERTA. * indicates stat. sig. better vs. HIERARCHICAL.
Context	Model Predictions
ROBERTA	HIERARCH.	+ANNOTATOR
Q: So my first question to you is, was the FBI aware of this Reddit post prior to offering Mr. Combetta immunity in May?R: I'm not sure. I know our team looked at it. I don't know whether they knew about it before then or not.Sentiments: 0, 0, -1, 0	answer+directcant_ans+lyingcant_ans+honest	cant_ans+lyingcant_ans+honest	cant_ans+honest
Table 9: Example of model predictions (incorrect ones in red). Taking into account the hierarchy correctly eliminates labels for the absent conversation act of 'answer'. (Not shown: adding the question makes no corrections to this prediction). Adding the mostly neutral sentiments corrects the false positive for the lying intent, and is able to predict the entire label set correctly.
contradicts our expectations of the importance of the question and qualitative evidence, but is consistent with the weak correlation results. We hypothesize a different representation of the question is needed for the model to exploit its signal, which we leave for future work.
Incorporating the annotator sentiments in +ANNOTATOR provides a statistically significant benefit that helps both the false positive and false negative rate of the smaller classes ans+overans and cant_ans+lying. In the example of Table 9 which has mostly neutral sentiments, the model corrects the false positive made by the Hierarchical model for cant_ans+lying.
From these results, we conclude that our task is heavily contextual with complex labels. On the one hand, taking into account the sentiments of the annotator leads to better predictions. On the other hand, we've shown annotator sentiment is not a simple reflection of intent. Furthermore, questions qualitatively influence the response labels, but linguistic features and labels of the question are not strongly correlated with the response and our model is not able to make effective use of it. The disagreements appear to reflect other axes, and this work begins to scratch the surface of understanding the subjective conversation acts and intents in conversational discourse.
In this paper, we tackle the subjectivity of discourse; that is, how ambiguities are resolved. We present a
novel English dataset containing multiple ground truths in the form of subjective judgments on the conversation acts and intents of a response in a question-response setting. We show the dataset contains genuine disagreements which turn out to be complex and not easily attributable to a single feature, such as annotator sentiment. The annotator rationales provide a window into understanding these complexities, and offer a rich source of linguistic devices. We propose a task to predict all possible interpretations of a response, whose results are consistent with our data analysis: incorporating the annotator bias helps the model significantly improve. We publicly release the dataset in hopes to spur further research by exploring the sequential nature of the hearings to employ CRF-type losses and other forms of aggregating annotator judgments.
We provide a detailed dataset statement in Appendix D. The data collected in this dataset is produced by the U.S. government and is freely available to the public. The ids of the crowdsourced workers that contributed to the annotation are anonymized. Workers were compensated an average of  $1.20 per HIT (approximately$ 8/hour), using the U.S. federal minimum wage as a minimum bar.
We recognize that crowdsourced workers, and thus the collected judgments in our dataset, are not representative of the U.S. population (Difallah et al., 2018).
We thank the annotators that contributed to this dataset. We thank reviewers and the first author's thesis committee for insightful feedback. We acknowledge the Texas Advanced Computing Center for grid resources. The first author was supported by the NSF Graduate Research Fellowship Program under Grant No. 2017247409.
Ashutosh Adhikari, Achyudh Ram, Raphael Tang, William L. Hamilton, and Jimmy Lin. 2020. Exploring the Limits of Simple Learners in Knowledge Distillation for Document Classification with DocBERT. In Proceedings of the 5th Workshop on Representation Learning for NLP, pages 72-77, Online. Association for Computational Linguistics.
Ron Artstein and Massimo Poesio. 2008. Survey Article: Inter-Coder Agreement for Computational Linguistics. Computational Linguistics, 34(4):555-596.
Nicholas Asher and Alex Lascarides. 2003. *Logics of conversation*. Cambridge University Press.
Nicholas Asher and Soumya Paul. 2016. Evaluating conversational success: Weighted message exchange games. In Proceedings of the 20th Workshop on the Semantics and Pragmatics of Dialogue - Full Papers, New Brunswick, NJ. SEMDIAL.
Nicholas Asher and Soumya Paul. 2018. Strategic Conversations Under Imperfect Information: Epistemic Message Exchange Games. Journal of Logic, Language and Information, 27(4):343-385.
John L Austin. 1962. How to Do Things with Words. The William James Lectures delivered at Harvard University in 1955.
Dwight Bolinger. 1957. Interrogative structures of American English: The direct question, volume 28. Publication of the American Dialect Society.-University of Alabama Press.
Peter Bull. 2008. "Slipperiness, Evasion, and Ambiguity": Equivocation and Facework in Noncommittal Political Discourse. Journal of language and social psychology, 27(4):333-344.
Harry Bunt, Volha Petukhova, Andrei Malchanau, Alex Fang, and Kars Wijnhoven. 2018. The DialogBank: dialogues with interoperable annotations. *Language Resources and Evaluation*, pages 1-37.
Jonathan P Chang, Justin Cheng, and Cristian Danescu-Niculescu-Mizil. 2020. Don't let me be misunderstood: Comparing intentions and perceptions in online discussions. In Proceedings of The Web Conference 2020, pages 2066-2077.
Zheqian Chen, Rongqin Yang, Zhou Zhao, Deng Cai, and Xiaofei He. 2018. Dialogue Act Recognition via CRF-Attentive Structured Network. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pages 225-234.
Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2020. ELECTRA: Pretraining Text Encoders as Discriminators Rather Than Generators. In International Conference on Learning Representations.
David E Clementson. 2018. Deceptively dodging questions: A theoretical note on issues of perception and detection. *Discourse & Communication*, 12(5):478-496.
Philip R. Cohen and C. Raymond Perrault. 1979. Elements of a plan-based theory of speech acts. Cognitive Science, 3(3):177-212.
Debopam Das, Manfred Stede, and Maite Taboada. 2017. The Good, the Bad, and the Disagreement: Complex ground truth in rhetorical structure analysis. In Proceedings of the 6th Workshop on Recent Advances in RST and Related Formalisms, pages 11-19. Association for Computational Linguistics.
Marie-Catherine de Marneffe, Scott Grimm, and Christopher Potts. 2009. Not a simple yes or no: Uncertainty in indirect answers. In Proceedings of the SIGDIAL 2009 Conference, pages 136-143, London, UK. Association for Computational Linguistics.
Djellel Difallah, Elena Filatova, and Panos Ipeirotis. 2018. Demographics and dynamics of mechanical turk workers. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, WSDM '18, page 135-143, New York, NY, USA. Association for Computing Machinery.
Esin Durmus and Claire Cardie. 2018. Exploring the role of prior beliefs for argument persuasion. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1035-1045, New Orleans, Louisiana. Association for Computational Linguistics.
Bradley Efron and Robert J Tibshirani. 1994. An introduction to the bootstrap. CRC Press.
Jonas Gabrielsen, Heidi Jønch-Clausen, and Christina Pontoppidan. 2017. Answering without answering: Shifting as an evasive rhetorical strategy. Journalism, pages 1-16.
Jonathan Ginzburg, Zulipiye Yusupujiang, Chuyuan Li, Kexin Ren, and Paweł Lupkowski. 2019. Characterizing the response space of questions: a corpus study for English and polish. In Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue, pages 320-330, Stockholm, Sweden. Association for Computational Linguistics.
Venkata Govindarajan, Benjamin Van Durme, and Aaron Steven White. 2019. Decomposing generalization: Models of generic, habitual, and episodic statements. Transactions of the Association for Computational Linguistics, 7:501-517.
Jeroen Antonius Gerardus Groenendijk and Martin Johan Bastiaan Stokhof. 1984. Studies on the Semantics of Questions and the Pragmatics of Answers. Ph.D. thesis, Univ. Amsterdam.
Christine Gunlogson. 2008. A question of commitment. *Belgian Journal of Linguistics*, 22(1):101-136.
Thomas Hanneforth, Silvan Heintze, and Manfred Stede. 2003. Rhetorical Parsing with Underspecification and Forests. In Companion Volume of the Proceedings of HLT-NAACL 2003 - Short Papers, pages 31-33. Association for Computational Linguistics.
Dan Jurafsky, Elizabeth Shriberg, and Debra Bia-asca. 1997. Switchboard SWBD-DAMSL labeling project coder's manual. Draft 13. Technical Report 97-02.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. In International Conference on Learning Representations.
Yang Liu, Ivan Titov, and Mirella Lapata. 2019. Single document summarization as tree induction. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1745-1755, Minneapolis, Minnesota. Association for Computational Linguistics.
Tyler McDonnell, Matthew Lease, Mucahid Kutlu, and Tamer Elsayed. 2016. Why Is That Relevant? Collecting Annotator Rationales for Relevance Judgments. In Fourth AAAI Conference on Human Computation and Crowdsourcing.
Tristan Miller, Maria Sukhareva, and Iryna Gurevych. 2019. A streamlined method for sourcing discourse-level argumentation annotations from the crowd. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1790-1796, Minneapolis, Minnesota. Association for Computational Linguistics.
Yixin Nie, Xiang Zhou, and Mohit Bansal. 2020. What can we learn from collective human opinions on natural language inference data? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9131-9143, Online. Association for Computational Linguistics.