SlowGuess/NAACL_2021_OCR huggingface dataset:
Myle Ott, Yejin Choi, Claire Cardie, and Jeffrey T. Hancock. 2011. Finding deceptive opinion spam by any stretch of the imagination. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 309-319, Portland, Oregon, USA. Association for Computational Linguistics.
Ellie Pavlick and Tom Kwiatkowski. 2019. Inherent Disagreements in Human Textual Inferences. Transactions of the Association for Computational Linguistics, 7:677-694.
Brian Plüss and Paul Piwek. 2016. Measuring non-cooperation in dialogue. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1925-1936, Osaka, Japan. The COLING 2016 Organizing Committee.
Massimo Poesio, Jon Chamberlain, Silviu Paun, Juntao Yu, Alexandra Uma, and Udo Kruschwitz. 2019. A Crowdsourced Corpus of Multiple Judgments and Disagreement on Anaphoric Interpretation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1778-1789, Minneapolis, Minnesota. Association for Computational Linguistics.
Martha E. Pollack. 1986. A model of plan inference that distinguishes between the beliefs of actors and observers. In 24th Annual Meeting of the Association for Computational Linguistics, pages 207-214, New York, New York, USA. Association for Computational Linguistics.
Christopher Potts. 2008. Indirect answers and cooperation: On Asher and Lascarides's 'Making the right commitments in dialogue'. University of Michigan Linguistics & Philosophy Workshop on Implicatures, 21:23.
Hannah Rohde, Alexander Johnson, Nathan Schneider, and Bonnie Webber. 2018. Discourse coherence: Concurrent explicit and implicit relations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2257-2267, Melbourne, Australia. Association for Computational Linguistics.
Stephanie Schoch, Diyi Yang, and Yangfeng Ji. 2020. "this is a problem, don't you agree?" framing and bias in human evaluation for natural language generation. In Proceedings of the 1st Workshop on Evaluating NLG Evaluation, pages 10-16, Online (Dublin, Ireland). Association for Computational Linguistics.
Merel Cléo Johanna Scholman. 2019. Coherence relations in discourse and cognition: comparing approaches, annotations and interpretations. Ph.D. thesis, Universität des Saarlandes.
Tal Schuster, Darsh Shah, Yun Jie Serene Yeo, Daniel Roberto Filizzola Ortiz, Enrico Santus, and Regina Barzilay. 2019. Towards debiasing fact verification models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3419-3425, Hong Kong, China. Association for Computational Linguistics.
John Rogers Searle. 1969. Speech acts: An essay in the philosophy of language, volume 626. Cambridge University Press.
Elizabeth Shriberg, Raj Dhillon, Sonali Bhagat, Jeremy Ang, and Hannah Carvey. 2004. The ICSI meeting recorder dialog act (MRDA) corpus. Technical report, INTERNATIONAL COMPUTER SCIENCE INST BERKELEY CA.
David R. Traum and James F. Allen. 1994. Discourse Obligations in Dialogue Processing. In 32nd Annual Meeting of the Association for Computational Linguistics, pages 1-8, Las Cruces, New Mexico, USA. Association for Computational Linguistics.
David R Traum and Elizabeth A Hinkelman. 1992. Conversation acts in task-oriented spoken dialogue. Computational Intelligence, 8(3):575-599.
Yannick Versley. 2011. Multilabel Tagging of Discourse Relations in Ambiguous Temporal Connectives. In Proceedings of the International Conference on Recent Advances in Natural Language Processing 2011, pages 154-161, Hisar, Bulgaria. Association for Computational Linguistics.
Douglas Walton. 2003. The interrogation as a type of dialogue. Journal of Pragmatics, 35(12):1771-1802.
Bonnie Webber and Aravind Joshi. 2012. Discourse Structure and Computation: Past, Present and Future. In Proceedings of the ACL-2012 Special Workshop on Rediscovering 50 Years of Discoveries, pages 42-54, Jeju Island, Korea. Association for Computational Linguistics.
Bonnie Webber, Rashmi Prasad, and Alan Lee. 2019. "Ambiguity in Explicit Discourse Connectives". In Proceedings of the 13th International Conference on Computational Semantics - Long Papers, pages 134-141, Gothenburg, Sweden. Association for Computational Linguistics.
A Annotation Task
Screenshots of the task are in Figures 6 and 7. For each HIT, we provide the hearing title, date and summary, along with titles of the politicians and witness. If there are intervening turns in the conversation that are not part of a question-response pair, we give the annotator the option to view the immediately preceding or following turn (e.g., see 'Show/Hide Next Turn' in Figure 6). Each HIT takes an average of 15 minutes to complete. To minimize context switching for annotators and roughly preserve the original conversation order, we publish only a small batch from one hearing at a time, waiting until it completes before publishing the sequentially next one or starting a new hearing.
Annotations that were collected are summarized in Table 10 for the question and the response, and in Table 11 for the HIT. The introductory example (Figure 1) is further labeled with all the annotations for illustrative purposes in Figure 8.
All models are trained with binary cross-entropy loss on an NVIDIA Quadro RTX 6000 GPU. For hyperparameter tuning, we search over the learning rates of [1e-5, 2e-5, 3e-5], warmup proportions of [0, 0.001, 0.01, 0.1] and weight decays of [0, 0.001, 0.01, 0.1]. We use early stopping based on development macro-F1 with a patience of 5 epochs and average results across 3 runs with different random initializations. For test, we train for 30 epochs and then evaluate on the test fold. If training does not improve by  $40\%$  in the first 10 epochs, then training is restarted.
For ROBERTA and all models that build on top of it, we use a learning rate of 3e-5, a warmup proportion of 0.1, a weight decay of 0.1 and batch size of 8, max sequence length of 512. The 4-fold cross-validation takes approximately 65 minutes.
C Model variants
Table 12 includes results with additional baselines, pretrained language models and adding other forms of context.
HAN is a Hierarchical Attention Network as implemented in Adhikari et al. (2020). LSTM is a regularized LSTM as implemented in Adhikari et al. (2020). For the pretrained models ALBERT (Lan et al., 2020) and ELECTRA (Clark et al., 2020), we use the implementations from Hugging Face.
For adding context, the models with  $*$  are the ones described in the main paper, and their cross-validation results are in Table 8. +ENTIRE QUESTION includes the entire question (not just the interrogative sentences as does +QUESTION). The +QUESTION INTENTS includes the annotators' perceived intents of the question (fed in as a space-separated sequence of numbers, where each number is mapped from [attack, neutral, favor] to [-1, 0, 1]). The +FINE-GRAINED WITNESS SENTIMENT and +FINE-GRAINED QUESTIONER SENTIMENT include the 7-valued sentiment of the annotator towards witness (or questioner) (fed in the same style where the numbers are mapped from [very negative, negative, somewhat negative, neutral, somewhat positive, positive, very positive] to [-3, -2, -1, 0, 1, 2, 3]). The +COARSE-GRAINED QUESTIONER SENTIMENT includes the 3-valued annotator sentiment towards the questioner (mapping from [negative, neutral, positive] to [-1, 0, 1]).
Question-Response 5 of 5
Mr. Engel : So do you adjust your algorithms to prevent individuals interested in violence or nefarious activities from being connected with other like-minded individuals?
Mr. Zuckerberg: Sorry. Could you repeat that?
How does the politician pose their question(s)? includes leading question, controversial assumption or attack?  $⑤$  yes  $①$  no
How does the witness respond?
$\bigcirc$  answers all reasonable questions
$\bigcirc$  answers partially or not all questions
$\bigcirc$  answers a different question
$\bigcirc$  says they don't understand/don't know/can't answer or asks a clarification question
Show/Hide Next Turn
Please briefly explain all your choices for Witness Intent:
Sentiment towards questioners and witness
In this congressional hearing:
1. What is your overall sentiment towards the questioners (the congressional members asking the questions)?
$\mathbb{O}$  Very Negative
Somewhat Negative
Somewhat Positive
$\bigcirc$  Very Positive
2. What is your overall sentiment towards the witness?
$\bigcirc$  Very Negative
Figure 6: Overview of the crowdsourced task (only last of 5 question-response pairs shown for space).
How does the politician pose their question(s)?
includes leading question, controversial assumption
or attack?
includes question or statement favoring the witness?
(a) question
$\circ$  answers all reasonable questions
$\bigcirc$  answers partially or not all questions
$\bigcirc$  answers a different question
$\bigcirc$  says they don't understand/don't know/can't
answer or asks a clarification question
Witness provides extra info?
to sway opinion in their favor
to be helpful or add critical details
no
(b) answer, overanswer
$\bigcirc$  answers all reasonable questions
$\mathbb{O}$  answers partially or not all questions
$\bigcirc$  answers a different question
says they don't understand/don't know/can't
Witness intent for giving a partial answer?
to dodge an unfavorable question or question
part
for a justified reason
(c) partial answer
$\bigcirc$  answers all reasonable questions
$\bigcirc$  answers partially or not all questions
$\bullet$  answers a different question
says they don't understand/don't know/can't
answer or asks a clarification question
Witness intent for answering a different
to reformulate or evade an unfavorable
question
to correct the questioner or other justified
(d) shifted answer
$\bigcirc$  answers all reasonable questions
$\bigcirc$  answers partially or not all questions
$\bigcirc$  answers a different question
says they don't understand/don't know/can't
Witness intent for saying they can't answer?
you believe they're lying
you believe they're sincere
(e) can't answer
Figure 7: Nested questions that appear for the question (a) and for the response (b-e).
Table 10: Annotations and examples labeled for the question and the response.
Annotation	Labels
Sentiment towards questioners	very negative, negative, somewhat negative, neutral, somewhat positive, positive, very positive
Sentiment towards witness	very negative, negative, somewhat negative, neutral, somewhat positive, positive, very positive
Table 11: Annotations and labels for the HIT.
Figure 8: The introductory example with all annotations from two annotators with conflicting interpretations of the responses (the question type is determined by a rule-based classifier).
Model	macro-F1
LSTM	42.3	(6.6)
HAN	43.1	(11.9)
ALBERT	55.9	(8.0)
ELECTRA	55.8	(4.7)
+QUESTION* (All interrogatives)	57.6	(4.0)
+ENTIRE QUESTION	53.4	(22.0)
+QUESTION INTENTS	57.5	(18.0)
+ANNOTATOR* (Coarse-grained Witness)	62.0	(2.0)
+FINE-GRAINED WITNESS SENTIMENT	60.0	(2.0)
+FINE-GRAINED QUESTIONER SENTIMENT	57.7	(13.0)
+COARSE-GRAINED QUESTIONER SENTIMENT	58.9	(5.0)
Table 12: Results on the held-out fold's dev set for additional baselines (top), pretrained language models (middle), and incorporating other contexts (bottom). The models with * indicate the contextual models described in the main paper (cross-validation results for these are in Table 8.)
D Data Statement
The latest version of the data statement is maintained at https://github.com/elisaF/subjective_discourse/blob/master/data/data_statement.md.
Data Statement for SubjectiveResponses
Data set name: SubjectiveResponses
Citation (if available): TBD
Data set developer(s): Elisa Ferracane
Data statement author(s): Elisa Ferracane
Others who contributed to this document: N/A
A. CURRATION RATIONALE
The purpose of this dataset is to capture subjective judgments of responses to questions. We choose witness testimonials in U.S. congressional hearings because they contain question-answer sessions, are often controversial and elicit subjectivity from untrained crowdsourced workers. The data is sourced from publicly available transcripts provided by the U.S. government (https://www.govinfo.gov/app/collection/chrg) and downloaded using their provided APIs (https://api.govinfo.gov/docs/). We download all transcripts from 113th-116th congresses (available as of September 18, 2019), then use regexes to identify speakers, turns, and turns containing questions. We retain hearings with only one witness and with more than 100 question-response pairs as a signal of argumentativeness. To ensure a variety of topics and political leanings, we sample hearings from each congress and eliminate those whose topic is too unfamiliar to an average American citizen (e.g. discussing a task force in the Nuclear Regulatory Commission). This process yields a total of 20 hearings: 4 hearings from the 113th congress (CHRG-113hhrg86195, CHRG-113hhrg88494, CHRG-113hhrg89598, CHRG-113hhrg93834), 5 hearings from the 114th (CHRG-114hhrg20722, CHRG-114hhrg22125, CHRG-114hhrg26003, CHRG-114hhrg95063, CHRG-114hhrg97630), 7 hearings from the 115th (CHRG-115hhrg25545, CHRG-115hhrg30242, CHRG-115hhrg30956, CHRG-115hhrg31349, CHRG-115hhrg31417, CHRG-115hhrg31504, CHRG-115hhrg32380), and 4 hearings from the 116th (CHRG-116hhrg35230, CHRG-116hhrg35589, CHRG-116hhrg36001, CHRG-116hhrg37282). For annotation, we then select the first 50 question-response pairs from each hearing.
Code used to create the dataset is available at https://github.com/elisaF/subjective_discourse.
B. LANGUAGE VARIETY/VARIETIES
- BCP-47 language tag: en-US
- Language variety description: American English as spoken in U.S. governmental setting
C. SPEAKER DEMOGRAPHIC
- Description: The speakers are from two groups: the questioners are politicians (members of Congress) and the witnesses can be politicians, businesspeople or other members of the general public.
- Age: No specific information was collected about the ages, but all are presumed to be adults (30+ years old).
- Gender: No specific information was collected about gender, but members of Congress include both men and women. The witnesses included both men and women.
- Race/ethnicity (according to locally appropriate categories): No information was collected.
- First language(s): No information was collected.
- Socioeconomic status: No information was collected.
Number of different speakers represented: 91 members of Congress and 20 witnesses.
- Presence of disordered speech: No information was collected but none is expected.
D. ANNOTATOR DEMOGRAPHIC
Annotators:
- Description: Workers on the Amazon Mechanical Turk platform who reported to live in the U.S. and had a  $>95\%$  approval rating with  $>500$  approved HITs were recruited during the time period of November 2019 - March 2020.
- Age: No information was collected.
- Gender: No information was collected.
- Race/ethnicity (according to locally appropriate categories): No information was collected.
- First language(s): No information was collected.
- Training in linguistics/other relevant discipline: None.
Annotation guideline developer:
- Description: Elisa Ferracane
Age: 40.
- Gender: Female.
- Race/ethnicity (according to locally appropriate categories): Hispanic.
- First language(s): American English.
- Training in linguistics/other relevant discipline: PhD candidate in computational linguistics.
E. SPEECH SITUATION
- Description: Witness testimonials in U.S. congressional hearings spanning the 114th-116th Congresses.
Time: 2013-2019
- Place: U.S. Congress
- Modality (spoken/signed, written): trans-scribed from spoken.
- Scripted/edited vs. spontaneous: mostly spontaneous, though members of Congress sometimes read questions they have written down
- Synchronous vs. asynchronous interaction: synchronous
- Intended audience: the U.S. government and the general public, as all hearings are both transcribed and televised
F. TEXT CHARACTERISTICS
The genre is political discourse in a highly structured setting where a chairperson runs the meeting, and each member of Congress is afforded 5 minutes to question the witness but can yield their time to others. Topics vary based on the congressional committee that is holding the hearing, and include oversight of other governmental bodies (e.g.,
IRS, Department of Justice) and inquiries into businesses suspected of misconduct (e.g., FaceBook, Wells Fargo).
G. RECORDING QUALITY
H. OTHER
I. PROVENANCE APPENDIX
About this document
A data statement is a characterization of a dataset that provides context to allow developers and users to better understand how experimental results might generalize, how software might be appropriately deployed, and what biases might be reflected in systems built on the software.
Data Statements are from the University of Washington. Contact: [datastatements@uw.edu](mailto:datastatements@uw.edu). This document template is licensed as [CC0](https://creativecommons.org/share-your-work/public-domain/cc0/).
This version of the markdown Data Statement is from June 4th 2020. The Data Statement template is based on worksheets distributed at the [2020 LREC workshop on Data Statements](https://sites.google.com/uw.edu/data-statements-for-nlp/), by Emily M. Bender, Batya Friedman, and Angelina McMillan-Major. Adapted to community Markdown template by Leon Derczyznski.