SlowGuess/NAACL_2022_OCR huggingface dataset:
Krishna C. Sanagavarapu<sup>1</sup>, Jathin P. Singaraju<sup>2</sup>, Anusha Kakileti<sup>2</sup>, Anirudh Kaza<sup>2</sup>, Aaron A. Mathews<sup>2</sup>, Helen Li<sup>2</sup>, Nathan R. Brito<sup>2</sup>, Eduardo Blanco<sup>1</sup>  
<sup>1</sup>Arizona State University, <sup>2</sup>University of North Texas, {ksanagav, eduardo.blanco} @asu.edu, {pran.singaraju, arkaza820} @gmail.com, {akakileti, AaronMathews, helenli, Nathanbrito} @unt.edu
In this paper, we explore the task of determining indirect answers to yes-no questions in real conversations. We work with transcripts of phone conversations in the Switchboard Dialog Act (SwDA) corpus and create SwDA-IndirectAnswers (SwDA-IA), a subset of SwDA consisting of all conversations containing a yes-no question with an indirect answer. We annotate the underlying direct answers to the yes-no questions (yes, probably yes, middle, probably no, or no). We show that doing so requires taking into account conversation context: the indirect answer alone is insufficient to determine the ground truth. Experimental results also show that taking into account context is beneficial. More importantly, our results demonstrate that existing corpora with synthetic indirect answers to yes-no questions are not beneficial when working with real conversations. Our best models outperform the majority baseline by a substantial margin, but the task remains a challenge (F1: 0.46).
Dialogue systems have become a reality enabled by large datasets and deep neural networks. Task-oriented (Wen et al., 2017) and open-domain (Tang et al., 2019) dialogue systems are commonplace. Neural approaches are popular (Zhang et al., 2018) although systems based on logical inference and rules outperform networks in open-domain dialogue (Finch et al., 2021). State-of-the-art systems face challenges keeping track of a conversation and avoiding inconsistencies (Welleck et al., 2019). Evaluation is also an open issue as automated metrics have several drawbacks (Liu et al., 2016). An alternative is to evaluate dialogue systems based on whether they can collaborate with humans to solve a problem (Lewis et al., 2017) or elicit some action such as donating to a cause (Wang et al., 2019).
People do not explicitly say what they mean when they speak to each other yet they seamlessly
A: Do you work outside of the home?  
B: No, I am not working currently.  
Underlying direct answer: No
A: Do you work outside of the home?  
B: Uh, last month I left my company.  
Underlying direct answer: No
A: Do you work outside of the home?
B: Uh, last month I left my company.
A: What happened? Stress?
B: But now I work for a marketing firm where I travel a lot. Underlying direct answer: Yes
Figure 1: Conversation snippets with a yes-no question (first turn by Sparker A). In the first example, the underlying direct answer is obvious given the answer by Speaker B. In the second example, determining the underlying direct answer requires commonsense knowledge. In the third example, it requires not only commonsense but also taking into account more than the turn by Speaker B immediately following the question.
carry on conversations. For example, customers asking Where are the $1 cups? reveal to the sales associate that they want to buy a (cheap) cup (Tatu, 2005). In this paper, we investigate the underlying direct answers to yes-no questions. A yes-no question is a question that expects a yes or no for an answer. As we shall see, we work with direct yes-no questions (e.g., Did you drive yourself to the airport) and indirect ones (e.g., I am not sure if you drove yourself to the airport).
Consider the conversation snippets in Figure 1. As shown in the first one, the conversation turn following a yes-no question may be a direct answer (i.e., a turn including yes, no, obviously, never or similar keywords). Indirect answers (e.g., second snippet) are more common. Speaker B leaving his company a month ago does not entail that he is jobless (and thus not working—at home or the office). Given this indirect answer alone, however, it is reasonable to conclude so thus the underlying direct answer is no. The broader conversation context often provides a more complete picture and the
Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4677 - 4695 July 10-15, 2022 ©2022 Association for Computational Linguistics
true underlying direct answer (e.g., third snippet). After a brief interjection by Speaker A, Speaker B states that he changed jobs and now travels a lot. Thus, the underlying direct answer is yes.
The focus of this paper is to determine the underlying direct answers to yes-no questions. Unlike previous efforts, we work with transcripts of real conversation as opposed to yes-no questions and indirect answers written by annotators on demand. The main contributions are:
1. We present SwDA-IA, a corpus consisting of the 2,544 yes-no questions in Switchboard (Jurafsky et al., 1997, SwDA) with Indirect Answers and their underlying direct answers.
2. We show that determining underlying direct answers requires context beyond the yes-no question and the next conversation turn. Indeed, the ground truth changes depending on whether we show annotators context.
3. We demonstrate that transfer learning with related tasks as well as synthetic yes-no questions and synthetic indirect answers only bring modest improvements. Specifically, synthetic data do not transfer to real conversations.
4. We provide insights into the most common errors made by our best performing model.
The work presented here provides evidence that determining the underlying direct answer to yes-no questions in real conversations is much more challenging than in synthetic data. Our best model obtains 0.49 F1 even when using only the context handpicked as important by annotators.
Yes-no questions and indirect answers have been studied for decades (Green and Carberry, 1999). Hockey et al. (1997) report that  $27\%$  of all questions in 18 hours of spontaneous speech are yes-no questions with indirect answers (Rossen-Knill et al., 1997). Indirect answers are often used to ask follow-up questions or provide explanations for negative answers (Stenstrom, 1984), prevent wrong interpretations of direct answers (Hirschberg, 1985) or show politeness (Brown and Levinson, 1978).
Yes-no questions have received considerable attention recently. Outside of the dialogue genre, Clark et al. (2019) present BoolQ, a corpus of 16,000 yes-no questions submitted to a search engine (e.g., Does France have a Prime Minister and
a President?) along with Wikipedia articles that may contain an answer. Two recent corpora consist of dialogues generated by crowdworkers on demand after being given some text to talk about: QuAC (Choi et al., 2018) and CoQA (Reddy et al., 2019). Unlike the work presented here, the dialogues in these corpora are limited to questions and answers about the text provided to crowdworkers. More importantly, we work with unrestricted, natural conversations rather than dialogues between paid annotators who are asked to have a conversation that resembles a questionnaire geared towards checking for understanding of a piece of text.
Yes-no questions in real conversations have been studied before at a small scale. de Marneffé et al. (2010) work with 224 questions-answer pairs involving gradable adjectives. We borrow from them the five labels to annotate underlying direct answers. de Marneffé et al. (2009) present a typology for 623 yes-no questions from SwDA. Our corpus is four times larger. Unlike them, we show that context is crucial to determine underlying direct answers and present experimental results.
The work by Louis et al. (2020) is the closest to ours. They present Circa, 34,268 yes-no questions and indirect answers written by crowdworkers given one of ten scenarios (e.g., scenario: Talking to a friend about music preferences, Q: Do you like guitars? A: I practice playing every weekend). Unlike us, Circa assumes that the turn following a yes-no question is enough to determine the underlying direct answer. Furthermore, we show that (a) determining underlying direct answers to yes-no questions is much harder in real conversations and (b) their synthetic data only bring minor improvements when working with real conversations.
3 SwDA-IA: A Corpus of Yes-No Questions and Indirect Answers
We present SwDA-IA, a corpus consisting of (a) the 2,544 yes-no questions in Switchboard with Indirect Answers and (b) manual annotations indicating the underlying direct answers. Three characteristics set our work apart. First, we work with real conversations as opposed to artificial, synthetic ones (Section 2). Second, we show that determining underlying direct answers requires taking into account context beyond the yes-no question and the next turn. Third, annotators pinpoint which turns within the context around a yes-no-question are useful to determine the underlying direct answer.
<sup>1</sup>Corpora available at https://github.com/krishna-chaitanya-sanagavarapu/SwDA-IA.
3.1 Collecting Yes-No Questions with Indirect Answers
While we could work with any dialogue corpora, we chose SwDA (Jurafsky et al., 1997) because it contains unrestricted conversations and the dialogue acts annotations simplify the process of identifying yes-no questions with indirect answers. SwDA contains transcripts of 1,155 five-minute phone conversations. In these conversations, two people talk about topics such as childhood, recycling, and news media. The conversations are unrestricted and often divert from the initial topic. 440 speakers participated in the corpus creation resulting in 122,646 utterances. We use the distribution in ConvoKit (Chang et al., 2020) for convenience.
After manually examining all the dialogue acts, we select all conversation turns containing a dialogue act that indicates a yes-no question (see the list in the supplementary materials). This step results in 5,846 yes-no question including indirect ones (e.g., "I don't know if you are familiar with that issue." is an indirect version of "Are you familiar with that issue?"). After selecting yes-no questions, we discard those that are followed by a turn containing a direct answer. To do so, we discard turns containing a dialogue act indicating direct answers (see list in the supplementary materials). After manually examining the 2,542 questions that are not filtered, we observed that some have a direct answer. In order to avoid them, we also consider a direct answer any turn that contains yes, yea, yeah, no way, nope, never, sure, right, you bet, of course, certainly, maybe, definitely, or uh huh.
These steps select 2,376 turns with 2,544 yes-no questions followed by an indirect answer (168 turns have more than one yes-no question). Only  $4.77\%$  of all utterances in SwDA are a yes-no question. However,  $43.52\%$  of all yes-no questions have an indirect answer (2,544 out of 5,846). More importantly,  $77.4\%$  of the 5-minute conversations in SwDA contain at least one yes-no question with an indirect answer. The supplementary materials provide an analysis of the yes-no questions we work with. For example,  $41.4\%$  are indirect (e.g., do not include a question mark).
3.2 Annotating Underlying Direct Answers
In order to determine the underlying direct answers to the 2,544 yes-no questions, we manually annotated them. Our label set consists of five options: (definitely) Yes, Probably Yes, (in the) Middle, Prob-
ably No and (definitely) No. Middle is chosen when annotators do not lean towards any of the other four labels (e.g., “A: Do you have kids? B: Do I have kids?”). These labels are heavily inspired by de Marneffte et al. (2010). Louis et al. (2020) include a few more options (e.g., “I am not sure”, “In the middle, neither yes or no” and “Other”) but they report very low frequencies (0.2–1.9%). These three labels are included in our “Middle.”
We found two common scenarios that require additional explanation in order to be consistent:
- If the answer is yes under certain conditions or sometimes yes, annotators are instructed to choose Probably Yes. For example, the correct label for “A: Do you travel a lot for pleasure?” B: We try to make one trip per year if I find a good sale” is Probably yes.
- If the yes-no question contains a negation, Yes and Probably Yes have their meaning reversed. For example, the correct label for “A: You didn’t move to Alaska, right? B: I have been in Alaska for 13 years now” is No (Yes would be correct if B had not moved to Alaska).
Annotation Process and Quality The annotation process took place in two independent phases in order to investigate the role of context in determining underlying direct answers. In the first phase, annotators were only shown the yes-no question and the conversation turn immediately after (i.e., the indirect answer). In the second phase, annotators were shown three turns before and after the yes-no question as context. Different annotators annotated the same question in each phase to avoid potential biases (i.e., recalling answers from the previous phase). In both phases, the interface showed conversation turns after brief delays in order to encourage annotators to read the conversation snippet in order (either two turns or seven turns). In addition to selecting the underlying direct answer, in the second phase annotators also tagged the conversation turns that help them determine it. We hosted the annotation interface internally and recruited in-house annotators. Seven native English speakers participated in the annotations.
Inter-Annotator Agreement After refining the explanation for each label and group discussions, we conducted a pilot with 200 questions and the seven annotators. The average weighted Fleiss' Kappa (Plewis and Unit, 1982) between all pairs of annotators was 0.80. Given the high agree
Figure 2: Heatmaps of the inter-annotator disagreements when annotators have access to (a) only the yes-no question and indirect answer (bottom left) and (b) conversation context. Numbers indicate percentages. Most disagreement are minor: between ProbYes and either Yes or Middle, or ProbNo and either No or Middle.
ment (Landis and Koch, 1977), the remaining 2,344 questions were divided into twelve batches, and two annotators annotated each batch. The average Weighted Fleiss' Kappa score for all the batches was 0.81 when not showing context to annotators (first phase) and 0.78 when showing context (second phase). Disagreements in both phases were adjudicated after manual examination. The "true" underlying direct answers are the one annotated when context is shown to annotators, and those are the ones we conduct experiments with (Section 5).
Figure 2 shows the percentage of all disagreements prior to the adjudication step when context is not and is shown to annotators (top right and bottom left). As expected given the high Kappa coefficients, most disagreements are minor: between (a) Yes (or No) and Probably Yes (or Probably No) or (b) Middle and Probably Yes or Probably No.
4 Corpus Analysis: the Role of Context
The annotations we collected in both phases differ substantially. Thus context beyond the yes-no question and indirect answer (i.e., the following turn) is needed to determine the ground truth in real conversations. To our knowledge, previous work does not take into account context (Section 2).
Table 1 presents the distribution of each label when the interface shows and does not show context to annotators. The major change is that almost half of the questions annotated with Middle without context are annotated with one of the other four labels with context. Note that out of the five labels, Middle is arguably the one that reveals the least information about the speakers.
% without context	% with context
Yes	35.7	42.4
ProbYes	14.9	19.5
Middle	28.0	14.9
ProbNo	7.7	9.9
No	13.6	13.3
Table 1: Label distribution when annotators have access to (a) only the yes-no question and indirect answer (without context), and (b) conversational context (3 turns before and after the question, with context).
Figure 3: Heatmap of the changes in ground truth depending on whether annotators have access to context in addition to the question and indirect answer. Numbers indicate percentages. Context is crucial to determine the underlying direct answer to a yes-no question.
Figure 3 shows how the ground truth changes when we show context to annotators. The most common change is from Middle to either Yes (14.9%) or ProbYes (13.4%). In other words, context allows annotators to select an underlying direct answer that is more meaningful. There are also many changes from ProbYes to Yes (13.5%) and vice versa (13.5%), suggesting that context provides further details (a clarification, condition, etc.) to determine the underlying direct answer.
We show examples of changes in ground truth when annotators do not have and have access to context in Table 2. In the first example (top left), the indirect answer  $(t_1)$  repeats the question and does not provide any clue about the underlying direct answer. The next turn by A  $(t_3)$ , however, leaves no doubt: the underlying direct answer is Yes. The second example (top right) shows a similar pattern, but it exemplifies negation in the question and uncertainty. Speaker A does not commit to a Yes (Yes, It didn't kill anything) but rather sharing that to his knowledge it has not (Probably Yes).
The third example (bottom left) shows how follow-up questions in context can clarify the under
$t_{-3}$  , A: Which is like twenty minutes away.
$t_{-2}$  ,B:Right.
$t_{-1}$ , A: But, uh, we don't have any fast foods here in this small city.
$t_0$ , B: That is probably very fortunate. Do you have kids?
$t_1$ , A: Do I have kids?
$t_2$ , B: Yeah.
$t_3$ , A: Well, I have a son, but he's grown up.
Underlying direct answer ...
without context: Middle; with context: Yes
$t_{-3}$  ,A:And it seem
$t_{-2}$  B: I bank at NCNB and they have a number that you can call. I always call in once every other week and check what checks have cleared
$t_{-1}$  ,A: Yeah.
$t_0$ , B: Do you do that?
$t_1$ , A: We have the same thing.
$t_2$ , B: You do too?
$t_3$ , A: Yeah.
without context: ProbYes; with context: Yes
$t_{-3}$  ,A:Um,not bad.
$t_{-2}$  ,B:Yeah
$t_{-1}$ , A: Occasionally you'll have a mound pop up, but that is expected.
$t_0$ , B: What else did it, it didn't kill anything right?
$t_1$ , A: Um.
$t_2$ , B: Not really.
$t_3$ , A: As far as I know it hasn't killed anything. Even the area of the grass that was underneath and around the mounds.
without context: Middle; with context: ProbYes
$t_{-3}$ , A: So, uh, you know, you need to go to a school that handles whatever it is you want to do.
$t_{-2}$  ,B: Yeah.Where did you go to school?
$t_{-1}$  ,A:Uh,University of Mississippi.
$t_0$ , B: Oh. Was that local
$t_1$ , A: Uh, well, it was, well, within the state.
$t_2$ , B: Uh huh.
$t_3$ , A: But it, it was not necessarily local.
Underlying direct answer...
without context: Yes; with context: No
Table 2: Examples of yes-no questions  $(t_0)$  with indirect answers  $(t_1)$ . Determining the underlying direct answers requires taking into account more conversation context than the question and indirect answer: the ground truth annotated by humans changes depending on whether they also have access to context  $(t_{-3}, t_{-2}, t_{-1}, t_2, \text{and} t_3)$ .
lying direct answer of a yes-no question ( $t_0$ ;  $t_2$  is also a yes-no question). Given  $t_1$  alone, annotators selected Probably Yes as there is some uncertainty about what same thing refers to. The following two turns ( $t_2$ ,  $t_3$ ) make it clear that the underlying direct answer is Yes. This example also shows how context before the yes-no question ( $t_{-2}$ ) is sometimes beneficial. The fourth example (bottom right) shows how context can flip the underlying direct answer. The definition of local is open to discussion (Is anything within the state local?), but after reading the  $t_3$  it is clear that according to Speaker A the university he went to is not local.
Which turns around a yes-no question are the most important? We show which turns annotators selected as useful to determine the underlying direct answers in Figure 4. Unsurprisingly, the turn immediately after the yes-no question (after $_1$ ) is almost always useful (96.4%). The next turns by either speaker are often useful, including the follow-up turn by the speaker asking the yes-no question (after $_2$ , 20.9%). The turns before the yes-no question (prev $_3$ -prev $_1$ ) are less often important. We also found that the turn before the yes-no question is the most useful (prev $_1$ , 12.5%) out of all the turns before the question.
Note that annotators almost always  $(96.4\%)$  deem useful the turn immediately following the
Figure 4: Percentage of times annotators rely in the context to determine the underlying direct answer to a yes-no question (three turns before and after; after<sub>1</sub> refers to the indirect answer).
question (after $_1$ , the indirect answer). This indicates that the indirect answer is rarely an interjection or filler (e.g., Uhm, Really?). Additionally, annotators selected at least one of the other turns in the context in  $50\%$  of questions, indicating that it is often the case that at least two turns are beneficial to determine the underlying direct answer.
We split SwDA-IA into training, development and test splits (70/15/15) randomly but making sure no split contains yes-no questions from the same
conversation. We consider each conversation snippet (yes-no question, three turns before, and three turns after) an instance, and build models to predict the underlying direct answer. Following previous work on (short) text classification and specifically on the same task with synthetic data (Louis et al., 2020), we build transformer-based classifiers fine-tuned using several strategies. We experimented with three transformers: BERT (Devlin et al., 2018), RoBERTa (Liu et al., 2019), and TODBERT (Wu et al., 2020) released by HuggingFace (Wolf et al., 2020). Note that the first two are pretrained with general-purpose English while the third is fine-tuned for task-oriented dialogues and includes an attention mechanism designed to keep track of dialogues. While the three transformers obtained roughly the same results (within 0.02 F1), RoBERTa outperformed the other two and we only report results with RoBERTa.
The ground truth depends on context beyond the yes-no question and indirect answer (Section 4). A RoBERTa-based classifier, however, may not benefit from context. In order to experiment with context, we conduct three types of experiments: feeding to the models (a) only the question and the turn immediately following the question (i.e., indirect answer), (b) the full context (i.e., seven turns), and (c) only the annotator-selected turns during the annotation process. We use a separator token to indicate a new turn in the input sequence.
Fine-Tuning RoBERTa We fine-tuned the RoBERTa-based classifier using corpora for related tasks and SwDA-IA, our corpus. Specifically, we used the following corpora in our experiments:
MNLI is a corpus for natural language inference (Williams et al., 2018). It contains premise-hypothesis pairs where the premise entails, is neutral with respect to, or contradicts the hypothesis. It contains 392k pairs for training, 9k for development, and 9k for test. The fine-tuned RoBERTa classifier obtains  $83\%$  development accuracy. Following Louis et al. (2020), we rewrite questions into declarative statements and map entailment to Yes, neutral to Middle, and contradiction to No.
BoolQ is a corpus for yes-no question answering (Clark et al., 2019). Questions were submitted to a search engine and Wikipedia articles containing and not containing the answer are included for each question. It contains 9.4k questions for training, 3.2k for development, and 3.2k for test. The fine-tuned RoBERTa classifier with BoolQ obtains
$73\%$  development accuracy. We map their Yes and No (correct and incorrect) to our Yes and No.