Venue: NAACL 2022

# Disentangling Indirect Answers to Yes-No Questions

in Real Conversations

Krishna C. Sanagavarapu\({}^{1}\), Jathin P. Singaraju\({}^{2}\), Anusha Kakiletti\({}^{2}\),

**Anirudh Kaza\({}^{2}\), Aaron A. Mathews\({}^{2}\), Helen Li\({}^{2}\), Nathan R. Brito\({}^{2}\), Eduardo Blanco\({}^{1}\) \({}^{1}\)**Arizona State University, \({}^{2}\)University of North Texas,

{ksanagav, eduardo.blanco}@asu.edu, {pran.singaraju, arkaza820}@gmail.com, {akakileti, AaronMathews, helenli, Nathanbrito}@unt.edu

###### Abstract

In this paper, we explore the task of determining indirect answers to yes-no questions in real conversations. We work with transcripts of phone conversations in the Switchboard Dialog Act (SwDA) corpus and create SwDA-IndirectAnswers (SwDA-IA), a subset of SwDA consisting of all conversations containing a yes-no question with an indirect answer. We annotate the underlying direct answers to the yes-no questions (yes, probably yes, middle, probably no, or no). We show that doing so requires taking into account conversation context: the indirect answer alone is insufficient to determine the ground truth. Experimental results also show that taking into account context is beneficial. More importantly, our results demonstrate that existing corpora with synthetic indirect answers to yes-no questions are not beneficial when working with real conversations. Our best models outperform the majority baseline by a substantial margin, but the task remains a challenge (F1: 0.46).

+
Footnote †: _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 4677 - 4695

+
Footnote †: _Underlying direct answer:_ Yes

## 1 Introduction

Dialogue systems have become a reality enabled by large datasets and deep neural networks. Taskoriented Wen et al. (2017) and open-domain Tang et al. (2019) dialogue systems are commonplace. Neural approaches are popular Zhang et al. (2018) although systems based on logical inference and rules outperform networks in open-domain dialogue Finch et al. (2021). State-of-the-art systems face challenges keeping track of a conversation and avoiding inconsistencies Welleck et al. (2019). Evaluation is also an open issue as automated metrics have several drawbacks Liu et al. (2016). An alternative is to evaluate dialogue systems based on whether they can collaborate with humans to solve a problem Lewis et al. (2017) or elicit some action such as donating to a cause Wang et al. (2019).

People do not explicitly say what they mean when they speak to each other yet they seamlessly carry on conversations. For example, customers asking _Where are the $1 cups?_ reveal to the sales associate that they want to buy a (cheap) cup Tatu (2005). In this paper, we investigate the underlying direct answers to yes-no questions. A yes-no question is a question that expects a _yes_ or _no_ for an answer. As we shall see, we work with direct yes-no questions (e.g., Did you drive yourself to the airport) and indirect ones (e.g., I am not sure if you drove yourself to the airport).

Consider the conversation snippets in Figure 1. As shown in the first one, the conversation turn following a yes-no question may be a direct answer (i.e., a turn including _yes_, _no_, _obviously_, _never_ or similar keywords). Indirect answers (e.g., second snippet) are more common. Speaker B _leaving his company a month ago_ does not entail that he is jobless (and thus not working--at home or the office). Given this indirect answer alone, however, it is reasonable to conclude so thus the underlying direct answer is _no_. The broader conversation context often provides a more complete picture and the

Figure 1: Conversation snippets with a yes-no question (first turn by Sparker A). In the first example, the underlying direct answer is obvious given the answer by Speaker B. In the second example, determining the underlying direct answer requires commonsense knowledge. In the third example, it requires not only commonsense but also taking into account more than the turn by Speaker B immediately following the question.



_true_ underlying direct answer (e.g., third snippet). After a brief interjection by Speaker A, Speaker B states that he changed jobs and now travels a lot. Thus, the underlying direct answer is _yes_.

The focus of this paper is to determine the underlying direct answers to yes-no questions. Unlike previous efforts, we work with transcripts of _real conversation_ as opposed to yes-no questions and indirect answers written by annotators on demand. The main contributions are:1

Footnote 1: Corpora available at [https://github.com/krishna-chaitanya-sanagavarapu/SwDA-IA](https://github.com/krishna-chaitanya-sanagavarapu/SwDA-IA).

1. We present SwDA-IA, a corpus consisting of the 2,544 yes-no questions in Switchboard Jurafsky et al. (1997), SwDA) with _I_ndirect Answers and their underlying direct answers.
2. We show that determining underlying direct answers requires context beyond the yes-no question and the next conversation turn. Indeed, the ground truth changes depending on whether we show annotators context.
3. We demonstrate that transfer learning with related tasks as well as synthetic yes-no questions and synthetic indirect answers only bring modest improvements. Specifically, synthetic data do not transfer to real conversations.
4. We provide insights into the most common errors made by our best performing model.

The work presented here provides evidence that determining the underlying direct answer to yes-no questions _in real conversations_ is much more challenging than in synthetic data. Our best model obtains 0.49 F1 even when using only the context handpicked as important by annotators.

## 2 Previous Work

Yes-no questions and indirect answers have been studied for decades Green and Carberry (1999). Hockey et al. (1997) report that 27% of all questions in 18 hours of spontaneous speech are yes-no questions with indirect answers Rossen-Knill et al. (1997). Indirect answers are often used to ask follow-up questions or provide explanations for negative answers Stenstrom (1984), prevent wrong interpretations of direct answers Hirschberg (1985) or show politeness Brown and Levinson (1978).

Yes-no questions have received considerable attention recently. Outside of the dialogue genre Clark et al. (2019) present BoolQ, a corpus of 16,000 yes-no questions submitted to a search engine (e.g., _Does France have a Prime Minister and a President?_) along with Wikipedia articles that may contain an answer. Two recent corpora consist of dialogues generated by crowdworkers on demand after being given some text to talk about: QuAC Choi et al. (2018) and CoQA Reddy et al. (2019). Unlike the work presented here, the dialogues in these corpora are limited to questions and answers about the text provided to crowdworkers. More importantly, we work with unrestricted, natural conversations rather than dialogues between paid annotators who are asked to have a conversation that resembles a questionnaire geared towards checking for understanding of a piece of text.

Yes-no questions in real conversations have been studied before at a small scale. de Marneffe et al. (2010) work with 224 questions-answer pairs involving gradable adjectives. We borrow from them the five labels to annotate underlying direct answers. de Marneffe et al. (2009) present a typology for 623 yes-no questions from SwDA. Our corpus is four times larger. Unlike them, we show that context is crucial to determine underlying direct answers and present experimental results.

The work by Louis et al. (2020) is the closest to ours. They present Circa, 34,268 yes-no questions and indirect answers written by crowdworkers given one of ten scenarios (e.g., scenario: Talking to a friend about music preferences, Q: Do you like guiats? A: I practice playing every weekend). Unlike us, Circa assumes that the turn following a yes-no question is enough to determine the underlying direct answer. Furthermore, we show that (a) determining underlying direct answers to yes-no questions is much harder in _real conversations_ and (b) their synthetic data only bring minor improvements when working with real conversations.

## 3 SwDA-IA: A Corpus of Yes-No Questions and Indirect Answers

We present SwDA-IA, a corpus consisting of (a) the 2,544 yes-no questions in Switchboard with _I_ndirect _A_nswers and (b) manual annotations indicating the underlying direct answers. Three characteristics set our work apart. First, we work with real conversations as opposed to artificial, synthetic ones (Section 2). Second, we show that determining underlying direct answers requires taking into account context beyond the yes-no question and the next turn. Third, annotators pinpoint which turns within the context around a yes-no-question are useful to determine the underlying direct answer.

 



### Collecting Yes-No Questions with Indirect Answers

While we could work with any dialogue corpora, we chose SwDA Jurafsky et al. (1997) because it contains unrestricted conversations and the dialogue acts annotations simplify the process of identifying yes-no questions with indirect answers. SwDA contains transcripts of 1,155 five-minute phone conversations. In these conversations, two people talk about topics such as childhood, recycling, and news media. The conversations are unrestricted and often divert from the initial topic. 440 speakers participated in the corpus creation resulting in 122,646 utterances. We use the distribution in ConvoKit Chang et al. (2020) for convenience.

After manually examining all the dialogue acts, we select all conversation turns containing a dialogue act that indicates a yes-no question (see the list in the supplementary materials). This step results in 5,846 yes-no question including indirect ones (e.g., "I don't know if you are familiar with that issue." is an indirect version of "Are you familiar with that issue?"). After selecting yes-no questions, we discard those that are followed by a turn containing a direct answer. To do so, we discard turns containing a dialogue act indicating direct answers (see list in the supplementary materials). After manually examining the 2,542 questions that are not filtered, we observed that some have a direct answer. In order to avoid them, we also consider a direct answer any turn that contains _yes_, _yea_, _yeah_, _no way_, _nope_, _never_, _sure_, _right_, _you bet_, _of course_, _certainly_, _maybe_, _definitely_, or _uh huh_.

These steps select 2,376 turns with 2,544 yes-no questions followed by an indirect answer (168 turns have more than one yes-no question). Only 4.77% of all utterances in SwDA are a yes-no question. However, 43.52% of all yes-no questions have an indirect answer (2,544 out of 5,846). More importantly, 77.4% of the 5-minute conversations in SwDA contain at least one yes-no question with an indirect answer. The supplementary materials provide an analysis of the yes-no questions we work with. For example, 41.4% are indirect (e.g., do not include a question mark).

### Annotating Underlying Direct Answers

In order to determine the underlying direct answers to the 2,544 yes-no questions, we manually annotated them. Our label set consists of five options: _(definitely) Yes_, _Probably Yes_, _(in the) Middle_, _Probably No_ and _(definitely) No_. _Middle_ is chosen when annotators do not lean towards any of the other four labels (e.g., "A: Do you have kids? B: Do I have kids?". These labels are heavily inspired by de Marneffe et al. (2010). Louis et al. (2020) include a few more options (e.g., "I am not sure", "In the middle, neither yes or no" and "Other") but they report very low frequencies (0.2-1.9%). These three labels are included in our "Middle."

We found two common scenarios that require additional explanation in order to be consistent:

* If the answer is _yes under certain conditions_ or _sometimes yes_, annotators are instructed to choose _Probably Yes_. For example, the correct label for "A: Do you travel a lot for pleasure? B: We try to make one trip per year if I find a good sale" is _Probably yes_.
* If the yes-no question contains a negation, _Yes_ and _Probably Yes_ have their meaning reversed. For example, the correct label for "A: You didn't move to Alaska, right? B: I have been in Alaska for 13 years now" is _No (Yes_ would be correct if B had not moved to Alaska).

Annotation Process and QualityThe annotation process took place in two independent phases in order to investigate the role of context in determining underlying direct answers. In the first phase, annotators were only shown the yes-no question and the conversation turn immediately after (i.e., the indirect answer). In the second phase, annotators were shown three turns before and after the yes-no question as context. Different annotators annotated the same question in each phase to avoid potential biases (i.e., recalling answers from the previous phase). In both phases, the interface showed conversation turns after brief delays in order to encourage annotators to read the conversation snippet in order (either two turns or seven turns). In addition to selecting the underlying direct answer, in the second phase annotators also tagged the conversation turns that help them determine it. We hosted the annotation interface internally and recruited inhouse annotators. Seven native English speakers participated in the annotations.

Inter-Annotator AgreementAfter refining the explanation for each label and group discussions, we conducted a pilot with 200 questions and the seven annotators. The average weighted Fleiss' Kappa Plewis and Unit (1982) between all pairs of annotators was 0.80. Given the high agree

ment (Landis and Koch, 1977), the remaining 2,344 questions were divided into twelve batches, and two annotators annotated each batch. The average Weighted Fleiss' Kappa score for all the batches was 0.81 when not showing context to annotators (first phase) and 0.78 when showing context (second phase). Disagreements in both phases were adjudicated after manual examination. The "true" underlying direct answers are the one annotated when context is shown to annotators, and those are the ones we conduct experiments with (Section 5).

Figure 2 shows the percentage of all disagreements prior to the adjudication step when context is not and is shown to annotators (top right and bottom left). As expected given the high Kappa coefficients, most disagreements are minor: between (a) _Yes_ (or _No_) and _Probably Yes_ (or _Probably No_) or (b) _Middle_ and _Probably Yes_ or _Probably No_.

## 4 Corpus Analysis: the Role of Context

The annotations we collected in both phases differ substantially. Thus context beyond the yes-no question and indirect answer (i.e., the following turn) is needed to determine the ground truth in real conversations. To our knowledge, previous work does not take into account context (Section 2).

Table 1 presents the distribution of each label when the interface shows and does not show context to annotators. The major change is that almost half of the questions annotated with _Middle_ without context are annotated with one of the other four labels with context. Note that out of the five labels, _Middle_ is arguably the one that reveals the least information about the speakers.

Figure 3 shows how the ground truth changes when we show context to annotators. The most common change is from _Middle_ to either _Yes_ (14.9%) or _ProbYes_ (13.4%). In other words, context allows annotators to select an underlying direct answer that is more meaningful. There are also many changes from _ProbYes_ to _Yes_ (13.5%) and vice versa (13.5%), suggesting that context provides further details (a clarification, condition, etc.) to determine the underlying direct answer.

We show examples of changes in ground truth when annotators do not have and have access to context in Table 2. In the first example (top left), the indirect answer (\(t_{1}\)) repeats the question and does not provide any clue about the underlying direct answer. The next turn by A (\(t_{3}\)), however, leaves no doubt: the underlying direct answer is _Yes_. The second example (top right) shows a similar pattern, but it exemplifies negation in the question and uncertainty. Speaker A does not commit to a _Yes_ (Yes, It didn't kill anything) but rather sharing that to his knowledge it has not (_Probably Yes_).

The third example (bottom left) shows how follow-up questions in conte

\begin{table}
\begin{tabular}{l r r} \hline \multicolumn{3}{c}{\% without context} & \% with context \\ \hline Yes & 35.7 & 42.4 \\ ProbYes & 14.9 & 19.5 \\ Middle & 28.0 & 14.9 \\ ProbNo & 7.7 & 9.9 \\ No & 13.6 & 13.3 \\ \hline \end{tabular}
\end{table}
Table 1: Label distribution when annotators have access to (a) only the yes-no question and indirect answer (without context), and (b) conversational context (3 turns before and after the question, with context).

Figure 3: Heatmap of the changes in ground truth depending on whether annotators have access to context in addition to the question and indirect answer. Numbers indicate percentages. Context is crucial to determine the underlying direct answer to a yes-no question.

Figure 2: Heatmaps of the inter-annotator disagreements when annotators have access to (a) only the yes-no question and indirect answer (bottom left) and (b) conversation context. Numbers indicate percentages. Most disagreement are minor: between _ProbYes_ and either _Yes_ or _Middle_, or _ProbNo_ and either _No_ or _Middle_.

 

lying direct answer of a yes-no question (\(t_{0}\); \(t_{2}\) is also a yes-no question). Given \(t_{1}\) alone, annotators selected _Probably Yes_ as there is some uncertainty about what _same thing_ refers to. The following two turns (\(t_{2}\), \(t_{3}\)) make it clear that the underlying direct answer is _Yes_. This example also shows how context _before_ the yes-no question (\(t_{-2}\)) is sometimes beneficial. The fourth example (bottom right) shows how context can flip the underlying direct answer. The definition of _local_ is open to discussion (Is anything within the state local?), but after reading the \(t_{3}\) it is clear that according to Speaker A the university he went to is not local.

Which turns around a yes-no question are the most important?We show which turns annotators selected as useful to determine the underlying direct answers in Figure 4. Unsurprisingly, the turn immediately after the yes-no question (after\({}_{1}\)) is almost always useful (96.4%). The next turns by either speaker are often useful, including the follow-up turn by the speaker asking the yes-no question (after\({}_{2}\), 20.9%). The turns before the yes-no question (prev\({}_{3}\)-prev\({}_{1}\)) are less often important. We also found that the turn _before_ the yes-no question is the most useful (prev\({}_{1}\), 12.5%) out of all the turns before the question.

Note that annotators almost always (96.4%) deem useful the turn immediately following the question (after\({}_{1}\), the indirect answer). This indicates that the indirect answer is rarely an interjection or filler (e.g., _Uhm_, _Really?_). Additionally, annotators selected at least one of the other turns in the context in 50% of questions, indicating that it is often the case that at least two turns are beneficial to determine the underlying direct answer.

## 5 Experiments and Discussion

We split SwDA-IA into training, development and test splits (70/15/15) randomly but making sure no split contains yes-no questions from the same

Figure 4: Percentage of times annotators rely in the context to determine the underlying direct answer to a yes-no question (three turns before and after; after\({}_{1}\) refers to the indirect answer).

 

conversation. We consider each conversation snippet (yes-no question, three turns before, and three turns after) an instance, and build models to predict the underlying direct answer. Following previous work on (short) text classification and specifically on the same task with synthetic data Louis et al. (2020), we build transformer-based classifiers fine-tuned using several strategies. We experimented with three transformers: BERT Devlin et al. (2018), RoBERTa Liu et al. (2019), and TODBERT Wu et al. (2020) released by HuggingFace Wolf et al. (2020). Note that the first two are pretrained with general-purpose English while the third is fine-tuned for task-oriented dialogues and includes an attention mechanism designed to keep track of dialogues. While the three transformers obtained roughly the same results (within 0.02 F1), RoBERTa outperformed the other two and we only report results with RoBERTa.

The ground truth depends on context beyond the yes-no question and indirect answer (Section 4). A RoBERTa-based classifier, however, may not benefit from context. In order to experiment with context, we conduct three types of experiments: feeding to the models (a) only the question and the turn immediately following the question (i.e., indirect answer), (b) the full context (i.e., seven turns), and (c) only the annotator-selected turns during the annotation process. We use a separator token to indicate a new turn in the input sequence.

Fine-Tuning RoBERTaWe fine-tuned the RoBERTa-based classifier using corpora for related tasks and SwDA-IA, our corpus. Specifically, we used the following corpora in our experiments:

**MNLI** is a corpus for natural language inference Williams et al. (2018). It contains premise-hypothesis pairs where the premise entails, is neutral with respect to, or contradicts the hypothesis. It contains 392k pairs for training, 9k for development, and 9k for test. The fine-tuned RoBERTa classifier obtains 83% development accuracy. Following Louis et al. (2020), we rewrite questions into declarative statements and map _entailment_ to _Yes_, _neutral_ to _Middle_, and _contradiction_ to _No_.

**BoolQ** is a corpus for yes-no question answering Clark et al. (2019). Questions were submitted to a search engine and Wikipedia articles containing and not containing the answer are included for each question. It contains 9.4k questions for training, 3.2k for development, and 3.2k for test. The fine-tuned RoBERTa classifier with BoolQ obtains 73% development accuracy. We map their _Yes_ and _No_ (correct and incorrect) to our _Yes_ and _No_.

**Circa** is a corpus with yes-no questions and indirect answers Louis et al. (2020). Unlike SwDA-IA, which includes questions and answers from real conversations, Circa collected questions and answers from crowdworkers who were given one of ten scenarios (Section 2). In other words, Circa does not include naturally occurring questions and answers. Additionally, Circa does not consider context as all answers are a single turn. Circa contains 3,431 yes-no questions and up to 10 indirect answers for each (total: 34,268 question-answer pairs). The fine-tuned RoBERTa classifier with Circa obtains 78% development accuracy. We map their _Middle_ and _I am not sure_ to our _Middle_.

**Our Corpus: SWDA-IA** We consider three versions of fine-tuning with our corpus: taking into account only the _Q_uestion, only the Answer, or both (_QA_). Intuitively, the indirect answer should be the most useful to determine the underlying direct answer but the question may also help.

### Results

Table 3 presents results fine-tuning with each corpus and the best performing combinations of two or more corpora. The supplementary materials provide results with all combinations and detailed results (P, R, and F1) for each label.

Training with the full context around the yes-no question (three turns before and three turns after) yields worse results than only training with the yes-no question and turn immediately after (top block vs. middle block). This may seem surprising, however, it is known that keeping track of a conversation across several turns is challenging Kim et al. (2020). Regardless of whether the model considers context, we observe that MNLI does not transfer to our task (0.13 and 0.12 with and without context, lower than the majority baseline) and Circa is the most useful out of the three previous corpora (MNLI, BooLQ, and Circa; 0.32 and 0.27 F1).

Fine-tuning with combinations of previous corpora does not surpass the 0.32 F1 obtained with Circa. Fine-tuning with SwDA-IA, however, brings substantial improvements. In particular, fine-tuning with SWDA-IA_QA in addition to combinations of previous corpora yields statistically significant higher results (indicated with \(\uparrow\); best: 0.46 F1).

These results lead to the conclusion that determining indirect answers to yes-no questions in real 

conversation requires fine-tuning with real conversations, as F1 scores jump over 40% when doing so (without context: 0.32 vs. 0.46, with context: 0.28 vs. 0.43). Further, the task barely benefits from the synthetic examples in Circa, MNLI, or BoolQ (SwDA-IA_QA F1: 0.44 and 0.45, best model fine-tuning with additional corpora: 0.46 and 0.49).

**Full vs. Annotator-Selected Context** Since the ground truth depends on context (Section 2) and our model obtains worse results with full context, it is reasonable to conclude that our RoBERTa-based classifier is unable to extract the required information from context. The bottom block of Table 3, however, shows that context is not useless. Indeed, feeding only the conversation turns handpicked as useful by annotators (out of all the context we consider) brings statistically significant better results than feeding the full context. These results are unrealistic as they require manual annotations, but lead to the conclusion that models that better understand context are worth exploring.

#### 5.1.1 Out-of-Domain Evaluation

In order to investigate whether models trained with SwDA-IA can determine underlying direct answers to yes-no questions from not only SwDA-IA (in-domain evaluation) but also other real conversations, we conducted an out-of-domain evaluation. Specifically, we annotated 200 yes-no questions from MRDA [7], a corpus of meeting transcripts, with the same procedure from Section 3. The results (Table 11 in the supple

\begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline  & \multicolumn{3}{c}{All labels} & Yes & ProbYes & Middle & ProbNo & No \\ \cline{2-9}  & P & R & F1 & F1 & F1 & F1 & F1 & F1 \\ \hline Majority Baseline & 0.18 & 0.42 & 0.25 & 0.59 & 0.00 & 0.00 & 0.00 & 0.00 \\ \hline RoBERTa without context and tuned with ... MNLI & 0.24 & 0.20 & 0.12* & 0.32 & 0.00 & 0.14 & 0.00 & 0.00 \\ BooIQ & 0.20 & 0.35 & 0.25 & 0.50 & 0.00 & 0.00 & 0.00 & 0.00 \\ Circa & 0.31 & 0.34 & 0.32 & 0.58 & 0.04 & 0.05 & 0.03 & 0.35 \\ MNL+Circa & 0.20 & 0.23 & 0.21 & 0.42 & 0.09 & 0.00 & 0.04 & 0.14 \\ MNLI+BoolQ+Circa & 0.26 & 0.29 & 0.24 & 0.50 & 0.00 & 0.20 & 0.04 & 0.03 \\ SwDA-IA\_Q & 0.18 & 0.43 & 0.26 & 0.45 & 0.18 & 0.22 & 0.22 & 0.18 \\ SwDA-IA\_A & 0.42 & 0.45 & 0.43* & 0.60 & 0.29 & 0.40 & 0.09 & 0.31 \\ SwDA-LA\_QA & 0.44 & 0.45 & 0.44* & 0.62 & 0.31 & 0.33 & 0.08 & **0.44** \\ MNLI+SWDA-IA\_QA & 0.46 & 0.43 & 0.45*† & 0.58 & 0.25 & 0.44 & 0.16 & 0.36 \\ PoolQ+Circa+SwDA-IA\_QA & 0.46 & 0.48 & **0.46**†† & **0.64** & 0.22 & **0.45** & **0.24** & 0.42 \\ MNLI+BoolQ+Circa+SwDA-IA\_QA & 0.41 & 0.40 & 0.40*† & 0.54 & 0.30 & 0.34 & 0.20 & 0.34 \\ \hline RoBERTa with full context and tuned with ... MNLI & 0.19 & 0.16 & 0.13* & 0.12 & 0.00 & 0.20 & 0.00 & 0.00 \\ BooIQ & 0.20 & 0.18 & 0.10* & 0.32 & 0.00 & 0.00 & 0.00 & 0.00 \\ Crica & 0.24 & 0.28 & 0.27 & 0.55 & 0.03 & 0.00 & 0.00 & 0.20 \\ BooIQ+Circa & 0.24 & 0.33 & 0.28 & 0.50 & 0.24 & 0.00 & 0.16 & 0.07 \\ MNLI+BoolQ+Circa & 0.21 & 0.26 & 0.23 & 0.34 & 0.28 & 0.08 & 0.00 & 0.10 \\ SwDA-IA\_Q & 0.32 & 0.39 & 0.33 & 0.57 & 0.08 & 0.26 & 0.02 & 0.24 \\ SwDA-IA\_A & 0.38 & 0.40 & 0.38 & 0.53 & 0.22 & 0.26 & 0.12 & 0.38 \\ SwDA-AA\_QA & 0.41 & 0.43 & **0.43***� 

mentary materials) show (unsurprisingly) worse results across all models. Interestingly, the results also show a similar trend than the in-domain evaluation: (a) SwDA-IA_QA is crucial and transfer learning with other corpora yields small benefits, and (b) annotator-selected corpus is beneficial.

## 6 Error Analysis

We conduct a manual qualitative analysis of 200 errors made by our best model (second row from the bottom in Table 3). This analysis sheds lights into what kind of yes-no questions, indirect answers, and context are the hardest. Table 12 in the supplementary materials presents the most common error types, and Table 4 presents examples.

We observe **long sentences** (over 20 tokens) in most errors (62.18%). It is worth noting that long sentences do not lead to many catastrophic mistakes: only 3.80% are between _Probably No_ and _Yes_, 3.80% between _Yes_ and _No_ and 2.37% between _Yes_ and _Probably No_. If **context contains a _yes token**, the model almost always predicts _Yes_. This accounts for 19.46% of errors. As exemplified in the bottom left example in Table 4, this error occurs even if the _yes_ token is before the yes-no question. We also observed **negation** in many errors. 11.08% of errors occur when there is a negation in either the question or the turn after (i.e., the turn after the question), and 7.28% when the negation is elsewhere in the context. In this case, the model predicts either _No_ or _Probably No_ although the gold label is _Yes_--likely because it (wrongly) learned that negation always indicates a _No_ direct answer. The top and bottom right examples in Table 4 show errors that contain a negation in context.

## 7 Conclusions

Yes-no questions with indirect answers are common in real conversations--77.4% of 5-minute phone conversations include at least one. In this paper, we investigate the underlying direct answers to such questions in real conversations.

We have presented SwDA-IA, the first corpus with annotations for this task on top of real conversations. We show that determining the underlying direct answer requires taking into account context, as the ground truth changes depending on whether we show annotators context around the question. Our analysis also shows that context after the question is more useful, including turns by both the author of the question and the other speaker. Experimental results demonstrate that solving the task with real conversations is challenging (F1: 0.46). More importantly, (a) doing so barely benefits from fine-tuning with related tasks (MNLI and BoolQ) and (b) Circa, a corpus of synthetic questions and synthetic indirect answers, barely outperforms the majority baseline with real conversations.

\begin{table}
\begin{tabular}{l} \hline \hline  & Negation in question or indirect Answer \\ \hline A: They’re not quite elderly, huh. \\ B: No. So I haven’t, uh, really been in that situation although they are thinking about my grandmother but, uh A: Uh huh. \\ B: _But that’s really about it. How about you? Have you been in that situation yet?_ \\ A: _Uh, no not for my parents._ I was around, uh, two sets of grandparents, uh, quite a bit. \\ B: Uh huh. \\ A: we, we put one, I put one grandfather in a rest home. \\ _Gold_: Yes; _Predicted_: No \\ \hline \hline  & Negation in context \\ \hline A: Uh, I work out with free weights. \\ B: No, uh, I mean the running. \\ A: Oh, uh, yeah. It is really the, uh, aerobic work out part. \\ B: _You do it, you do a mile in about eight minutes or less?_ \\ A: _Un, about seven minutes._ \\ B: Ul hull. Yeah. Then you don’t get, uh, out of breath. \\ A: Uh,no I do. \\ \hline \hline  & _Gold_: Yes; _Predicted_: No \\ \hline \end{tabular}
\end{table}
Table 4: Examples of the most frequent error types by the best model. The yes-no question and indirect answer are in italics. We show context as shown to annotators during the corpus creation process.

 

Our future plans include designing models that better encode conversational context in order to obtain better results. We are also interested in exploring applications in dialogue systems to avoid inconsistencies due to misunderstandings of indirect answers to yes-no questions.

## References

* Brown and Levinson (1978) Penelope Brown and Stephen C Levinson. 1978. Universals in language usage: Politeness phenomena. In _Questions and politeness: Strategies in social interaction_, pages 56-311. Cambridge University Press.
* Chang et al. (2020) Jonathan P Chang, Caleb Chiam, Liye Fu, Andrew Z Wang, Justine Zhang, and Cristian Danescu-Niculescu-Mizil. 2020. Convokit: A toolkit for the analysis of conversations. _arXiv preprint arXiv:2005.04246_.
* Choi et al. (2018) Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wentau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018. QuAC: Question answering in context. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 2174-2184, Brussels, Belgium. Association for Computational Linguistics.
* Clark et al. (2019) Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. PoolIQ: Exploring the surprising difficulty of natural yes/no questions. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 2924-2936, Minneapolis, Minnesota. Association for Computational Linguistics.
* de Marneffe et al. (2009) Marie-Catherine de Marneffe, Scott Grimm, and Christopher Potts. 2009. Not a simple yes or no: Uncertainty in indirect answers. In _Proceedings of the SIGDIAL 2009 Conference_, pages 136-143, London, UK. Association for Computational Linguistics.
* de Marneffe et al. (2010) Marie-Catherine de Marneffe, Christopher D. Manning, and Christopher Potts. 2010. "was it good? it was provocative." learning the meaning of scalar adjectives. In _Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics_, pages 167-176, Uppsala, Sweden. Association for Computational Linguistics.
* Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. _arXiv preprint arXiv:1810.04805_.
* Finch et al. (2021) Sarah E. Finch, James D. Finch, Daniil Huryn, William M. Hutsell, Xiaoyuan S. Huang, Han He, and Jinho D. Choi. 2021. An Approach to Inference-Driven Dialogue Management within a Social Chatbot. In _Proceedings of the 4th Alexa Prize Specialhot Grand Challenge_, AlexaPrize'21.
* Green and Carberry (1999) Nancy Green and Sandra Carberry. 1999. Interpreting and generating indirect answers. _Computational Linguistics_, 25(3):389-435.
* Hirschberg (1985) Julia Bell Hirschberg. 1985. _A theory of scalar impliciture (natural languages, pragmatics, inference)_. Ph.D. thesis, University of Pennsylvania.
* Hockey et al. (1997) Beth Ann Hockey, Deborah Rossen-Knill, Beverly Spejewski, Matthew Stone, and Stephen Isard. 1997. Can you predict responses to yes/no questions? yes, no, and stuff. In _Fifth european conference on speech communication and technology_. Citeseer.
* Jurafsky et al. (1997) Daniel Jurafsky, Elizabeth Shriberg, and Debra Biasca. 1997. Switchboard SWBD-DAMSL shallow-discourse-function annotation coders manual, draft 13. Technical Report 97-02, University of Colorado, Boulder Institute of Cognitive Science, Boulder, CO.
* Kim et al. (2020) Sungdong Kim, Sohee Yang, Gyuwan Kim, and Sangwoo Lee. 2020. Efficient dialogue state tracking by selectively overwriting memory. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 567-582, Online. Association for Computational Linguistics.
* Landis and Koch (1977) J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. _biometrics_, pages 159-174.
* Lewis et al. (2017) Mike Lewis, Denis Yarats, Yann Dauphin, Devi Parikh, and Dhruv Batra. 2017. Deal or no deal? end-to-end learning of negotiation dialogues. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 2443-2453, Copenhagen, Denmark. Association for Computational Linguistics.
* Liu et al. (2016) Chia-Wei Liu, Ryan Lowe, Iulian Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 2122-2132, Austin, Texas. Association for Computational Linguistics.
* Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach.
* Louis et al. (2020) Annie Louis, Dan Roth, and Filip Radlinski. 2020. "I'd rather just go to bed": Understanding indirect answers. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 7411-7425, Online. Association for Computational Linguistics.
* McNemar (1947) Quinn McNemar. 1947. Note on the sampling error of the difference between correlated proportions or percentages. _Psychometrika_, 12(2):153-157 

* Plewis and Unit (1982) Ian Plewis and Thomas Coram Research Unit. 1982. Statistical methods for rates and proportions.
* Reddy et al. (2019) Siva Reddy, Danqi Chen, and Christopher D Manning. 2019. Coqa: A conversational question answering challenge. _Transactions of the Association for Computational Linguistics_, 7:249-266.
* Rossen-Knill et al. (1997) Deborah Rossen-Knill, Beverly Spejewski, Beth Ann Hockey, Stephen Isard, and Matthew Stone. 1997. Yes/no questions and answers in the map task corpus.
* Shriberg et al. (2004) Elizabeth Shriberg, Raj Dhillon, Sonali Bhagat, Jeremy Ang, and Hannah Carvey. 2004. The ICSI meeting recorder dialog act (MRDA) corpus. In _Proceedings of the 5th SIGdial Workshop on Discourse and Dialogue at HLT-NAACL 2004_, pages 97-100, Cambridge, Massachusetts, USA. Association for Computational Linguistics.
* Stenstrom (1984) Anna-Brita Stenstrom. 1984. _Questions and responses in English conversation_. Krieger Pub Co.
* Tang et al. (2019) Jianheng Tang, Tiancheng Zhao, Chenyan Xiong, Xiaodan Liang, Eric Xing, and Zhiting Hu. 2019. Target-guided open-domain conversation. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 5624-5634, Florence, Italy. Association for Computational Linguistics.
* Tatu (2005) Marta Tatu. 2005. Automatic discovery of intentions in text and its application to question answering. In _Proceedings of the ACL Student Research Workshop_, pages 31-36, Ann Arbor, Michigan. Association for Computational Linguistics.
* Wang et al. (2019) Xuewei Wang, Weiyan Shi, Richard Kim, Yoojung Oh, Sijia Yang, Jingwen Zhang, and Zhou Yu. 2019. Persusasion for good: Towards a personalized persuasive dialogue system for social good. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 5635-5649, Florence, Italy. Association for Computational Linguistics.
* Welleck et al. (2019) Sean Welleck, Jason Weston, Arthur Szlam, and Kyunghyun Cho. 2019. Dialogue natural language inference. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 3731-3741, Florence, Italy. Association for Computational Linguistics.
* Wen et al. (2017) Tsung-Hsien Wen, David Vandyke, Nikola Mrksic, Milica Gasic, Lina M. Rojas-Barahona, Pei-Hao Su, Stefan Ultes, and Steve Young. 2017. A network-based end-to-end trainable task-oriented dialogue system. In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers_, pages 438-449, Valencia, Spain. Association for Computational Linguistics.
* Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 1112-1122. Association for Computational Linguistics.
* Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jenrite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-art natural language processing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 38-45, Online. Association for Computational Linguistics.
* Wu et al. (2020) Chien-Sheng Wu, Steven C.H. Hoi, Richard Socher, and Caiming Xiong. 2020. TOD-BERT: Pre-trained natural language understanding for task-oriented dialogue. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 917-929, Online. Association for Computational Linguistics.
* Zhang et al. (2018) Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Personalizing dialogue agents: I have a dog, do you have pets too? In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 2204-2213, Melbourne, Australia. Association for Computational Linguistics.

## Appendix A Supplementary Materials

### Collecting Yes-No Questions with Indirect Answers

Dialogue acts to select yes-no questionsThe process to collect yes-no questions from SwDA uses dialogue act annotations (Section 3.1). Here is the full list of dialogue acts we consider and their descriptions:

* qm: Rhetorical Questions
* qy: Yes-No question
* qy\(\wedge\)d: Declarative Yes-No question
* \(\wedge\)g: Tag-Question
* qy\(\wedge\)t: Yes-No question about task
* qy\(\wedge\)r: Yes-No question repeat self
* qy\(\wedge\)m: Yes-No question mimic other
* qy\(\wedge\)h: Question in response to a question
* qy\(\wedge\)c: Yes-No question about communication
* qy\(\wedge\)2: Yes-No question collaborative completion
* qy (\(\wedge\)q) : Yes-No question quoted material
* qy\(\wedge\)g: Yes-No question tag-question 

* qy/q/l: Yes-No question tag-question about task
* qy/q/r: Yes-No question tag-question repeat self
* qy/q/c: Yes-No question tag-question about communication
* qy/d/t: Declarative Yes-No question about task
* qy/d/r: Declarative Yes-No question repeat self
* qy/d/m: Declarative Yes-No question mimic other
* qy/d/h: Declarative Yes-No question in response to a question
* qy/d/nc: Declarative Yes-No question about communication
* qy/d (\(\wedge\)q): Declarative Yes-No question quoted material
* qy/c/lr: Yes-No question about-communication repeat self

Dialogue acts to discard yes-no questions with indirect answersWe also use dialogue acts to discard yes-no questions with indirect answers (Section 3.1). Here is the full list of dialogue acts we consider and their descriptions:

* ny: Yes answers
* nn: No answers
* ny/Ar: Yes answers repeat self
* nn/r: No answers repeat self
* b: Acknowledge (Backchannel)
* br: Signal-non-understanding
* x: Non-verbal
* aa: Agree/Accept

Analyzing Yes-No Questions with Indirect AnswersThe yes-no questions with indirect answers in SwDA-IA have an average length of 39 tokens. 41.4% of the questions are indirect, i.e., do not contain a question mark despite they are yes-no questions. For example, _You must be the supervisor in this office._ is an indirect yes-no question. Figure 5 displays the most salient tokens in the yes-no questions and indirect answers. Finally, the most common first tokens in the questions are as follows:

* [topsep=0pt]
* _Do_: 8%
* _Well_: 5%
* _Is_: 4%
* _You_: 3.9%
* _So_: 3.8%
* _And_: 3%

Figure 5: Word cloud displaying the most salient tokens of the yes-no questions (top) and indirect answers (bottom) in SwDA-IA.

 

* _Did_: 2.7%
* _Are_: 2.3%
* _Do_: 2.1%
* _Have_: 1.4%

### Corpus Analysis: Context and Underlying Direct Answers

#### Experiments and Discussion

Tables 7-10 presents additional results complementing the results in Table 3 of the main paper. Table 11 presents the detailed out-of-domain evaluation (Section 5.1.1 in the main paper).

For all the models, we tuned the learning rate (values 5e-5, 3e-5, 2e-5), number of epochs (2, 3, 4), and batch size (16, 32) in an exhaustive combination of these parameters. Table 6 presents the hyperparameter settings with best performance on the development set fine-tuned with RoBERTa and no context.

#### Error Analysis

Table 12 presents the most common error types by our best model. This table complements Section 6 in the main paper.

 



\begin{table}
\begin{tabular}{l r l c} \hline \hline Model & learning rate & epochs & batch size \\ \hline BoolQ+Circa+SwDA-IA\_QA & 2e-5 & 3 & 16 \\ SwDa-IA\_QA & 2e-5 & 4 & 16 \\ MNLI+SwDA-IA\_QA & 2e-5 & 4 & 16 \\ BoolQ+SwDA-IA\_QA & 3e-5 & 3 & 16 \\ Circa*SwDA-IA\_QA & 2e-5 & 4 & 16 \\ MNLI+BoolQ+SwDA-IA\_QA & 3e-5 & 3 & 16 \\ MNLI+Execta+SwDA-IA\_QA & 2e-5 & 3 & 16 \\ NMLI+Circa+SwDA-IA\_QA & 2e-5 & 3 & 16 \\ PoolQ+Circa+SwDA-IA\_QA & 2e-5 & 4 & 16 \\ MNLI+BoolQ+Circa+SwDA-IA\_QA & 3e-5 & 3 & 16 \\ \hline \end{tabular}
\end{table}
Table 6: Models with hyperparameter settings of best performance on the development set fine-tuned with RoBERTa and no context.

\begin{table}
\begin{tabular}{l} \hline \hline \(t_{-3}\), A: really \\ \(t_{-2}\), B: But she started it \\ \(t_{-1}\), A: That’s good \\ \(t_{0}\), B: _Well, she was depressed to begin with right?_ \\ \(t_{1}\), A: _That’s one way to see it_ \\ \(t_{2}\), B: Yeah. \\ \(t_{3}\), A: But, she also might be really calm. \\ Underlying direct answer ... \\ _without context_: Yes; _with context_: Middle \\ \hline \hline \end{tabular} 
\begin{tabular}{l} \hline \hline \(t_{-3}\), A: I like to see you drive through Burger King now \\ and the bags are recycled paper \\ \(t_{-2}\), B: Uh huh. \\ \(t_{-1}\), A: I feel like people are more aware of it or becoming \\ more aware of it \\ \(t_{0}\), B: _do you like to buy more recycled item_ \\ \(t_{1}\), A: _I'm not as good about searching something out like that_ \\ \(t_{2}\), B: oh really. \\ \(t_{3}\), A: but I have been reading a lot about it \\ Underlying direct answer ... \\ _without context_: No; _with context_: Middle \\ \hline \hline \end{tabular}
\end{table}
Table 5: Examples of yes-no questions (\(t_{0}\)) with indirect answers (\(t_{1}\)). Determining the underlying direct answers requires taking into account more conversation context than the question and indirect answer: the ground truth annotated by humans changes depending on whether they also have access to context (\(t_{-3}\), \(t_{-2}\), \(t_{-1}\), \(t_{2}\), and \(t_{3}\)). This examples complement Table 2 in the main paper with additional examples of less frequent label changes.





\begin{table}
\begin{tabular}{l c c c c c c c c} \hline  & \multicolumn{3}{c}{All labels} & Yes & ProbYes & Middle & ProbNo & No \\ \cline{2-9}  & P & R & F1 & F1 & F1 & F1 & F1 & F1 \\ \hline Majority Baseline & 0.18 & 0.42 & 0.25 & 0.59 & 0.00 & 0.00 & 0.00 & 0.00 \\ \hline RoBERTa without context and tuned with ... & & & & & & & \\ MNLI & 0.24 & 0.20 & 0.12* & 0.32 & 0.00 & 0.14 & 0.00 & 0.00 \\ BooIQ & 0.20 & 0.35 & 0.25 & 0.50 & 0.00 & 0.00 & 0.00 & 0.00 \\ Circa & 0.31 & 0.34 & 0.32 & 0.58 & 0.04 & 0.05 & 0.03 & 0.35 \\ MNLI+BoolQ & 0.08 & 0.20 & 0.10* & 0.32 & 0.00 & 0.00 & 0.00 & 0.00 \\ MNLI+Circa & 0.20 & 0.23 & 0.21 & 0.42 & 0.09 & 0.00 & 0.04 & 0.14 \\ PoolO+Circa & 0.19 & 0.18 & 0.11** & 0.22 & 0.00 & 0.00 & 0.00 & 0.24 \\ MNLI+BoolQ+Circa & 0.26 & 0.29 & 0.24 & 0.50 & 0.00 & 0.20 & 0.04 & 0.03 \\ SwDA-IA\_Q & 0.18 & 0.43 & 0.26 & 0.45 & 0.18 & 0.22 & 0.22 & 0.18 \\ SwDA-IA\_A & 0.42 & 0.45 & 0.43* & 0.60 & 0.29 & 0.40 & 0.09 & 0.31 \\ SwDA-IA\_QA & 0.44 & 0.45 & 0.44* & 0.62 & 0.31 & 0.33 & 0.08 & **0.44** \\ MNLI+SwDA-IA\_QA & 0.46 & 0.43 & 0.45*† & 0.58 & 0.25 & 0.44 & 0.16 & 0.36 \\ BooIQ+SwDA-IA\_QA & 0.39 & 0.42 & 0.39*† & 0.58 & 0.22 & 0.32 & 0.06 & 0.36 \\ Circa+SWDA-IA\_QA & 0.42 & 0.44 & 0.42*



\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline  & \multicolumn{3}{c}{Yes} & \multicolumn{4}{c}{ProbYes} \\ \cline{2-7}  & P & R & F1 & P & R & F1 \\ \hline Majority Baseline & 0.42 & 1.00 & 0.59 & 0.00 & 0.00 & 0.00 \\ RoBERTa without context and tuned with ... & & & & & \\ MNLI & 0.34 & 0.38 & 0.32 & 0.00 & 0.00 & 0.00 \\ BooIQ & 0.49 & 0.52 & 0.50 & 0.00 & 0.00 & 0.00 \\ Circa & 0.51 & 0.69 & 0.58 & 0.33 & 0.02 & 0.04 \\ MNLI+BoolQ & 0.28 & 0.34 & 0.32 & 0.00 & 0.00 & 0.00 \\ MNLI+Circa & 0.40 & 0.43 & 0.42 & 0.17 & 0.06 & 0.09 \\ BoolQ+Circa & 0.15 & 0.49 & 0.22 & 0.00 & 0.00 & 0.00 \\ MNLI+BoolQ+Circa & 0.48 & 0.52 & 0.50 & 0.00 & 0.00 & 0.00 \\ SwDA-IA\_Q & 0.38 & 0.48 & 0.45 & 0.17 & 0.19 & 0.18 \\ SWDA-LA\_A & 0.58 & 0.62 & 0.60 & 0.29 & 0.30 & 0.29 \\ SWDA-IA\_QA & 0.60 & 0.64 & 0.62 & 0.30 & 0.34 & 0.31 \\ MNLI+SwDA-IA\_QA & 0.56 & 0.60 & 0.58 & 0.19 & 0.3 & 0.25 \\ PoolQ+SwDA-IA\_QA & 0.55 & 0.59 & 0.58 & 0.15 & 0.49 & 0.22 \\ Circa*SvDaA-IA\_QA & 0.59 & 0.63 & 0.60 & 0.22 & 0.30 & 0.27 \\ MNLI+BoolQ+SWDA-IA\_QA & 0.60 & 0.64 & 0.62 & 0.30 & 0.35 & **0.32** \\ MNLI+circa+SwDA-IA\_QA & 0.60 & 0.64 & 0.62 & 0.24 & 0.30 & 0.29 \\ PoolQ+CircaQ+SwDA-IA\_QA & 0.61 & 0.65 & **0.64** & 0.15 & 0.49 & 0.22 \\ MNLI+BoolQ+Circa+SwDA-IA\_QA & 0.52 & 0.56 & 0.54 & 0.28 & 0.32 & 0.30 \\ \hline RoBERTa with full context and tuned with ... & & & & & \\ MNLI & 0.22 & 0.11 & 0.12 & 0.00 & 0.00 & 0.00 \\ BookO & 0.28 & 0.35 & 0.32 & 0.00 & 0.00 & 0.00 \\ Circa & 0.54 & 0.56 & 0.55 & 0.28 & 0.02 & 0.03 \\ MNLI+BoolQ & 0.28 & 0.32 & 0.30 & 0.00 & 0.00 & 0.00 \\ MNLI+Circa & 0.18 & 0.22 & 0.20 & 0.29 & 0.02 & 0.02 \\ PoolQ+Circa & 0.48 & 0.53 & 0.50 & 0.22 & 0.28 & 0.24 \\ MNLI+BoolQ+Circa & 0.38 & 0.29 & 0.34 & 0.25 & 0.30 & 0.28 \\ SwDA-IA\_Q & 0.59 & 0.54 & 0.57 & 0.30 & 0.05 & 0.08 \\ SwDA-IA\_A & 0.50 & 0.56 & 0.53 & 0.21 & 0.23 & 0.22 \\ SWDA-IA\_QA & 0.58 & 0.59 & **0.59** & 0.22 & 0.26 & 0.24 \\ MNLI+WeWDA-IA\_QA & 0.50 & 0.58 & 0.54 & 0.33 & 0.41 & **0.38** \\ PoolQ+SwDA-IA\_QA & 0.56 & 0.62 & 0.59 & 0.18 & 0.22 & 0.20 \\ Circa+SwDA-IA\_QA & 0.62 & 0.56 & 0.58 & 0.29 & 0.07 & 0.11 \\ MNLI+BoolQ+SwDA-IA\_QA & 0.57 & 0.54 & 0.56 & 0.12 & 0.23 & 0.18 \\ MNLI+Acci*SwDA-IA\_QA & 0.54 & 0.60 & 0.58 & 0.24 & 0.18 & 0.20 \\ PoolQ+Circa+SwDA-IA\_QA & 0.55 & 0.61 & 0.58 & 0.22 & 0.26 & 0.24 \\ MNLI+BoolQ+Circa+SwDA-IA\_QA & 0.59 & 0.54 & 0.56 & 0.24 & 0.16 & 0.19 \\ \hline RoBERTa with annotator-selected context & & & & & \\ and tuned with ... & & & & & \\ MNLI & 0.28 & 0.33 & 0.30 & 0.00 & 0.00 & 0.00 \\ BooIQ & 0.18 & 0.23 & 0.20 & 0.00 & 0.00 & 0.00 \\ Circa & 0.59 & 0.58 & 0.58 & 0.30 & 0.02 & 0.04 \\ MNLI+BoolQ+SwDA-IA\_QA & 0.58 & 0.58 & 0.10 & 0.18 & 0.14 \\ MNLI+BoolQ+Circa & 0.57 & 0.59 & 0.58 & 0.10 & 0.18 & 0.14 \\ BowlQ+Circa & 0.59 & 0.58 & 0.58 & 0.28 & 0.03 & 0.06 \\ MNLI+BoolQ+Circa & 0.54 & 0.60 & 0.56 & 0.22 & 0.26 & 0.24 \\ SwDA-IA\_Q & 0.52 & 0.56 & 0.54 & 0.20 & 0.24 & 0.22 \\ SwDA-IA\_A & 0.56 & 0.60 & 0.58 & 0.26 & 0.22 & 0.24 \\ SwDA-IA\_QA & 0.58 & 0.64 & 0.62 & 0.20 & 0.28 & 0.24 \\ MNLI+WeWDA-IA\_QA & 0.59 & 0.65 & 0.62 & 0.30 & 0.32 & 0.31 \\ PoolQ+SwDA-IA\_QA & 0.54 & 0.62 & 0.58 & 0.26 & 0.34 & 0.30 \\ Circa+SwDA-IA\_QA & 0.56 & 0.60 & 0.58 & 0.18 & 0.14 & 0.16 \\ MNLI+BoolQ+SwDA-IA\_QA & 0.58 & 0.62 & 0.59 & 0.20 & 0.28 & 0.24 \\ MNLI+Circa+SwDA-IA\_QA & 0.62 & 0.66 & **0.64** & 0.38 & 0.43 & **0.40** \\ PoolQ+Circa+SwDA-IA\_QA & 0.60 & 0.65 & 0.62 & 0.26 & 0.29 & 0.27 \\ MNLI+BoolQ+Circa+SwDA-IA\_QA & 0.63 & 0.66 & 0.64 & 0.38 & 0.32 & 0.30 \\ \hline \end{tabular}
\end{table}
Table 8: Results obtained with the test set. We present Precision (P), Recall (R) and F1 scores for the _Yes_ and _Probably Yes_ labels. These results complement Table 3 in the main paper.





\begin{table}
\begin{tabular}{l r c c} \hline \hline  & \multicolumn{3}{c}{Middle} \\ \cline{2-4}  & P & R & F1 \\ \hline Majority Baseline & 0.00 & 0.00 & 0.00 \\ RoBERTa without context and tuned with ... & & & \\ MNLI & 0.22 & 0.18 & 0.14 \\ BooIQ & 0.00 & 0.00 & 0.00 \\ Circa & 0.35 & 0.02 & 0.05 \\ MNLI+BoolQ & 0.00 & 0.00 & 0.00 \\ MNLI+Circa & 0.00 & 0.00 & 0.00 \\ PoolQ+Crica & 0.00 & 0.00 & 0.00 \\ MNLI+BoolQ+Circa & 0.13 & 0.48 & 0.20 \\ SwDA-IA\_Q & 0.14 & 0.50 & 0.22 \\ SwDA-IA\_A & 0.50 & 0.29 & 0.40 \\ SWDA-IA\_QA & 0.29 & 0.35 & 0.33 \\ MNLI+SwDA-IA\_QA & 0.39 & 0.46 & 0.44 \\ BoolQ+SwDA-IA\_QA & 0.30 & 0.35 & 0.32 \\ Circa+swDa-IA\_QA & 0.19 & 0.31 & 0.24 \\ MNLI+BoolQ+SWDA-IA\_QA & 0.24 & 0.35 & 0.30 \\ MNLI+Creza+SwDA-IA\_QA & 0.24 & 0.29 & 0.28 \\ BookQ+Circa+SwDA-IA\_QA & 0.38 & 0.50 & **0.45** \\ MNLI+BoolQ+Circa+SwDA-IA\_QA & 0.28 & 0.38 & 0.34 \\ \hline RoBERTa with full context and tuned with ... & & & \\ MNLI & 0.23 & 0.18 & 0.20 \\ BooIQ & 0.00 & 0.00 & 0.00 \\ Circa & 0.00 & 0.00 & 0.00 \\ MNLI+BoolQ & 0.00 & 0.00 & 0.00 \\ MNLI+Circa & 0.22 & 0.26 & 0.24 \\ BowlQ+Circa & 0.00 & 0.00 & 0.00 \\ MNLI+BoolQ+Circa & 0.20 & 0.04 & 0.08 \\ SwDA-IA\_Q & 0.22 & 0.28 & 0.26 \\ SwDA-IA\_A & 0.25 & 0.27 & 0.26 \\ SwDA-IA\_QA & 0.31 & 0.32 & **0.32** \\ MNLI+SwDA-IA\_QA & 0.14 & 0.18 & 0.16 \\ BooIQ+SwDA-IA\_QA & 0.27 & 0.21 & 0.23 \\ Circa+SwDA-IA\_QA & 0.33 & 0.27 & 0.3 \\ MNLI+BoolQ+SwDA-IA\_QA & 0.14 & 0.18 & 0.16 \\ MNLI+Accra*SwDA-IA\_QA & 0.19 & 0.15 & 0.18 \\ BooQO+Circa+SwDA-IA\_QA & 0.24 & 0.28 & 0.26 \\ MNLI+BoolQ+Circa+SwDA-IA\_QA & 0.18 & 0.22 & 0.20 \\ \hline RoBERTa with annotator-selected context & & & \\ and tuned with ... & & & \\ MNLI & 0.19 & 0.01 & 0.02 \\ BooIQ & 0.00 & 0.00 & 0.00 \\ Circa & 0.30 & 0.02 & 0.04 \\ MNLI+BoolQ & 0.00 & 0.00 & 0.00 \\ MNLI+Circa & 0.29 & 0.02 & 0.04 \\ BooIQ+Circa & 0.32 & 0.03 & 0.06 \\ MNLI+BoolQ+Circa & 0.00 & 0.00 & 0.00 \\ SwDA-IA\_Q & 0.10 & 0.16 & 0.12 \\ SwDA-IA\_A & 0.38 & 0.44 & 0.40 \\ SwDA-IA\_QA & 0.39 & 0.41 & 0.40 \\ MNLI+sWDA-IA\_QA & 0.35 & 0.28 & 0.30 \\ BooIQ+SwDA-IA\_QA & 0.36 & 0.33 & 0.35 \\ Circa+SwDA-IA\_QA & 0.48 & 0.52 & 0.50 \\ MNLI+BoolQ+SwDA-IA\_QA & 0.26 & 0.34 & 0.30 \\ MNLI+Circa+SwDA-IA\_QA & 0.40 & 0.44 & **0.42** \\ BooIQ+Circa+SwDA-IA\_QA & 0.38 & 0.44 & 0.40 \\ MNLI+BoolQ+Circa+SwDA-IA\_QA & 0.35 & 0.40 & 0.37 \\ \hline \end{tabular}
\end{table}
Table 9: Results obtained with the test set. We present Precision (P), Recall (R) and F1 scores for the _Middle_ label. These results complement Table 3 in the main paper



\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline  & \multicolumn{3}{c}{No} & \multicolumn{4}{c}{ProbNo} \\ \cline{2-7}  & P & R & F1 & P & R & F1 \\ \hline Majority Baseline & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 \\ RoBERTa without context and tuned with ... & & & & & \\ MNLI & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 \\ BooIQ & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 \\ Circa & 0.37 & 0.34 & 0.35 & 0.25 & 0.01 & 0.03 \\ MNLI+BoolQ & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 \\ MNLI+Circa & 0.23 & 0.16 & 0.14 & 0.25 & 0.02 & 0.04 \\ BoolQ+Circa & 0.18 & 0.50 & 0.24 & 0.00 & 0.00 & 0.00 \\ MNLI+BoolQ+Circa & 0.30 & 0.02 & 0.03 & 0.21 & 0.04 & 0.04 \\ SwDA-IA_Q & 0.20 & 0.16 & 0.18 & 0.15 & 0.48 & 0.22 \\ SWDaLA\_A & 0.50 & 0.38 & 0.31 & 0.18 & 0.05 & 0.09 \\ SWDA-IA\_QA & 0.43 & 0.44 & **0.44** & 0.15 & 0.06 & 0.08 \\ MNLI+SwDA-IA\_QA & 0.56 & 0.30 & 0.36 & 0.14 & 0.21 & 0.16 \\ PoolQ+SwDA-IA\_QA & 0.34 & 0.38 & 0.36 & 0.28 & 0.04 & 0.06 \\ Circa*SwDha-IA\_QA & 0.35 & 0.37 & 0.36 & 0.15 & 0.48 & 0.21 \\ MNLI+BoolQ+SWDA-IA\_QA & 0.28 & 0.34 & 0.32 & 0.12 & 0.30 & 0.18 \\ MNLI+circa+SwDA-IA\_QA & 0.28 & 0.33 & 0.31 & 0.21 & 0.23 & 0.22 \\ PoolQ+Circa+SwDA-IA\_QA & 0.38 & 0.44 & 0.42 & 0.22 & 0.26 & **0.24** \\ MNLI+BoolQ+Circa+SwDA-IA\_QA & 0.33 & 0.34 & 0.34 & 0.19 & 0.21 & 0.20 \\ \hline RoBERTa with full context and tuned with ... & & & & & \\ MNLI & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 \\ BookQ & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 \\ Circa & 0.15 & 0.25 & 0.20 & 0.00 & 0.00 & 0.00 \\ MNLI+BoolQ & 0.28 & 0.10 & 0.14 & 0.00 & 0.00 & 0.00 \\ MNLI+Circa & 0.24 & 0.12 & 0.16 & 0.28 & 0.03 & 0.05 \\ PoolQ+Circa & 0.28 & 0.05 & 0.07 & 0.28 & 0.13 & 0.16 \\ MNLI+BoolQ+Circa & 0.30 & 0.06 & 0.10 & 0.00 & 0.00 & 0.00 \\ SwDA-IA\_Q & 0.22 & 0.26 & 0.24 & 0.20 & 0.01 & 0.02 \\ SwDA-IA\_A & 0.34 & 0.40 & 0.38 & 0.20 & 0.10 & 0.12 \\ SWDA-IA\_QA & 0.38 & 0.42 & **0.39** & 0.10 & 0.18 & 0.14 \\ MNLI+WeWDA-IA\_QA & 0.40 & 0.36 & 0.38 & 0.19 & 0.10 & 0.14 \\ PoolQ+SwDA-IA\_QA & 0.40 & 0.34 & 0.36 & 0.00 & 0.00 & 0.00 \\ Circa+SwDa-IA\_QA & 0.34 & 0.40 & 0.38 & 0.00 & 0.00 & 0.00 \\ MNLI+BoolQ+SwDA-IA\_QA & 0.30 & 0.34 & 0.32 & 0.11 & 0.10 & 0.10 \\ MNLI+Acct*SwDA-IA\_QA & 0.29 & 0.32 & 0.31 & 0.20 & 0.07 & 0.11 \\ PoolQ+Circa+SwDA-IA\_QA & 0.24 & 0.32 & 0.28 & 0.28 & 0.10 & 0.14 \\ MNLI+BoolQ+Circa+SwDA-IA\_QA & 0.16 & 0.20 & 0.18 & 0.20 & 0.14 & **0.16** \\ \hline RoBERTa with annotator-selected context & & & & & \\ and tuned with ... & & & & & \\ MNLI & 0.10 & 0.18 & 0.14 & 0.00 & 0.00 & 0.00 \\ BooIQ & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 \\ Circa & 0.32 & 0.40 & 0.38 & 0.00 & 0.00 & 0.00 \\ MNLI+BoolQ & 0.20 & 0.25 & 0.21 & 0.00 & 0.00 & 0.00 \\ MNLI+Circa & 0.36 & 0.29 & 0.33 & 0.00 & 0.00 & 0.00 \\ PoolQ+Circa & 0.29 & 0.02 & 0.04 & 0.28 & 0.02 & 0.04 \\ MNLI+BoolQ+Circa & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 \\ SwDA-IA\_Q & 0.20 & 0.26 & 0.23 & 0.08 & 0.16 & 0.11 \\ SwDA-IA\_A & 0.33 & 0.39 & 0.36 & 0.12 & 0.20 & 0.17 \\ SwDA-IA\_QA & 0.40 & 0.44 & 0.42 & 0.11 & 0.14 & 0.12 \\ MNLI+swDA-IA\_QA & 0.36 & 0.40 & 0.38 & 0.10 & 0.16 & 0.12 \\ PoolQ+SwDA-IA\_QA & 0.44 & 0.39 & 0.41 & 0.16 & 0.13 & 0.14 \\ Circa+SwDa-IA\_QA & 0.45 & 0.55 & **0.51** & 0.14 & 0.11 & 0.12 \\ MNLI+BoolQ+SwDA-IA\_QA & 0.34 & 0.40 & 0.37 & 0.18 & 0.20 & **0.19** \\ MNIL+Circa+SwDA-IA\_QA & 0.40 & 0.43 & 0.42 & 0.16 & 0.17 & 0.16 \\ PoolQ+Circa+SwDA-IA\_QA & 0.41 & 0.40 & 0.40 & 0.13 & 0.22 & 0.17 \\ MNLI+BoolQ+Circa+SwDA-IA\_QA & 0.36 & 0.44 & 0.41 & 0.13 & 0.17 & 0.15 \\ \hline \end{tabular}
\end{table}
Table 10: Results obtained with the test set. We present Precision (P), Recall (R) and F1 scores for the _No_ and _Probably No_ labels.These results complement Table 3 in the main paper.





\begin{table}
\begin{tabular}{l c c c c c c c c}  & \multicolumn{3}{c}{All labels} & Yes & ProbYes & Middle & ProbNo & No \\ \cline{2-9}  & P & R & F1 & F1 & F1 & F1 & F1 & F1 \\ \hline Majority Baseline & 0.22 & 0.40 & 0.28 & 0.61 & 0.00 & 0.00 & 0.00 & 0.00 \\ \hline RoBERTa without context and tuned with ... MNLI & 0.14 & 0.06 & 0.08 & 0.30 & 0.00 & 0.02 & 0.00 & 0.14 \\ BoolQ & 0.22 & 0.16 & 0.20 & 0.36 & 0.00 & 0.00 & 0.00 & 0.12 \\ Circa & 0.23 & 0.32 & 0.27 & 0.55 & 0.03 & 0.00 & 0.04 & 0.20 \\ MNLI+Circa & 0.24 & 0.14 & 0.18 & 0.30 & 0.05 & 0.08 & 0.00 & 0.14 \\ MNLI+BoolQ+Circa & 0.20 & 0.18 & 0.20 & 0.43 & 0.00 & 0.00 & 0.00 & 0.18 \\ SwDA-IA\_Q & 0.20 & 0.25 & 0.18 & 0.30 & 0.24 & 0.06 & 0.00 & 0.10 \\ SwDA-IA\_A & 0.31 & 0.37 & 0.32 & 0.56 & 0.13 & 0.04 & 0.00 & **0.34** \\ SWDA-IA\_QA & 0.28 & 0.37 & 0.34 & 0.58 & **0.19** & 0.18 & 0.11 & 0.30 \\ MNLI+SwDA-IA\_QA & 0.29 & 0.36 & 0.34 & 0.56 & 0.12 & 0.29 & 0.00 & 0.40 \\ BoolQ+Circa+SwDA-IA\_QA & 0.26 & 0.39 & **0.35** & **0.58** & 0.16 & **0.22** & **0.12** & 0.32 \\ MNLI+BoolQ+Circa+SwDA-IA\_QA & 0.31 & 0.32 & 0.30 & 0.50 & 0.19 & 0.21 & 0.10 & 0.20 \\ \hline RoBERTa with full context and tuned with ... MNLI & 0.17 & 0.13 & 0.10 & 0.12 & 0.00 & 0.15 & 0.00 & 0.00 \\ BoolQ & 0.18 & 0.16 & 0.09 & 0.25 & 0.00 & 0.00 & 0.00 & 0.00 \\ Circa & 0.24 & 0.18 & 0.20 & 0.42 & 0.09 & 0.14 & 0.00 & 0.04 \\ BooIQ+Circa & 0.20 & 0.30 & 0.22 & 0.44 & 0.22 & 0.00 & 0.10 & 0.07 \\ MNLI+BoolQ+Circa & 0.18 & 0.22 & 0.20 & 0.30 & 0.24 & 0.05 & 0.00 & 0.10 \\ SwDA-IA\_Q & 0.25 & 0.15 & 0.16 & 0.36 & 0.09 & 0.00 & 0.03 & 0.12 \\ SwDA-IA\_A & 0.28 & 0.32 & 0.30 & 0.43 & 0.17 & 0.23 & 0.21 & 0.18 \\ SwDA-IA\_QA & 0.34 & 0.38 & **0.32** & **0.54** & 0.18 & **0.24** & 0.10 & 0.21 \\ MNLI-SwDA-IA\_QA & 0.32 & 0.37 & 0.32 & 0.54 & **0.22** & 0.12 & 0.11 & **0.23** \\ PoolQ+Circa+SwDA-IA\_QA & 0.29 & 0.33 & 0.30 & 0.44 & 0.18 & 0.23 & **0.22** & 0.18 \\ MNLI+BoolQ+Circa+SwDA-IA\_QA & 0.20 & 0.27 & 0.25 & 0.40 & 0.10 & 0.18 & 0.10 & 0.15 \\ \hline \end{tabular}




\begin{table}
\begin{tabular}{l r c c} \hline \hline Error Type & \% & Gold & Pred. \\ \hline Long sentences & 21.20 & ProbYes & Yes \\  & 15.51 & Yes & ProbYes \\  & 9.49 & Yes & Middle \\  & 6.01 & ProbNo & No \\  & 3.80 & ProbNo & Yes \\  & 3.80 & Yes & No \\  & 2.37 & Yes & ProbNo \\ \hline Explicit _yes_ in context & 11.87 & Middle & Yes \\  & 7.59 & ProbNo & Yes \\ \hline Negation in ... question or ind. answer & 7.59 & Yes & No \\  & 3.48 & Yes & ProbNo \\ in context & 3.80 & Yes & No \\  & 3.80 & Yes & ProbNo \\ \hline \hline \end{tabular}
\end{table}
Table 12: Qualitative analysis of the errors made by our best model (Table 3, second row from the bottom). We indicate the frequency of the most common combinations of gold and predicted labels for each error type. _Context_ here does not include the yes-no question and indirect answer. This table complements Section 6 in the main paper.