SlowGuess/NAACL_2022_OCR huggingface dataset:
Circa is a corpus with yes-no questions and indirect answers (Louis et al., 2020). Unlike SwDA-IA, which includes questions and answers from real conversations, Circa collected questions and answers from crowdworkers who were given one of ten scenarios (Section 2). In other words, Circa does not include naturally occurring questions and answers. Additionally, Circa does not consider context as all answers are a single turn. Circa contains 3,431 yes-no questions and up to 10 indirect answers for each (total: 34,268 question-answer pairs). The fine-tuned RoBERTa classifier with Circa obtains  $78\%$  development accuracy. We map their Middle and I am not sure to our Middle.
Our Corpus: SWDA-IA We consider three versions of fine-tuning with our corpus: taking into account only the Question, only the Answer, or both (QA). Intuitively, the indirect answer should be the most useful to determine the underlying direct answer but the question may also help.
Table 3 presents results fine-tuning with each corpus and the best performing combinations of two or more corpora. The supplementary materials provide results with all combinations and detailed results (P, R, and F1) for each label.
Training with the full context around the yes-no question (three turns before and three turns after) yields worse results than only training with the yes-no question and turn immediately after (top block vs. middle block). This may seem surprising, however, it is known that keeping track of a conversation across several turns is challenging (Kim et al., 2020). Regardless of whether the model considers context, we observe that MNLI does not transfer to our task (0.13 and 0.12 with and without context, lower than the majority baseline) and Circa is the most useful out of the three previous corpora (MNLI, BooLQ, and Circa; 0.32 and 0.27 F1).
Fine-tuning with combinations of previous corpora does not surpass the 0.32 F1 obtained with Circa. Fine-tuning with SwDA-IA, however, brings substantial improvements. In particular, fine-tuning with SWDA-IA_QA in addition to combinations of previous corpora yields statistically significant higher results (indicated with  $\dagger$ ; best: 0.46 F1).
These results lead to the conclusion that determining indirect answers to yes-no questions in real
All labels	Yes	ProbYes	Middle	ProbNo	No
P	R	F1	F1	F1	F1	F1	F1
Majority Baseline	0.18	0.42	0.25	0.59	0.00	0.00	0.00	0.00
RoBERTa without context and tuned with ...
MNLI	0.24	0.20	0.12*	0.32	0.00	0.14	0.00	0.00
BoolQ	0.20	0.35	0.25	0.50	0.00	0.00	0.00	0.00
Circa	0.31	0.34	0.32	0.58	0.04	0.05	0.03	0.35
MNLI+Circa	0.20	0.23	0.21	0.42	0.09	0.00	0.04	0.14
MNLI+BoolQ+Circa	0.26	0.29	0.24	0.50	0.00	0.20	0.04	0.03
SwDA-IA_Q	0.18	0.43	0.26	0.45	0.18	0.22	0.22	0.18
SwDA-IA_A	0.42	0.45	0.43*	0.60	0.29	0.40	0.09	0.31
SwDA-IA_QA	0.44	0.45	0.44*	0.62	0.31	0.33	0.08	0.44
MNLI+SwDA-IA_QA	0.46	0.43	0.45*†	0.58	0.25	0.44	0.16	0.36
BoolQ+Circa+SwDA-IA_QA	0.46	0.48	0.46*†	0.64	0.22	0.45	0.24	0.42
MNLI+BoolQ+Circa+SwDA-IA_QA	0.41	0.40	0.40*†	0.54	0.30	0.34	0.20	0.34
RoBERTa with full context and tuned with ...
MNLI	0.19	0.16	0.13*	0.12	0.00	0.20	0.00	0.00
BoolQ	0.20	0.18	0.10*	0.32	0.00	0.00	0.00	0.00
Circa	0.24	0.28	0.27	0.55	0.03	0.00	0.00	0.20
BoolQ+Circa	0.24	0.33	0.28	0.50	0.24	0.00	0.16	0.07
MNLI+BoolQ+Circa	0.21	0.26	0.23	0.34	0.28	0.08	0.00	0.10
SwDA-IA_Q	0.32	0.39	0.33	0.57	0.08	0.26	0.02	0.24
SwDA-IA_A	0.38	0.40	0.38	0.53	0.22	0.26	0.12	0.38
SwDA-IA_QA	0.41	0.43	0.43*	0.59	0.24	0.32	0.14	0.39
MNLI-SwDA-IA_QA	0.44	0.42	0.42*†	0.54	0.38	0.16	0.14	0.38
BoolQ+Circa+SwDA-IA_QA	0.39	0.42	0.40*†	0.58	0.24	0.26	0.14	0.28
MNLI+BoolQ+Circa+SwDA-IA_QA	0.34	0.40	0.36*†	0.56	0.19	0.20	0.16	0.18
RoBERTa with annotator-selected context and tuned with ...
SwDA-IA_QA	0.43	0.47	0.45*	0.62	0.24	0.40	0.12	0.42
BoolQ+SwDA-IA_QA	0.42	0.44	0.43*†‡	0.58	0.30	0.35	0.14	0.41
MNLI+Circa+SwDA-IA_QA	0.52	0.48	0.49*†‡	0.64	0.40	0.42	0.16	0.42
MNLI+BoolQ+Circa+SwDA-IA_QA	0.44	0.46	0.44*†‡	0.64	0.30	0.37	0.15	0.41
Table 3: Results obtained with the test set. We present results with a RoBERTa-based classifier fine-tuned with two related corpora (MNLI, BoolQ), synthetic data for the same problem (Circa), and our corpus (SwDA-IA; only Questions, only Answers, or both). We only show the best performing combinations; the supplementary materials provide all of them. We indicate statistical significance (McNemar's test (McNemar, 1947) with  $p < 0.05$ ) as follows: * with respect to the baseline, † with respect to the same fine-tuning except SwDA-IA, and ‡ with respect to the model with full context. Training with real conversation (SwDA-IA) is crucial.
conversation requires fine-tuning with real conversations, as F1 scores jump over  $40\%$  when doing so (without context: 0.32 vs. 0.46, with context: 0.28 vs. 0.43). Further, the task barely benefits from the synthetic examples in Circa, MNLI, or BoolQ (SwDA-IA_QA F1: 0.44 and 0.45, best model fine-tuning with additional corpora: 0.46 and 0.49).
Full vs. Annotator-Selected Context Since the ground truth depends on context (Section 2) and our model obtains worse results with full context, it is reasonable to conclude that our RoBERTa-based classifier is unable to extract the required information from context. The bottom block of Table 3, however, shows that context is not useless. Indeed, feeding only the conversation turns handpicked as useful by annotators (out of all the context we con
sider) brings statistically significant better results than feeding the full context. These results are unrealistic as they require manual annotations, but lead to the conclusion that models that better understand context are worth exploring.
5.1.1 Out-of-Domain Evaluation
In order to investigate whether models trained with SwDA-IA can determine underlying direct answers to yes-no questions from not only SwDA-IA (in-domain evaluation) but also other real conversations, we conducted an out-of-domain evaluation. Specifically, we annotated 200 yes-no questions from MRDA (Shriberg et al., 2004), a corpus of meeting transcripts, with the same procedure from Section 3. The results (Table 11 in the supple
Long sentences
A: He is still living by himself in a little farmhouse. My grandmother died a couple of years ago. But he doesn’t want to move away.
B: Uh huh.
A: And he recently had to have an operation but he just really doesn’t want to go to a nursing home.
B: Is he able to, uh, still do everything himself pretty well?
A: Well, he was until this operation. He has arthritis.
B: Oh, yeah.
A: And now I don’t really think he’s doing that well. I have one aunt that really looks after him a lot.
Gold: ProbNo; Predicted: Yes
Explicit yes in context
A: Oh how neat.
B: So, she’d always dreamed of doing that too.
A: Yeah, that’s great.
B: So. Yeah. So is there any place you would talk me into?
A: Uh.
B: It sounds like we’ve been to some of the same places.
Gold: Middle; Predicted: Yes
Negation in question or indirect Answer
A: They're not quite elderly, huh.
B: No. So I haven't, uh, really been in that situation although they are thinking about my grandmother but, uh A: Uh huh.
B: But that's really about it. How about you? Have you been in that situation yet?
A: Uh, no not for my parents. I was around, uh, two sets of grandparents, uh, quite a bit.
B: Uh huh.
A: we, we put one, I put one grandfather in a rest home.
Gold: Yes; Predicted: No
Negation in context
A: Uh, I work out with free weights.
B: No, uh, I mean the running.
A: Oh, uh, yeah. It is really the, uh, aerobic work out part.
B: You do it, you do a mile in about eight minutes or less?
A: Uh, about seven minutes.
B: Uh huh. Yeah. Then you don't get, uh, out of breath.
A: Uh, no I do.
Gold: Yes; Predicted: No
Table 4: Examples of the most frequent error types by the best model. The yes-no question and indirect answer are in italics. We show context as shown to annotators during the corpus creation process.
mentary materials) show (unsurprisingly) worse results across all models. Interestingly, the results also show a similar trend than the in-domain evaluation: (a) SwDA-IA_QA is crucial and transfer learning with other corpora yields small benefits, and (b) annotator-selected corpus is beneficial.
We conduct a manual qualitative analysis of 200 errors made by our best model (second row from the bottom in Table 3). This analysis sheds lights into what kind of yes-no questions, indirect answers, and context are the hardest. Table 12 in the supplementary materials presents the most common error types, and Table 4 presents examples.
We observe long sentences (over 20 tokens) in most errors (62.18%). It is worth noting that long sentences do not lead to many catastrophic mistakes: only  $3.80\%$  are between Probably No and Yes,  $3.80\%$  between Yes and No and  $2.37\%$  between Yes and Probably No. If context contains a yes token, the model almost always predicts Yes. This accounts for  $19.46\%$  of errors. As exemplified in the bottom left example in Table 4, this error occurs even if the yes token is before the yes-no question. We also observed negation in many errors.  $11.08\%$  of errors occur when there is a negation in either the question or the turn after (i.e., the turn after the question), and  $7.28\%$  when the negation is else-
where in the context. In this case, the model predicts either No or Probably No although the gold label is Yes—likely because it (wrongly) learned that negation always indicates a No direct answer. The top and bottom right examples in Table 4 show errors that contain a negation in context.
Yes-no questions with indirect answers are common in real conversations— $77.4\%$  of 5-minute phone conversations include at least one. In this paper, we investigate the underlying direct answers to such questions in real conversations.
We have presented SwDA-IA, the first corpus with annotations for this task on top of real conversations. We show that determining the underlying direct answer requires taking into account context, as the ground truth changes depending on whether we show annotators context around the question. Our analysis also shows that context after the question is more useful, including turns by both the author of the question and the other speaker. Experimental results demonstrate that solving the task with real conversations is challenging (F1: 0.46). More importantly, (a) doing so barely benefits from fine-tuning with related tasks (MNLI and BoolQ) and (b) Circa, a corpus of synthetic questions and synthetic indirect answers, barely outperforms the majority baseline with real conversations.
Our future plans include designing models that better encode conversational context in order to obtain better results. We are also interested in exploring applications in dialogue systems to avoid inconsistencies due to misunderstandings of indirect answers to yes-no questions.
Penelope Brown and Stephen C Levinson. 1978. Universals in language usage: Politeness phenomena. In Questions and politeness: Strategies in social interaction, pages 56-311. Cambridge University Press.
Jonathan P Chang, Caleb Chiam, Liye Fu, Andrew Z Wang, Justine Zhang, and Cristian Danescu-Niculescu-Mizil. 2020. Convokit: A toolkit for the analysis of conversations. arXiv preprint arXiv:2005.04246.
Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wentau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018. QuAC: Question answering in context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2174-2184, Brussels, Belgium. Association for Computational Linguistics.
Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2924–2936, Minneapolis, Minnesota. Association for Computational Linguistics.
Marie-Catherine de Marneffe, Scott Grimm, and Christopher Potts. 2009. Not a simple yes or no: Uncertainty in indirect answers. In Proceedings of the SIGDIAL 2009 Conference, pages 136-143, London, UK. Association for Computational Linguistics.
Marie-Catherine de Marneffe, Christopher D. Manning, and Christopher Potts. 2010. "was it good? it was provocative." learning the meaning of scalar adjectives. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 167-176, Uppsala, Sweden. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Sarah E. Finch, James D. Finch, Daniil Huryn, William M. Hutsell, Xiaoyuan S. Huang, Han He, and Jinho D. Choi. 2021. An Approach to Inference-Driven Dialogue Management within a Social Chatbot. In Proceedings of the 4th Alexa Prize Socialbot Grand Challenge, AlexaPrize'21.
Nancy Green and Sandra Carberry. 1999. Interpreting and generating indirect answers. Computational Linguistics, 25(3):389-435.
Julia Bell Hirschberg. 1985. A theory of scalar implications (natural languages, pragmatics, inference). Ph.D. thesis, University of Pennsylvania.
Beth Ann Hockey, Deborah Rossen-Knill, Beverly Spejewski, Matthew Stone, and Stephen Isard. 1997. Can you predict responses to yes/no questions? yes, no, and stuff. In *Fifth European conference on speech communication and technology*. CiteSeer.
Daniel Jurafsky, Elizabeth Shriberg, and Debra Bíasca. 1997. Switchboard SWBD-DAMSL shallow-discourse-function annotation coders manual, draft 13. Technical Report 97-02, University of Colorado, Boulder Institute of Cognitive Science, Boulder, CO.
Sungdong Kim, Sohee Yang, Gyuwan Kim, and Sang-Woo Lee. 2020. Efficient dialogue state tracking by selectively overwriting memory. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 567-582, Online. Association for Computational Linguistics.
J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. biometrics, pages 159-174.
Mike Lewis, Denis Yarats, Yann Dauphin, Devi Parikh, and Dhruv Batra. 2017. Deal or no deal? end-to-end learning of negotiation dialogues. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2443-2453, Copenhagen, Denmark. Association for Computational Linguistics.
Chia-Wei Liu, Ryan Lowe, Iulian Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2122-2132, Austin, Texas. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach.
Annie Louis, Dan Roth, and Filip Radlinski. 2020. "I'd rather just go to bed": Understanding indirect answers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7411-7425, Online. Association for Computational Linguistics.
Quinn McNemar. 1947. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2):153-157.
Ian Plewis and Thomas Coram Research Unit. 1982. Statistical methods for rates and proportions.
Siva Reddy, Danqi Chen, and Christopher D Manning. 2019. Coqa: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249-266.
Deborah Rossen-Knill, Beverly Spejewski, Beth Ann Hockey, Stephen Isard, and Matthew Stone. 1997. Yes/no questions and answers in the map task corpus.
Elizabeth Shriberg, Raj Dhillon, Sonali Bhagat, Jeremy Ang, and Hannah Carvey. 2004. The ICSI meeting recorder dialog act (MRDA) corpus. In Proceedings of the 5th SIGdial Workshop on Discourse and Dialogue at HLT-NAACL 2004, pages 97-100, Cambridge, Massachusetts, USA. Association for Computational Linguistics.
Anna-Brita Stenström. 1984. Questions and responses in English conversation. Krieger Pub Co.
Jianheng Tang, Tiancheng Zhao, Chenyan Xiong, Xiaodan Liang, Eric Xing, and Zhiting Hu. 2019. Target-guided open-domain conversation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5624-5634, Florence, Italy. Association for Computational Linguistics.
Marta Tatu. 2005. Automatic discovery of intentions in text and its application to question answering. In Proceedings of the ACL Student Research Workshop, pages 31-36, Ann Arbor, Michigan. Association for Computational Linguistics.
Xuewei Wang, Weiyan Shi, Richard Kim, Yoojung Oh, Sijia Yang, Jingwen Zhang, and Zhou Yu. 2019. Persuasion for good: Towards a personalized persuasive dialogue system for social good. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5635-5649, Florence, Italy. Association for Computational Linguistics.
Sean Welleck, Jason Weston, Arthur Szlam, and Kyunghyun Cho. 2019. Dialogue natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3731-3741, Florence, Italy. Association for Computational Linguistics.
Tsung-Hsien Wen, David Vandyke, Nikola Mrkšić, Milica Gašić, Lina M. Rojas-Barahona, Pei-Hao Su, Stefan Ultes, and Steve Young. 2017. A network-based end-to-end trainable task-oriented dialogue system. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 438-449, Valencia, Spain. Association for Computational Linguistics.
Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Chien-Sheng Wu, Steven C.H. Hoi, Richard Socher, and Caiming Xiong. 2020. TOD-BERT: Pre-trained natural language understanding for task-oriented dialogue. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 917-929, Online. Association for Computational Linguistics.
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Personalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2204-2213, Melbourne, Australia. Association for Computational Linguistics.
A.1 Collecting Yes-No Questions with Indirect Answers
Dialogue acts to select yes-no questions The process to collect yes-no questions from SwDA uses dialogue act annotations (Section 3.1). Here is the full list of dialogue acts we consider and their descriptions:
- qh: Rhetorical Questions
- qy: Yes-No question
- qy $\wedge$ d: Declarative Yes-No question
-  $\wedge g$ : Tag-Question
- qy $\wedge$ t: Yes-No question about task
- qy∧r: Yes-No question repeat self
- qy $\wedge$ m: Yes-No question mimic other
- qy $\wedge$ h: Question in response to a question
- qy∧c: Yes-No question about communication
-  $q_{Y} \wedge 2$ : Yes-No question collaborative completion
-  $q_{Y}(\wedge q)$ : Yes-No question quoted material
- qy $\wedge$ g: Yes-No question tag-question
- qy $\wedge$ g $\wedge$ t: Yes-No question tag-question about task
- qy $\wedge$ g $\wedge$ r: Yes-No question tag-question repeat self
- qy $\wedge$ g $\wedge$ c: Yes-No question tag-question about communication
- qy $\wedge$ d $\wedge$ t: Declarative Yes-No question about task
- qy $\wedge$ d $\wedge$ r: Declarative Yes-No question repeat self
- qy $\wedge$ d $\wedge$ m: Declarative Yes-No question mimic other
- qy $\wedge$ d $\wedge$ h: Declarative Yes-No question in response to a question
- qy∧d∧c: Declarative Yes-No question about communication
- qy $\wedge$ d ( $\wedge$ q): Declarative Yes-No question quoted material
- qy $\wedge$ c $\wedge$ r: Yes-No question about-communication repeat self
Dialogue acts to discard yes-no questions with indirect answers We also use dialogue acts to discard yes-no questions with indirect answers (Section 3.1). Here is the full list of dialogue acts we consider and their descriptions:
- ny: Yes answers
- nn: No answers
- ny $\wedge$ r: Yes answers repeat self
- nn $\wedge$ r: No answers repeat self
b: Acknowledge (Backchannel)
- br: Signal-non-understanding
x: Non-verbal
- aa: Agree/Accept
Analyzing Yes-No Questions with Indirect Answers The yes-no questions with indirect answers in SwDA-IA have an average length of 39 tokens.  $41.4\%$  of the questions are indirect, i.e., do not contain a question mark despite they are yes-no questions. For example, You must be the supervisor in this office. is an indirect yes-no question. Figure 5 displays the most salient tokens in the yes-no questions and indirect answers. Finally, the most common first tokens in the questions are as follows:
- Do:  $8\%$
Well:  $5\%$
- Is:  $4\%$
- You:  $3.9\%$
So:  $3.8\%$
- And:  $3\%$
Figure 5: Word cloud displaying the most salient tokens of the yes-no questions (top) and indirect answers (bottom) in SwDA-IA.
- Did: 2.7%
Are:  $2.3\%$
- Do:  $2.1\%$
- Have:  $1.4\%$
A.2 Corpus Analysis: Context and Underlying Direct Answers
A.3 Experiments and Discussion
Tables 7-10 presents additional results complementing the results in Table 3 of the main paper. Table 11 presents the detailed out-of-domain evaluation (Section 5.1.1 in the main paper).
For all the models, we tuned the learning rate (values 5e-5, 3e-5, 2e-5), number of epochs (2, 3, 4), and batch size (16, 32) in an exhaustive combination of these parameters. Table 6 presents the hyperparameter settings with best performance on the development set fine-tuned with RoBERTa and no context.
A.4 Error Analysis
Table 12 presents the most common error types by our best model. This table complements Section 6 in the main paper.
$t_{-3}$  A: really
$t_{-2}$  ,B:But she started it
$t_{-1}$  ,A:That's good
$t_0$ , B: Well, she was depressed to begin with right?
$t_1$ , A: That's one way to see it
$t_2$ , B: Yeah.
$t_3$ , A: But, she also might be really calm.
without context: Yes; with context: Middle
$t_{-3}$ , A: I like to see you drive through Burger King now and the bags are recycled paper
$t_{-2}$  ,B:Uh huh.
$t_{-1}$  A: I feel like people are more aware of it or becoming more aware of it
$t_0$ , B: do you like to buy more recycled item
$t_1$ , A: I'm not as good about searching something out like that
$t_2$ , B: oh really.
$t_3$ , A: but I have been reading a lot about it
without context: No; with context: Middle
Table 5: Examples of yes-no questions  $(t_0)$  with indirect answers  $(t_1)$ . Determining the underlying direct answers requires taking into account more conversation context than the question and indirect answer: the ground truth annotated by humans changes depending on whether they also have access to context  $(t_{-3}, t_{-2}, t_{-1}, t_2, \text{and} t_3)$ . This examples complement Table 2 in the main paper with additional examples of less frequent label changes.
Model	learning rate	epochs	batch size
BoolQ+Circa+SwDA-IA_QA	2e-5	3	16
SwDA-IA_QA	2e-5	4	16
MNLI+SwDA-IA_QA	2e-5	4	16
BoolQ+SwDA-IA_QA	3e-5	3	16
Circa+SwDA-IA_QA	2e-5	4	16
MNLI+BoolQ+SwDA-IA_QA	3e-5	3	16
MNLI+Circa+SwDA-IA_QA	2e-5	3	16
BoolQ+Circa+SwDA-IA_QA	2e-5	4	16
MNLI+BoolQ+Circa+SwDA-IA_QA	3e-5	3	16
Table 6: Models with hyperparameter settings of best performance on the development set fine-tuned with RoBERTa and no context.