Interpreting Indirect Answers to Yes-No Questions in Multiple Languages
Authors: Zijie Wang
Md Mosharaf Hossain
Shivam Mathur
Terry Cruz Melo
Kadir Bulut Ozler
Keun Hee Park
Jacob Quintero
MohammadHossein Rezaei
Shreya Nupur Shakya
Md Nayem Uddin
Eduardo Blanco
Venue: EMNLP 2023 Findings (2023)
Keywords: Multilinguality
Question Answering
Yes-no Question

Abstract: Yes-no questions expect a yes or no for an answer, but people often skip polar keywords. Instead, they answer with long explanations that must be interpreted. In this paper, we focus on this challenging problem and release new benchmarks in eight languages. We present a distant supervision approach to collect training data, and demonstrate that direct answers (i.e., with polar keywords) are useful to train models to interpret indirect answers (i.e., without polar keywords). We show that monolingual fine-tuning is beneficial if training data can be obtained via distant supervision for the language of interest (5 languages). Additionally, we show that cross-lingual fine-tuning is always beneficial (8 languages).

Interpreting Indirect Answers to Yes-No Questions in Multiple Languages
Zijie Wang" Md Mosharaf Hossain Shivam Mathur
Terry Cruz Melo'
Kadir Bulut Ozler' Keun Hee Park Jacob Quintero' MohammadHossein Rezaci
Shreya Nupur Shakya' Md Nayem Uddin Eduardo Blanco
University of Arizona AAmaoon Walmart Global Tech *Arizona State University
zi zijieaangeariz.n... edu
Abstract
en
Q: [...] Does still feel like that? A: never really felt was that way. [...] Interpretation: No
Yes-no questions expect a yes for answer, but people often skip polar keywords. Instead, they answer with long explanations that must be interpreted. In this paper, focus on this challenging problem and release new benchmarks in eight languages. We present distant supervision approach to collect training data. We also demonstrate that direct answers (i.e., with polar keywords) are useful train models to interpret indirect answers (i.e., without polar keywords). Experimental results demonstrate that monolingual fine-tuning is beneficial if training data can be obtained via distant supervision for the language of interest (5 languages). Additionally, we show that cross-lingual fine-tuning always beneficial (8 languages).
hi
Q: अम बार के बमत से का भा महमन हे (Are you satisfied with this budget?)
A: बहबर बगर केt अशंशकरशा का मनकाी [मलतार I
(This a better budget, the economy will strengthened.)
Interpretation: Yes
Q: Teoman't in konserine gidecek misin? (Are you going to Teoman's 's concert?) A: Patronum gelecek ayki planların ne olduğunu henüz söylemedi.
(My boss hasn't 't told me next month's plan.) Interpretation: Middle
Table 1: Yes-no questions and answers in English (en), Hindi (hi) and Turkish (tr). Answers do not include a yes keyword, but their interpretations are clear.
1 Introduction
Multilingual Question-Anseen Anwwering has recently re- ceived substantial attention (Ruder and Sil, 2021; Shi et al., 2022). State-of-the-ard models such as XLM-E (Chi al., 2022), however, achieve only 68% 76% F1-score on multilingual Question- Answering benchmarks such as MIQA (Lewis et al., 2020) and XQuAD (Artetxe et al., 2020). Large Language Models (LLMs) such as Instruct- GPT (Ouyang et al.. 2022) obtain promising results bench-
Yes-no questions are questions that expect yes or no for an answer. Humans, however, often an- swer these kinds of questions without using a yes or no keyword. Rather, they provide indirect answers that must be interpreted to reveal the underlying meaning (see examples in Table 1). Indirect an- swers are used to ask follow-up questions pro- vide explanations for negative answers (Thompson. 1986), prevent wrong interpretations (Hirschberg, 1985), show politeness (Brown and Levinson, 1978). This true least in English and the eight additional languages we work with. Note that ques- tion answering is usually defined as finding an an- swer to a question given a collection of documents. On the other hand, interpreting indirect answers to yes-no questions is defindd as mapping a known answer to its correct interpretation.
with several English marks (Rajpurkar et al., 2016; Choi et al., 2018; Lin et al., 2022). Closed proprietary LLMs for which the training data is unknown raise issues regarding replicability and data leakage (Carlini et al., 2021). Open-source LLMs such as LLaMa are pretrained on Latin or Cyrillic scripts (Touvron et al., 2023) thus have limitations with other scripts.
All authors except the first and last authors listed in alphabetical order.
Many NLP problems were initially investigated in English (Kann et al., 2019). Even though yes-no questions and indirect answers have been studied
The work does not relate to position Amazon. Work done at Arizona State University.

---

for decades (Hockey et al., 1997: Green and Car- berry, 1999), previous efforts to date have predomi- nantly focused on English (Section 2). this paper, we tackle this challenging problem in eight addi- tional languages: Hindi (hi), Korean (ko), Chinese (zh), Bangla (bn), Turkish (tr), Spanish (es), Nepali (ne), and Persian (fa).
Yes-No Questions have also been studied for decades (Hockey et al., 1997: Green and Carberry. 1999), Recent work includes large corpora such as BoolQ (Clark et al., 2019) (16,000 yes-no questions submitted to a search engine) and extensions includ- ing unanswerable questions (Sulem et al., 2022). Yes-no questions have also been studied within the dialogue domain (Choi et al., 2018; 8; Reddy et al., 2019). These dialogues, however, are synthetic and constrained a handful of topics. Circa (Louis et al., 2020), Friends- QTA (Damgaard et al., 2021), and SWDA-IA (Sanagavarapu et al., 2022), also ex- plore yes- no questions in dialogues (crowdsoureed, modified TV scripts, and phone conversations). We are inspired by their interpretations and use their corpora in our experiments (Section 5). All com- putational works yes-no questions to date in English. We are the first to target eight additional languages. Crucially, we do by exploring distant su- pervision and cross-lingual learning; our approach does not require tedious manual annotations.
This paper focuses on multilingual interpretation of indirect answers to yes-no questions. Doing so opens the door to several applications. For example, dialogue systems could avoid inconsistencies and contradictions (Nie et al., 2021; Li et al., 2022). Consider the examples in Table 1. Follow-up turns such as How long have you felt like that?, What else is required for your support?, and I can't 't wait see you the concert (one per example) would be puzzling and probably frustrating to hear.
The main contributions are as follows:!
1. A distant supervision approach to collect yes- no questions and direct answers along with their interpretations. We use this approach to collect training data in five languages.
Multilingual Pretraining and Learning Several efforts have investigated multilingual language model pretraining. Both mBERT (Devlin et al., 2019) and XLM-RoBERTa (Conneau al., 2020) are masked language models pretrained on mul- tilingual corpora. XLM-Align (Chi et al., 2021) improves cross- cross-lingual domain adaptation using self-labeled word alignment.
2. Evaluation benchmarks in eight languages in which no resources exist for interpreting indi- rect answers to yes-no questions.
3. Experimental results showing that training with the direct answers obtained via distant supervision is beneficial interpret indirect answers in the same language.
4. Experimental results expanding on (3) and showing that multilingual training is benefi- cial, even when no additional training data for the language of interest is available.
Previous works have also focused on better fine- tuning approaches for cross-lingual transfer (Joty et al., 2017: Zheng et al., 2021). Others use ma- chine translations to create synthetic multilingual training data or translate multilingual test bench- marks into English (Artetxe et al., 2023). In ad- dition, several efforts have been made on mul- tilingual Question Answering tasks. Wu et al. (2022) present a siamese semantic disentanglement model for multilingual machine reading compre- hension. Riabi al. (2021) propose multilingual QA dataset leveraging question generation models to generate synthetic instances. Other work devel- ops a Spanish SQuAD dataset based on a Translate- Align-Retrieve method (Carrino et al., 2020).
2 Related Work
Question Answering Researchers have targeted, among others, factual questions (Morales et al., 2016), why questions (Lal et al., 2022), questions in context (Choi et al., 2018), questions over proce- dural texts (Tandon et al., 2019), and natural ques- tions submitted to a search engine (Kwiatkowski et al., 2019). The problem is defined as finding answers to a given question (often within set of documents). Unlike this line of work, interpreting indirect answers yes-no questions about deter- mining the underlying meaning of answers not finding them.
In this paper, we avoid translations. Instead, use distant supervision. Our approach (a) is conceptually simple, (b) only requires unannotated corpora and rules for yes-no questions and direct answers in the languages of interest, and, impor- tantly, (c) obtains statistically significantly better results than cross lingull learning from English.
Code, benchmarks, and multilingual train- ing data obtained via distant supervision avail- able at https:// // wwg.thb..coo. wwt.t/oooowwwwwweeeoeee jie/ yn- quest con-multii ngual

---

Circa
SWDA-IA
Friends-QIA
Friends-QIA: 6 options). We cluster them into Yes, No and Middle following our definitions (Section 4, Appendix A) for comparison purposes. Because Circa and SWDA-IA are the only corpora with "naturally occurring" questions and answers, we choose to not work with Friends-QIA.
# yes-no qs.
34,268
2,544
5,930
context?
no
yes
no
% Interpret.
Yes 56.0
61.9
48.8
No Middle 2.8
37.5
23.2
24.6
14.9
26.6
3.2 Distant Supervision for New Languages
Table 2: Statistics of existing English corpora with yes- no questions, indirect answers, and their interpretations.
We follow distant supervision approach to collect multilingual training data for interpreting indirect answers to yes-no questions. The only require- ments are (a) (relatively) large unannotated corpora in the languages of interest and (b) rules to identify yes-no questions, direct answers, and their inter- pretations in each language. We found that native speakers can write robust rules after few iterations. Source Corpora We made an effort to identify relevant corpora for the eight languages we work with but could only do so in five languages. The other three languages (Bangla, Nepali, and Persian) are spoken by millions, and we could certainly find digital texts these languages. But (a) creating a large collection of dialogues and (b) splitting noisy transcripts into turns and sentences are outside the scope of this paper. Further, doing so would raise copyright and ethical considerations.
3 Obtaining Training Data
For training purposes, we work with six languages. Specifically, we work with existing English cor- pora (Section 3.1) and corpora obtained in five additional languages via distant supervision (Sec- tion 3.2). As shall see, multilingual transfer learning is successful even when examples in the language of interest are available (Section 5.3).
3.1 Existing English Corpora
There are three English corpora that include yes-no questions, indirect answers, and their interpreta- tions: Circa (Louis et al., 2020), SWDA-IA (Sana- gavarapu et al., 2022) and Friends- RTAA (Damgaard et al., 2021). Table 2 presents statistics. Context refers to the text around the question and indirect answer (i.e.. dialogue turns before and after).
For Chinese, we select (a) NaturalConv (Wang et al., 2021), a synthetic dialogue dataset writ- ten by crowdworkers covering several topics, and (b) LCCC-base (Wang et al., 2020), posts and replies extracted from Weibo, a Chinese social media platform. For Spanish, we choose Call- Friend (Canavan and Zipperlen, 1996). a corpus consisting of unscripted telephone conversations. For Hindi, we collect questions and answers from Twitter using their API. For Korean, we select question-answeringg answering corpus from AI Hub, which includes civil complaints and replies from pub- lie organizations. For Turkish, we identify FAQ (i.e., Frequently Asked Questions) and CQA (i.e., Community Question Answering) datasets from MFAQ (De Bruyn et al., 2021). All of these cor- pora used in accordance with their licenses. These corpora are diverse not only in terms of language. Indeed, they include different genres (written forum discussions, dialogue transcripts, question-answer pairs, etc.) and domains (social media, informal conversations, etc).
Circa was created by asking crowdworkers to write 34k yes-no questions that fit 9 scenarios (e.g., friends talking about food) and indirect answers. It does not include context. We note that the fre- quency of Middle interpretation is much lower than the other two interpretations. SWDA-IA includes 2.5k yes-no questions and indirect answers from SWDA (Stolcke et al., 2000), telephone conver- sation dataset. It includes context (three turns be- fore and after). Unlike Circa, questions and an- swers come from transcriptions of (almost) uncon- strained conversations. Friends-OUe QIA includes 5.9k yes-no questions and indirect answers derived from Friends, a TV show. Unlike Circa and SWDA-IA, questions and answers in Friends- QIA were man- ually modified to facilitate the task. For example, they remove some yes and no keywords in answers (e.g., yes, yeah, yep) but not others (e.g.. of course, absolutely). Further, they add relevant information from the context to questions and delete interjec- tions (c.g., Hey!) among others.
Identifying Yes-No Questions We define rules based on lexical matching to identify yes-no ques-
The three datasets do not consider the same inter- pretations (Circa: 8 options, SWDA-IA: 5 options,
Pwww. a hub. or.kr

---

Hindi
Korean
Chinese
Turkish
Spanish
# total turms context?
n/a
n/a 7,220k
4,264k
64k
no
no
yes
no
yes
# yes-no questions identified
7,637
23,457
213,018
509,265
2,941
precision (random sample of 200)
1.00
0.89
0.99
0.99
0.98
with indirect answers # with direct answers precision (random sample of 200)
4,528
10,481
135,890
233,533
1,452
3,109
12,976
77,128
275,732
1,489
0.65
0.96
0.93
0.97
0.93
Table 3: Statisties and evaluation of the rules used for distant supervision. The rules to identify yes-no questions almost always correct. The ratio of indirect answers varies across languages, but is always high (lowest: 44.7% in Korean; highest: 64.8% in Chinese). The rules to interpret direct answers are also very precise (0.93-0.97) except Hindi (0.65), most likely because work with Hindi tweets. We use the interpretations of direct answers obtained via distant supervision in these five languages to automatically interpret indirect answers in eight languages.
tions from the aforementioned corpora in each lan- guage. The Hindi rules are follows. A tweet contains yes-no question it:
Analysis The result of the rules for distant super- vision consists of yes-no questions, direct answers, and (noisy) keyword- interpretations. We pri- oritize precision over recall, as we need as little noise as possible for training purposes. We estimate quality with sample of 200 instances per language (Table 3). The rules to identify yes-no questions are almost perfect across four out of five languages (Precision: 0.98-1.00). We identify thousands of yes-no questions in all the languages, and many of those 335.2%665.3% are followed by direct answer that we can interpret with our rules. The ratio of yes and no interpretations varies across lan- guages, and the precision of the rules to interpret direct answers is high (0.93-0.97) in all languages except Hindi (0.65), We believe that (a) the ratios of yes and no depend on the domain of the source corpora and (b) the low precision in Hindi is duc to the fact that work with Twitter opposed to more formal texts. Note that all the question- answer pairs identified via distant supervision are interpreted with yes or no, Regardless of the qual- ity the data, whether it is useful in the training process is an empirical question (Section 5).
contains question mark and any of these bi- grams: का आान (do you), का बमा (do we), शपा पह (will this), शपा काी (does this, does this ever), पह दा सकाा (can this happen); does not contain these words: कहह (where), का (why), कमन (how), कोन (who), किममा (whose), कान (which), या (or), का (when); has between 3 and 100 tokens; and does not (a) contain links, @ mentionns #hash- tags, or numbers or (b) come from unverified users, retweets, or replies tweets.
Identifying Direct Answers The next set of rules identifies which yes-no questions are followed by a direct answer. We have rules that identify direct an- swers and their interpretations based on yes and no keywords. We complement these rules with rules to discard some answers that cannot be reliably interpreted regardless of keywords. The rules for Hindi are follows. A reply tweet yes-no question is a direct answer if it:
contains yes keywords: दी (Yes), हा (yes), सो (yes), डी (yes), सहर (sure), सही (correct), निरिर हक (definitely), yes, yeah, sure, of course, 100%; or no keywords: नही (No), नही (no), मन (don't), न (not), no, never, n't (We include few English keywords since people code-switch between these languages); does not contain links, #tags, or more than one @ @mention (i.e., only replies the user who asked the yes-no question);
Benchmarks in New Languages
We are the first to work on interpreting answers to yes-no questions in languages other than English (Hindi, Korean, Chinese, Bangla, Turkish, Span- ish, Nepali, and Persian). We set to work with questions and answers written in the languages of interest to avoid translationese (Koppel and Or- dan, 2011), so we create new benchmarks with 300 question-answer pairs per dataset for each lan- guage (Chinese and Turkish: 600 samples; other languages: 300 samples). For the five languages
does not contain question marks; and has between 6 to 30 tokens.
Appendix B details the rules for other languages.

---

[hi] Q: बाा दमा लि भोका शान के लिपर शी बहा हy ? (Are we born only to be cheated?)
[tr] Q: aşksız mutlu olabilir misiniz?
(Can you be happy without love?) her şekilde mutlu olmasım bilirim.
A: अभीनपपपा शम उन şा मं बरा पप T (Unfortunately we are born in that cra.) Interpretation: Yes
(I know how to be happy anyway) Interpretation: Yes
[es] Q: TTe ha seguido molestundo él?
[ko] Q: サ z 2 A 2 F5 : ل 2
(Has he continued bother you?)
(1 think have a cold. Can I have medicine for headache relief?)
cada rato.
--5 9f + zy 2 2. E| X f L/
(All the time.)
(I'll just give you cold medicine rather than a headache reliever.)
Interpretation: Yes
Interpretation: No
[zh] Q: 4 "" S
[ne] Q: पम निररमाता A
(Any water bottle in car?)
(Do you believe in secularism?)
A: A A
A: मंीााानकी शपसशा ड सपरे मा-परड
(Let's just use a can.)
(It is provision of the constitution. So should be accepted.)
Interpretation: No
Interpretation: Middle
[bn] Q: शााा भारास कके शाड पिाा (मका
[fa] Q: fددs نمير مي0 ofoon y4 زز
पा शार रिाा (Can you please check if I have a fever?)
(You are looking from a high elevation, is the weather clean?)
A: Sidd pimnse gedigled دارم اللي
A: श कमा
(I now clearly looking at the Damavand (Mountain).)
(It's less now.)
Interpretation: Yes
Interpretation: Yes
Table Examples from the benchmarks we create in eight languages. Answers whose interpretations lean towards yes or are annotated as such; middle is used for 50/50 splits and unresponsive answers.
for which there are large unannotated corpora and we have built rules for (Section 3), we select yes-no question-answer pairs without a direct answer (i.e., those identified by our rules as yes-no questions but not followed by a direct answer). We collect 300 question-answer pairs for the other languages from Bangla2B+ (Bhattacharjee et al., 2022, Bangla), Kantipur Daily (Nepali). 3 and LDC2019T11 (Mo- hammadi, 2019, Persian).
request. In the Spanish (es) example, the answer affirms the inquiry without using yes or similar key- words. In the Nepali (ne) example, the answer is interpreted as middle since it mentions laws rather than discussing an opinion. In the Chinese (zh) ex- ample, the answer suggests using a can, implying that they do not have any water bottles in the car. In the Bangla (bn) example, the answer confirms that the questioner has a fever (despite it is lower). Inter-A nnotator Agreements Native speakers in each language performed the annotations. We were able to recruit at least two annotators for five lan- guages (zh, hi, es, tr, and bn) and one annotator for the rest (ne, ko, fa). Inter-annotator agreements (linearly weighted Cohen's 's k) are as follows: Turk- ish: 0.87, Hindi: 0.82, Spanish: 0.76, Bangla: 0.73,
Annotation Guidelines We manually annotate the interpretations of indirect answers to yes-no ques- tions in the eight languages using three labels:
Yes: the answer leans towards yes, including probably yes, yes under some conditions, and strong affirmative answers (e.g., Absolutely!).. No: the answer leans towards no.
Middle: Yes or No interpretations do not apply. Our interpretations are a coarser version than those used in previous work. Appendix A presents a mapping, and Appendix C details the guidelines.
and Chinese: 0.67. These coefficients are consid- ered substantial (0.6-0.8) or (nearly) perfect (>0.8) (Artstein and Poesio. 2008). We refer the reader to Appendix D for a detailed Data Statement.
Table 4 presents examples. In the Hindi (hi) example, the yes interpretation relies on the neg- ative sentiment of the answer. In the Turkish (tr) example, the yes interpretation requires common- sense knowledge about fasting, which includes not drinking water. Similarly, the Persian (fa) example requires commonsense (seeing a mountain implics clear weather). In the Korean (ko) example, the an- swer provides an alternative, thereby rejecting the
Dataset Statistics and Label Frequency Table 5 presents the statistics of our training datasets and benchmarks in the new languages. Since we obtain the training datasets via distant supervision (Ta- ble 3), they only include yes-no questions with direct answers and thus their interpretations are limited to be yes or no. For the benchmarks, we se- lect instances from each language randomly. This allows us to work with the real distribution of inter- pretations instead of artificially making it uniform.
Phttps://ekantippur..

---

Hindi
Korean
Chinese
Bangla
Turkish
Spanish
Nepali
Persian
Training instances
# instances
3,109
12,976
77,128
n/a
275,732
1,489
n/a
n/a
yes
27.0
92.0
57.7
n/a
94.4
60.1
n/a
n/a
% no
73.0
8.0
42.3
n/a
5.6
39.1
n/a
n/a
Benchmarks (Test instances)
# instances
300
300
600
300
600
300
300
300
% yes
44.4
52.7
38.5
34.3
43.7
54.2
46.7
46.7
no
43.3
27.3
30.5
32.0
32.7
12.0
36.3
32.0
% middle
12.7
20.0
31.0
33.7
23.6
32.8
17.0
21.3
Table 5: Number of instances and label frequency in the datasets obtained via distant supervision (top, five languages), benchmarks new languages (bottom, eight languages). interpretations are almost always the most frequent in both training datasets and benchmarks. Middle is the least frequent in five languages in benchmarks.
Hindi
Korean
Chinese
Bangla
Turkish
Spanish
Nepali
Persian
Baselines
Majority
0.27
0.37
0.21
0.18
0.27
0.34
0.30
0.29
Random
0.35
0.34
0.34
0.31
0.33
0.36
0.37
0.34
XLM-RoBERTa trained with
Circa
0.42
0.58
0.34
0.42
0.54
0.45
0.51
0.55
SWDA-IA
0.29
0.56
0.34
0.23
0.47
0.36
0.51
0.47
Circa SWDA-IA
0.33
0.60
0.36
0.45
0.55
0.43
0.53
0.55
Table Results (FI) obtained with XI.M-RoBERT,, multilingual transformer, trained with Einlissh corpora (i.e., gross-lingual in the new language). Appendix E presents additional results (P and R). Training with Circa and SWDA-IA more beneficial most cases. Training with SWDA-IA alone, however, shows limited improvements.
As shown in Table 5 (bottom), yes is always the most frequent label (34.3%-54.2%)) but the label distributions vary across languages.
trained and validated exclusively in English (with Circa and SWDA-IA). Monolingual and multilin- gual fine-tuning use the validation split in each language for hyperparameter tuning. The test set is always the same. We will discuss results with XL.MRRoEETT,, as outperforms XLM-Align by a small margin. Appendix F and G detail the results with XLM-Align and the hyperparameters.
5 Experiments
We develop multilingual models to reveal the un- derlying interpretations indirect answers to yes- no questions. We follow three settings: (a) cross- lingual learning to the new language (after training with English, Section 5.1); (b) monolingual fine- tuning via distant supervision with the language of interest (Section 5.2); and (c) multilingual fine- tuning via distant supervision with many languages (Section 5.3). Following previous work (Section 2), we use the question and answer as inputs.
5.1 Cross-Lingual Learning from English
We start the experiments with the simplest setting: training with existing English corpora (Circa and SWDA-IA) and evaluating with our benchmarks. We consider this cross-lingust learning base- line since the model was neither fine-tuned nor validated with the new languages.
We conduct our experiments with two multilin- gual transformers: XIBM-RoBERTa (Conneau et al., 2020) and XLM-Align (Chi et al., 2021). Both obtained from HuggingFace (Wolf al., 2020) and were pretrained hundreds of languages. We split our benchmarks into validation (20%) and test (80%), and report results with the test split. lingull learning was conducted with a model
Table 6 presents the results. Training with Circa is always better than (or equal to) SWDA-IA, but combining both yields the best results with most languages. The only exception is Hindi, where training with Circa obtains better results. This is intuitive considering that Circa much larger dataset than SWDA-IA (Table 2). The lower re-

---

Yes
No
Middle
All
P
R
F
P
R
F
P
R
F
P
R
F
%Apen
Hindi
0.47
0.79
0.59
0.49
0.22
0.31
0.25
0.13
0.17
0.45
0.46
0,43
2
Korean
0.70
0.84
0.76
0.61
0.76
0.68
0.75
0.07
0.12
0.69
0.67
0.62
3
Chinese
0.47
0.81
0.59
0.53
0.56
0.54
0.92
0.08
0.14
0.63
0.50
0.43
19
Turkish
0.63
0.87
0.73
0.72
0.57
0.63
0.46
0.26
0.33
0.62
0.63
0.60
9
Spanish 0.56
0.66
0.60
0.21
0.48
0.29
0.71
0.20
0.31
0.57
0.47
0.46
2
Table Results obtained multiple languages with MI.--RBBERT,, a multilingual transformer, trained blending with English corpora (best combination from Table 6) and the instances obtained with distant supervision in the language we evaluate with. %AFe" indicates the improvement compared to training with English only (Table 6).
Training with the new language is always beneficial and only requires defining a handful of rules.
sults with SWDA-IA for Hindi are likely due to the mismatch of domains (SWDA-IA: phone conversa- tion transcripts, Hindi: Twitter).
glish and the language of interest. In other words, we answer the following questions: Does multilin- gual fine-tuning yield better results?
We use the same blending strategy as in mono- lingual fine-tuning (Section 5.2) but adopt greedy approach find the best combination of additional languages train with. We start with the model trained with English and the data for the language of interest if available (Table 7). Otherwise, we start with the model trained with English (Table 6). Then, we add data from an additional language (one at a time) and select the best based on the results with the validation split. We continue this process until including additional languages does not yield higher results with the validation split,
5.2
Tine-Tunigg with the New Language
We continue the experiments by training models with English corpora (gold annotations, Circa and SWDA-IA) and the data obtained via distant su- pervision for the language of interest (noisy data, Section 3.2 and Table 5). We adopt the fine-tuning methodology by Shnarch al. (2018) to blend training data from English corpora and the addi- tional instances obtained via distant supervision. Briefly, we start the training process (first epoch) with the concatenation of the English data and data for the new language. Then, we reduce the data for the new language by ratio a after each epoch. We choose the English corpora based on the best combination from Table 6 for each language. We found that reducing the data for the new language (as opposed the English data) yields better re- sults. We believe this is because the former: (a) is noisier (distant supervision vs. human annotations) and (b) docs not include any middle interpretations.
Table presents the results. The technique is suc- cessful. Compared monolingual fine-tuning (Ta- ble 7), multilingual fine-tuning always beneficial (%AF = 2-28). More importantly, the improve- ments with respect to cross-lingual learning (Table 6) are even higher across all languages (%A prm = 2-53), For five out of eight languages, the improve- ments at least 11 % and statistically significant (McNemar's test, p 0.05). Note that multilingual fine-tuning obtains significantly better results with two of the languages for which we could not find source corpora to use distant supervision: Nepali (17%) and Bangla (22%). We hypothesize that the low gains with Persian might be due to the fact that it is the only one using right-to-left script.
Table 7 presents the results. Recall that addi- tional data is only available in five languages. The results show that blended training with English and the new language always beneficial. The improvements (%A Fea) range from 2% to 19%. They are the largest with the languages for which we have the most additional data (Table 3): Turkish (9%) and Chinese (19%). Note that despite the ad- ditional Hindi data is noisy (0.67 Precision, Table 3), observe improvements (2%).
Which languages are worth training with? Some of the language combinations that are worth blend- ing with surprising (Table 8). For example, Chinese (zh) is beneficial for Hindi (hi), Turkish (tr) and Spanish (es). This may be due the fact that Chinese is one of the languages for which we collect the most data via distant supervision. Simi- larly, Turkish (tr) is useful for Korean (ko).
5.3 Fine- Fine-Tuning with Several I Languages
We close the experiments exploring whether it is beneficial to train with languages other than En-

---

Yes
No
Middle
All
Blending w/
P
R
F
P
R
F
P
R
F
P
R
F
%APRR
%AF
Hindi
hi, zh, es
0.53
0.66
0.59
0.51
0.50
0.51
0.25
0.03
0.06
0.49
0.52
0.49
16
Korean
ko, tr
0.68 0.88
0.77
0.65
0.67
0.66
0.83
0.11
0.20
0.70
0.68
0.63
5
2
Chinese zh, es, ko
0.56 0.69 0.63
0.53
0.68
0.59
0.69
0.31
0.43
0.59
0.56
0.55
53*
28"
Bangla hi
0.53 0.64 0.58
0.47 0.60
0.52
0.76
0.42
0.54
0.59
0.55
0.55
22"
n/a
Turkish tr, zh
0.65 0.83 0.73 0.69 0.69
0.69
0.41
0.18
0.25
0.61
0.64
0.61
11"
2
Spanish es, zh, hi,
0.55 0.71 0.62 0.26 0.32 0.29
0.58
0.29
0.38
0.52
0.51
0.49
Nepali tr, hi
0.69 0.77 0.73 0.62
0.69
0.65
0.41
0.18
0.25
0.62
0.64 0.62
17
n/a
Persian hi
0.60 0.80 0.69 0.53 0.58 0.55 0.73 0.19 0.31 0.61 0.59 0.56
2
n/a
Table Results obtained with (best combination from Table 5.1) and the instances obtained with distant supervision in other languages (Column 2). We show only the best combination languages. %APe and % indicate the improvements compared to training with English only (best in Table 6) and blending English and the language evaluate with (Table 7). An asterisk indicates that the improvements are statistically significant (McNemar's test (McNemar, 1947), p < 0.05).
multilingual transformer, trained blending with English corpora
0.52
0.58
0.67
similar a language the more likely it is to be useful multilingual fine-tuning. There are only few exceptions (e.g., Korean with Hindi and Turkish).
68 tr zh
ko
0.42
0.64 0.67
es
fr zh hi
ko
5.4 Examples of Errors
0.51 0154 0,58
0,67
Table presents questions and answers for which our best model fails identify the correct inter- pretation. In the Chinese (zh) example, the no interpretation could be obtained by contrasting ur- ban area and airport. However, the model lacks such commonsense knowledge. Similarly, in the Bangla (bn) example, a wife being afraid of dogs most likely indicates that there are dogs in the household. In the Korean (ko) example, the answer provides both yes and for different days of the weekend, but the model appears consider only Sunday. the Spanish (es) example, the question has a negation thus the answer ought to inter- preted as yes. Answers in both the Turkish (tr) and Hindi (hi) examples include a polar distractor that confuses the model (tr: no longer yet the au- thor has favorite game; hi: stop yet the answer ought to be interpreted as nobody laxes. The answer Nepali (ne) indicates it could (or could not) happen while the model only takes as affirmation. The answer in Persian (fa) is a rhetorical question with negation, thus the correct interpretation is yes.
tr
hi
kn
zh
0.34 0.39 0.42
0.55
0.64
zh
ko
bn
zh bi
0.41
0.52
tr ko
hizh
es
0.37
0.5
0.59 0.61
0.69
Cs
zh
ko tr
hi
ne
0.4 0.44
0.5 0.54
zh ko hi
tr
fa
Figure 1: Language similarities between the eight lan- guages evaluate with (boldfaced) and each of the languages we use distant supervision with (hi, ko, zh, ur, and es). Underlining indicates that language worth blending with (Table 8). The more similar the lan- guages, the more likely to useful, although there are exceptions (e.g., Korean and either Turkish Hindi).
stop paying
In order to analyze these surprising results, we sort all language pairs by their similarity (Figure 1). We define language similarity the cosine similarity between their lang2vec vectors (Littell et al., 2017), more specifically, we use syntactic and language family features from the URIEL ty- pological database. Generally speaking, the more
5.5 A Note on Large language Models
Large Language Models (LLMs) obtain impressive results in many tasks (Mishra et al., 2022). Re- searchers have also shown that LLMs can solve problems even with malicious prompts (Webson and Paylick, 2022), casting a shadow on what they

---

Instance
Prediction
Gold
[zh] Q: W (Do you live near the airport?)
A: : (I live in the urban area, Xiyuan Hotel.)
yes
[oo] Q: E 1 (Do you open weekends too?)
A: _ && J œ|
@ q 82 gage & #7 (On weekends,
no middle
we are open until 6pm on Saturday and closed on Sunday.)
[es] Q: y no se lo ha operado todavía? (And you haven't had surgery yet?) A: bueno cuando yo he estado Esta segunda vez (Well I've been here for the 2nd time.)
no yes
[tr] Q: en sevdiginiz bilgisayar veya mobil oyun var m²? (Do you have a favorite computer mobile game?)
A: oyunlarna bagumluhğum kalmadı, lakin 3 senedir arada bir oynadığum dead by daylight var. (I am no longer addicted to PC games, but there is Dead by Daylight, which have been playing occasionally for years.)
no yes
[ne] Q: मराा शपपरद, सीो HET पह काानन सलिं पि भ सबी क f ? (Is there discussion that after going to the government, the case can be dropped?) A: सिो ह मिदे (TTat could be.)
yes
middle
[fa] Q: 9 ميكنه كارو إين كسي (Does anyone do this?) A: 9 تسيكن (Don't they?)
yes
[hi] Q: का हम दकसा दना शद कर दे (Should we stop paying taxes?)
A: माा बिना मही बद कर रो, पर दभमा तो दना दी पडपा (Even if you stop breathing,
yes
you will still have to pay taxes.)
[bn] Q: भाशनना रासि शरर भाद, भान (Sir, do you have dog in your house?)
A: माता MIm करश सा भासा (My wife is afraid of dogs.)
yes
nO
Table 9: Examples of error interpretations to instances from our benchmarks, made by best model (Table 8).
may understand. LLMs do not outperform our best model (Table 8). First, we note that open-source LLMs are "pretrained only on Latin or Cyrillic scripts" (Touvron et al., 2023), thus they cannot process most of the languages we work with. Sec- ond, closed-source LLMs such ChatGPT are likely to have been pretrained with the data in our benchmarks, so data leakage is an issue. We ran- domly selected 30 instances for each language from our benchmarks and fed them to ChatGPT via the online interface using the prompt in Appendix H. The best model for each language (Table 8) outper- forms ChatGPT by 16.4% on average with these instances (F1: 54.25 vs. 63.13). Regardless of pos- sible data leakage, we acknowledge that ChatGPT obtains impressive results in a zero-shot setting.
33.7% depending on the language (Table 3).
We have created new evaluation benchmarks in eight languages: Hindi, Korean, Chinese, Bangla, Turkish, Spanish, Nepali, and Persian. Addition- ally, we present a distant supervision approach identify yes-no questions and direct answers. Train- ing with this new, noisy, language-speciff, data is always beneficial and significantly better in five languages. Cross-lingual learning from many lan- guages to new language is the best strategy. This is true even for the three languages for which dis- tant supervision was not explored because of diffi culties finding large unannotated corpora (Bangla, Nepali, and Persian). Our approach successfully learns interpret indirect answers from direct an- swers obtained via distant supervision.
6 Conclusions
Our future work includes exploring more robust multilingual fine-tuning (Zheng et al., 2021). We also plan to explore applications of the fundamen- tal research presented here. In particular, we are interested in improving dialogue consistency and avoiding inconsistencies by ensuring that generated turns are compatible with the correct interpretation of indirect answers to yes-no questions.
We have tackled for the first time the problem of interpreting indirect answers to yes-no questions in languages other than English. These kinds of answers do not include yes or no keywords. Their interpretations, however, are often either Yes or No. Indeed, Middle accounts for between 12.7% and

---

Limitations
results with the test set. The results are taken from the output of one run. We acknowledge that the average of multiple runs (e.g.. 10) would be more reliable, but they also require much more computa- tional resources (literally, 10 more times).
We double-annotate the benchmarks in five out of the eight languages. We not able do so in the other three languages (Bangla, Nepali, and Persian) since we are unable to recruit a second native speaker. While great care was taken, includ- ing single annotators double and triple checking their work, we acknowledge that not reporting inter- annotator agreements in these three languages is weakness.
Ethical Considerations
Data sources and collection. Our benchmarks use source texts in accordance with their licenses. We collect corpora in Spanish and Persian from Lin- guistic Data Consortium" under a non-commercial license. Samples in Turkish from MFAQ are used under the Apache- 2.0 license. Samples in Chinese from NaturalConv and LCCC are used under a non- commercial license and the MIT license respec- tively. Samples in Hindi from Twitter are used un- der the Twitter developer policy. Samples in Ko- rean from Al Hub are used under a non-commercial license. Samples in Bangla from Bangla2B+ are used under the CC BY-NC-SA 4.0 license. Sam- ples in Nepali from Kantipur Daily are used under non-commercial license.
We adopt three labels (Yes, No and Middle) to represent the interpretations of answers to yes-no questions. Some previous works use finer-grained label sets as we discussed in Section 3.1. Consider- ing that (a) there is no universal agreement about the possible ways to interpret answers to yes-no questions and (b) interpretations of answers other than the three we adopted are rare (less than 10% in Circa and Friends-QIA QIA, and less than 20% in SWDA-IA), we argue that three labels are sound.
We run the experiments with two models: XLM- RoBERTa and Xlgg-lign. We acknowledge that there are other transformers such as InfoXLM and XLM-E. However, they either obtain similar (or worse) results on existing benchmarks, or not open-source at the time of writing. Regarding open- source Large Language Models (LLMs) such as LLaMa and Alpaca, we do not run experiments on them since (a) fine-tuning with even the smallest version (parameters: 7B) requires significant com- puting resources; (b) they are pretrained only on Latin or Cyrillic scripts thus may have limitations targeting other languages.
Acknowledgments
We would like to thank Siyu Liu and Xiao Liu for their help in annotating the Chinese benchmark, and the Chamelcon platform (Keahey al., 2020) for providing computational resources. We also thank the reviewers for their insightful comments.
This material is based upon work supported by the National Science Foundation under Grant No. 1845757. Any opinions, findings, and conclu- sions or recommendations expressed in this mate- rial are those of the authors and do not necessarily reflect the views of the NSF
Our in-context learning experiments are con- ducted with ChatGPT based on GPT-3.5 (text- davinci-002), which was the state-of-the-art model at the time we conducted the experiments. GPT- 4 is already out, and better language models will continue to come out. We acknowledge that testing with GPT-4 may obtain better results, but argue doing so not necessary: (a) querying on GPT-4 based ChatGPT interface is limited to a few sam- ples per hour, and (b) OpenAl has never announced the list of supported languages so we would be shooting in the dark. A more serious issue is the fact that GPT-X may have been trained with the data we work with, any results are to be taken with grain of salt.
References
Mikel Artetxe, Vedanuj Goswami, Shruti Bhosale, An- gela Fan, and Luke Zettlemoyer. 2023. Revisiting machine translation for cross lingual classification. arXiv preprint arXiv:2305.1422
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of mono- lingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623-4633 Online. Association for Computational Linguistics.
We tune several hyperparameters (including the blending factor a, listed in Table 12 and Table 13) with the train and development splits, and report
'https://www. 1dc. upenn. upenn.edu/ 'nttps://develoon wtwpprrttwww... tw tter.ccmm com/en/ wnwprrreeeeeeten--pllaoa- and-policy

---

Ron Artstein and Massimo Poesio. 2008, Survey article: Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4):555-596
In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages 2174-2184, Brussels, Belgium. Association for Com- putational Linguistics.
Emily M. Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Compitational Linguistics,
Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova, 2019. BoolQ: Exploring the surprising difficulty of natural yes/uo questions. In Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 2924-2936, Minneapolis, Min- nesota. Association for Computational Linguistics.
Abhik Bhattacharjee, Tahmid Hasan, Wasi Ahmad, Kazi Samin Mubasshir, Md Saiful Islam, Anindya Iqbal, M. Sohel Rahman, and Rifat Shahriyar. 2022. BanglaBERT: Language model pretraining and benchmarks for low-resource language under- standing evaluation in Bangla. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 1318-1327, Seattle, United States. Asso- ciation for Computational Linguistics.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross-ling lingual representation learning at scale. In Pro- ceedings of the 58th Anmual Meeting of the Asso- ciation for Computational Linguistics, pages 8440- 8451, Online. Association for Computational Lin- guistics.
Penelope Brown and Stephen Levinson. 1978, Uni- versals in language usage: Politeness phenomena. In Questions and politenexs: Strategies in social inter- action, pages 66111. Cambridge University Press.
Alexandra Canavan and George Zipperlen. 1996. CALLFRIEND Spanish-Non-Caribbee Dialect.
Cathrine Damgaard, Paulina Toborek, Trine Eriksen, and Barbara Plank. 2021. "I'll be there for you": The one with understanding indirect answers. In Pro- ceedings of the 2nd Workshop on Computational Ap- proaches to Discourse, pages 1-11, Punta Cana, Do- minican Republic and Online. Association for Com- putational Linguistics.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Vosss, Katherine Lee, Adam Roberts, Tom B Brown, Dawn Song, Ul- far Erlingsson, et al. 2021. Extracting training data from large language models. In USENIX Security Symposium, volume
Casimiro Pio Carrino, Marta R. Costa-jussà, and José A. R. Fonollosa. 2020. Automatic Spanish trans- lation of SQuAD dataset for multi-lingual question answering. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 55 15 5523, Marseille, France. European Language Re- sources Association.
Maxime De Bruyn, Ehsan Lotfi, Jeska Buhmann, and Walter Daelemans. 2021. MFAQ: a multilingual FAQ dataset. Proceedings of the 3rd Workshop on Ma- chine Reading for Question Answering, pages 1-13, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186 Minneapolis, Minnesota. Association for Computational Linguistics.
Zewen Chi, Li Dong, Bo Zheng, Shaohan Huang, Xian- Ling Mao, Heyan Huang, and Furu Wei. 2021. Im- proving pretrained cross-lingual language models via self-labeled word alignment. In Proceedings of the 59th Annual Meeting of the Association for Compu- lational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Vol- ume Long Papers). pages 1418-3430, Online. As- sociation for Computational Linguistics.
Nancy Green and Sandra Carberry. 1999. Interpreting and generating indirect answers. Computational Lin- guistics, 25(3):389-435
Zewen Chi, Shaohan Huang, Li Dong, Shuming Ma, Bo Zheng, Saksham Singhal, Payal Bajaj, Xia Song, Xian-Ling Mao, Heyan Huang, and Furu Wei. 2022. XLMMEE: Cross-lingual language model pre-training via ELECTRA. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics ( (Volume I: Long Papers), pages 61 6170-1
Julia Bell Hirschberg. 1985. A theory of scalar impli- cature (natural languages. pragmaties, inference). Ph.D. thesis, University of Pennsylvania.
Dublin, Ireland. Association for Computational Lin- guistics.
Beth Ann Hockey, Deborah Rossen- Knill,, Beverly Spe- jewski, Matthew Stone, and Stephen Isard. 1997. Can you predict responses to yes/no questions? yes, no, and stuff. In Fifth european conference on speech communication and technology. Citeseer.
Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen- tau Yih, Yejin Choi, Percy Liang, and Luke Zettle- moyer. 2018. QuAC: Question answering in context.

---

Shafiq Joty, Preslav Nakov, Lluis Marquez, and Israa Jaradat. 2017. Cross-languass learning with adver- sarial neural networks. In Proceedings of the s Conference on Computational Natural Language Learning (CoNLL 2017). pages 226-237. Vancouver, Canada. Association for Computational Linguistics.
Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 321 421--3252, Dublin, Ireland. Association for Computational Linguistics.
Katharina Kann, Kyunghyun Cho, and Samuel R. Bow- man. 2019. Towards realistic practices in low- resource natural language processing: The develop- ment set. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-JJCNLP) pages 3342-3349,, Hong Kong, China. Association for Com- putational Linguistics.
Patrick Littell, David R. Mortensen, Ke Lin, Katherine Kairis, Carlisle Turner, and Lori Levin. 2017. URIEL and lang2vee: Representing languages as typological, geographical, and phylogenetic vectors. In Proceed- ings of the 15th Conference of the European Chap- ter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 8-14, Valencia, Spain. Association for Computational Linguisties.
Xiao Liu, Yifan Zhou, Shuhei Ikemoto, and Heni Ben Amor. 2023. a-mdf: An attention- based multimodal differentiable filter for robot state estimation. In 7th Annual Conference on Robot Learning.
Kate Keahey, Jason Anderson, Zhuo Zhen, Pierre Riteau, Paul Ruth, Dan Stanzione, Mert Cevik, Ja- cob Colleran. Haryadi S. Gunawi, Cody Hammock, Joe Mambretti, Alexander Barnes, François Hal- bach, Alex Rocha, and Joe Stubbs. 2020. Lessons learned from the chameleon testbed. Proceedings of the 2020 USENIX Annual Technical Conference (USENIX ATC 20). USENIX Association.
Annie Louis, Dan Roth, and Filip Radlinski. 2020. "I'd rather just go to bed": Understanding indirect an- swers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 741 1-425,, Online. Association for Computational Linguisties.
Moshe Koppel and Noam Ordan. 2011. Translationese and its dialects. In Proceedings of the 49th An- nual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 1318- 1326, Portland, Oregon, USA. Association for Computational Linguistics.
Quinn McNemar. 1947. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika. 12(2):153-157.
Swaroop Mishra, Daniel Khashabi, Chitta Baral. Yejin Choi, and Hannaneh Hajishirzi. 2022. Reframing instructional prompts to GPTk's s language. In Find- ings the Association for Computational Linguistics: ACI. 2022. pages 589-612, Dublin, Ireland. Associa- tion for Computational Linguistics.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin. Jacob Devlin, Ken- ton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- Wei Chang. Andrew M. Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov, 2019. Natu- ral questions: A benchmark for question answering research. Transactions of the Association for Compu- tational Linguistics, 7:452-466.
Volodymyr Mnih, Nicolas Heess, Alex Graves, et al. 2014. Recurrent models of visual attention. Ad- vances in neural information processing systems, 27.
Yash Kumar Lal, Niket Tandon, Tanvi Aggarwal, Ho- race Liu, Nathanael Chambers, Raymond Mooney, and Niranjan Balasubramanian. 2022. Using com- monsense knowledge to answer why-questions. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1204-1219, Abu Dhabi, United Arab Emirates. Asso- ciation for Computational Linguistics.
Ariana Negar Mohammadi. 2019. Corpus of Conversa- tional Persian Transcripts.
Alvaro Morales, Varot Premtoon, Cordelia Avery, Sue Felshin, and Boris Katz. 2016. Learning answer questions from Wikipedia infoboxes. In Proceedings of the 2016 Conference on Empirical Methods in Nat- ural Language Processing, pages 1930-1933 1935, Austin, Texas, Association for Computational Linguistics.
Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2020. MLQA: Evalu- ating cross-lingual extractive question answering. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 7315- 7330, Online. Association for Computational Lin- guistics.
Yixin Nie, Mary Williamson, Mohit Bansal, Douwe Kiela, and Jason Weston. 2021. I like fish, espe- cially dolphins: Addressing contradictions in dia- logue modeling. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1699-1713, Online, Association for Computational Linguistics.
Weizhao Li. Junsheng Kong, Ben Liao, and Yi Cai. 2022. Mitigating contradictions in dialogue based on contrastive learning. In Findings of the Associa- tionfor Computational Linguistics: ACL 2022, pages 2781-2788, Dublin, Ireland. Association for Compu- tational Linguistics.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.

---

2022. Training language models follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744
Elior Sulem, Jamaal Hay, and Dan Roth. 2022. Yes, no or IDK: The challenge of unanswerable yes/no questions. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, pages 1075-1085, Seattle, United States. Association for Computational Linguisties.
Pranav Rajpurkar. Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natu-
ral Language Processing, pages 233-2222, Austin, Texas. Association for Computational Linguisties.
Niket Tandon, Bhavana Dalvi, Keisuke Sakaguchi, Pe- ter Clark, and Antoine Bosselut. 2019. WIQA: A dataset for "what if..." reasoning over procedural text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNLP) pages 6076- 6085, Hong Kong, China. Association for Computa- tional I. Linguisties.
Siva Reddy. Dangi Chen, and Christopher D. Manning. 2019. CoQA: A conversational question answering challenge. Transactions of the Association for Com- putational Linguistics, 7:249-266.
Arij Riabi, Thomas Scialom, Rachel Keraron, Benoit Sagot, Djamé Seddah, and Jacopo Staiano. 2021. Synthetic data augmentation for zero-shot cross- lingual question answering. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7016-7036 Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Sandra A Thompson. 1986. Questions and responses in english conversation by anna-brita stenström. Lan- guage, 62(1):113-214
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302. 13971.
Sebastian Ruder and Avi Sil. 2021. Multi-domain mul- tilingual question answering. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: Tutorial Abstracts, pages 17- 21, Punta Cana, Dominican Republic & Online. As- sociation for Computational Linguistics.
Xiaoyang Wang. Chen Li, Jianqiao Zhao, and Dong Yu. 2021. Naturalconv: A chinese dialogue dataset to- wards multi-tur tulti-turmmnn topic-driven conversation. Proceed- ings of the AAAI Conference on Artificial Intelligence, 35( 1400-
Krishna Sanagavarapu, Jathin Singaraju, Anusha Kak- ileti, Anirudh Kaza, Aaron Mathews, Helen Li, Nathan Brito, and Eduardo Blanco. 2022. Disentan- gling indirect answers to yes- no questions in real con- versations. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 4677-4695, Seattle, United States. Association for Computational Linguisties.
Yida Wang, Pei Ke, Yinhe Zheng, Kaili Huang, Yong Jiang, Xiaoyan Zhu, and Minlie Huang. 2020. A large-scale chinese short-text conversation dataset. In Natural Language Processing and Chinese Com- puting: 9th CCF International Conference, NLPCC 2020, Zhengzhou, China, October 14-18, 2020, Pro- ceedings, Part I, page 91-103, Berlin, Heidelberg. Springer-Verlag
Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, et al. 2022. Language models are multilingual chain-of th-tht reasoners. arXiv preprint 10.002220.
Albert Webson and Ellie Pavlick. 2022. Do prompt- based models really understand the meaning of their prompts? In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 2300-2344, Seattle, United States. Association for Computational Linguistics.
Eyal Shnarch, Carlos Alzate, Lena Dankin, Mar- tin Gleize. Yufang Hou, Leshem Choshen, Ranit Aharonov, and Noam Slonim. 2018 Will it blend? blending weak and strong labeled data in a neural network for argumentation mining. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Voluee 2: Short Papers). pages 599 605, Melbourne, Australia. Association for Computational I Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shieifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of the-ahr art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 45, Online, Association for Computational Linguistics.
Andreas Stolcke, Klaus Ries, Noah Coccaro, Eliza- beth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van Ess-Dykema, and Marie Meteer. 2000. Dialogue act modeling for au- tomatic tagging and recognition of conversational speech. Computational Linguistics, 26(3)3339-77.

---

Linjuan Wu, Shaojuan Wu, Xiaowang Zhang, Deyi Xiong. Shizhan Chen, Zhiqiang Zhuang, and Zhiy- ong Feng. 2022. Learning disentangled semantic representations for zero-shot cross-lingual transfer in multilingual machine reading comprehension. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics (Volume I: Long Papers), pages 991-1000, Dublin, Ireland. As- sociation for Computational Linguistics.
Turkish For the Turkish corpus (MRQA), we define the following rules to identify yes-no ques- tions:
The conversation turn contains any of the following keywords: myym, miyim, muyum, mitüüm, musm, misin, musum, müsin, m. mi, mu, mii, myyz, miyiz, muyuz, müyüžz musini, misiniz, musunuz, müsünüzz. All of these words come from the same root (mi, m, mu, mü)) which is used to make sentence yes- no question. There is no direct translation into English.
Bo Zheng, Li Dong, Shaohan Huang, Wenhui Wang. Zewen Chi, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, and Furu Wei. 2021. Consistency reg- ularization for cross-lingual fine-tuning. In Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joins Conference on Natural Language Processing (Volume 1: Long Papers), pages 3403 3417, Online. Association for Computational Linguistics.
The turn does not contain wh-questions key- words: ne (what), nerede (where), ne zaman (when), nasil (how), nasil (informal how), ne- den (why), kim (who). hangi (which), and kimin (whose).
A Mapping Heterogeneous Interpretations to Yes, No, and Middle
The turn is less than 50 tokens.
The rules to identify direct answers in Turkish rely on yes (evet (yes), evt (yes, informal), eet (yes, informal), tabii (of course), tabi (of course, infor- mal), tabiiki (of course, informal), tabiki (of course, informal), aynen (absolutely), and huht (yes, infor- mal)) and no (hayar (no), hayir (informal no), hyr (informal no), and yoo (informal no)) keywords. Regardless of keywords, we discard question- answer pairs if the is_accepted field is set to false, which indicates the answer not selected.
The three English corpora (Circa, SWDA-IA, Friends-QIA) use compatible but different interpre- tations for yes-no questions (Section 3.1). Here we list the mapping to our three labels (Yes, No, and Middle). This mapping is straightforward given their definitions.
Circa (relaxed labels):
Yes + Yes
No > No
Yes, subject to some conditions + Yes
In the middle, neither yes nor no - Middle
Other: discard
Spanish For the Spanish corpus (CallFriend), we define the following rules to identify yes-no ques- tions. A conversation turn includes a yes-no ques- tion if:
N/A: discard
SWDA-IA:
Yes - Yes
Probably Yes + Yes
it contains a verb;
Middle + Middle
ends with a question mark ("?"); and does not contain the following words or phrases: por que (why), cuando (when), donde (where), como (how), cuanto (how much/many), quien (who), cual (which, sin- gular), or cuales (which, plural).
Probably No > No
No > No
Friends-QIA:
Yes + Yes
No >> No
Yes, subject to some conditions + Yes
The rules to identify direct answers in Spanish are defined as follows: The answer turn (i.e., the turn after the question turn) contains yes keywords (si (yes), claro (sure), correct (correcto), vale (ok). por supuesto (sure), quizas (maybe), de acuerdo (understood), asi es (that's right)) or no keywords (no (no), nah nope, no se (I don't 't know), no lo se (I don't 't know), no estoy seguro (1 am not sure), ni idea (no idea)).
Neither yes nor no - Middle
Other: discard
N/A: discard
B Rules to Identify Yes-No Question and Direct Answers in Multiple languages
Hindi In addition to the rules listed in Section 3.2, we remind the reader that Hindi data was collected before Twitter changed its policies on verified ac- counts and API usage.
In Spanish, we consider both the accented (and proper) spelling and the unaccented ones).

---

Chinese For the Chinese corpora (NaturalConv and LCCC-base), we define the following rule to identify yes-no questions: conversation turn ends with a modal particle me or Uif, followed by a question mark (?? ').0
end in the following modal particle words or phrases that indicate statements instead of questions, as in 11 I e t| be 7 # क - (I was calling if can file a civic complaint about reckless driving). The full list of particles is as follows: 2 72 T. , 전. 2 S. AER. # t 2 L-2 77 2o A, R 7 $ ) 7/7 J| WF ||
We also define rules identify direct answers in the turn following yes-no questions. Our rules are defined as follows. The answer turn (i.e., the turn after the question turn) starts with:
yes keywords (&t (right), X (okay), Rg (uh- huh), .8. (uh-huh), 말 & (of course), L
(must)) or no keywords: (7s (no), and it (ab- sence)): : or
The rules to identify direct answers in Korean
rely on keywords:
the first verb in the question, either in the affir- mative or negative form. For example, in the yes-no question = DA HA?? (Can I sit here?), the first verb is oi E. (can). This rule matches turns that start with the verb øj E (can) its negated form R EX (cannot).
yes keywords: H| (yes), all (yeah), J (right), and hoo (correct);
no keywords: of şt (ah no). of At (yeah, no), + (unfortunately), o) o] (ney), and o) 1 (no)
C Annotation Guidelines for the Benchmarks
Korean For the Korean corpus, we define the following rules to identify yes-no questions. Recall that the corpus contains questions and answers, so the rules are designed differentiate between yes-no questions and other questions rather than identifying yes-no questions in any corpus. A turn contains a yes-no question if it does not:
We conduct manual annotations to obtain ground truth interpretations (i.e., gold labels) for indirect answers. We work with three labels: Yes, No, and Middle (Unknown), In order to minimize inconsis- tencies, we define labels as follows:
Yes: The answer leans towards (or implies) yes or yes under certain conditions or constraints. The latter could be interpreted as probably yes (c.g., Q: Ever done this before? A: Once.). No: The answer leans towards (or implies) no, no under certain conditions or constraints (probably no), or provides arguments for no. The last two could be interpreted as probably no (e.g., Q: Can I at least have drink? A: It's ten thirty in the morning.)
contain the following keywords indicating wh- questions: o] " 7 (how), b / (what), $. /} (what is), of ua (which is). f (what), #| of & (what), b oil Ω (what), @ प (how much), 5 (how much available), 5-777-4.2 (how much do you need), म Al (how many), (how big), A (how big), 전 t (how many times), @l (when), c] (where), of 전 (where to), (which one), 전 ll & (how much time does it take), म E (how many minutes), y 2 (how many minutes), of uy l (how), में A] (what time), शी क य ol (what VIN number), H] 8 &. R. (how much cost), o) (where), m (how much), o) प
(which one), of प >
Middle (Unknown): The answer is unrespon- sive (e.g., changes the topic) or uninformative (c.g.. "I don't 't know"). It should imply or lean towards neither yes nor no. (e.g.. Q: Can you connect me to someone else? A: Well what's the situation?)
H.# (which part), 2f e (why not), = क (which direction), of m
B (which item), ^] 7| (until what time), x 7| (by what time), म. e (what date), M A] (what time). मे o (what date), d] to-o OF (what should know), #| L & (what do need), E ci] (how many cars), 말 hf (how many cars), 2 & (what are the digits); and
D Data Statement
As recommended by Bender and Friedman (2018), we provide data statement to better understand the new data presented in this paper.
Curation Rationale
We develop new datasets to help interpret indirect answers to yes-no questions in eight languages.
This is Chinese, not the English question mark ('?')

---

First, we adopt a rule-based approach to collect yes- no questions, direct answers, and interpretations of the answers in five languages. The interpretations were obtained by automatically mapping the posi- tive keyword to Yes, and the negative keyword to No. This data is noisy (see quality estimation in Ta- ble 3; Precision = 0.93-0.97 except in Hindi (0.67)) and intended to be used for distant supervision. Second, we collect yes-no questions with indi- rect answers and manually annotate their interpre- tations in the eight languages we work with. For the five languages we used distant supervision with, we collect questions followed by answers not iden- tified by our rules (i.e., without polar keywords). For the other three languages, we collect questions and indirect answers from scratch.
of Turkish, and one of Korean. All of them are highly proficient in English. Ethnic backgrounds are as follows: four individuals are from East Asia, four are from South Asia, one is from the Mid- dle East, one is from Europe, one is from North America, and one is from South America. Socioe- conomic backgrounds are as follows: all annotators reported that they are middle class.
Educational backgrounds are as follows. We have two undergraduate, nine graduate students, and one with a doctoral degree. Nine of them work in NLP-rreated research areas, two work in other research areas within Computer Science, and one has Mathematics background.
Speech Situation
Spanish and Chinese corpora come with context (i.e., the turn before the question and after the an- swer) since they were obtained from conversations. The corpora include only the questions and answers in the other languages.
Chinese corpora are from two sources: (a) written by crowdsourcing workers who are given specific topic, and (b) written by users on social media plat- forms. The Spanish corpus consists of transcripts of telephone conversations between humans. The Nepali corpus is written by journalists from a daily news website. The Korean corpus is written by civilians and covers complaints and replies from public organizations. The Hindi corpus is written by Twitter users. The Turkish corpus is written by humans and consists of frequently asked ques- tions and community questions. The Bangla corpus comes from a crawled dataset from online sources. The Persian corpus is transcriptions of spoken lan- guage by humans having informal conversations, including telephone calls and face-to-face interac- tions.
We list all the original data sources for our data in Section 3.
Language Variety
We work with eight languages to develop our datasets. The language list is as follows: Bangla spoken in Bangladesh (bn-BD), Chinese as spo- ken in Mminllnd China and written with simplified characters (zh-CN), Korean as spoken South Ko- rea KR), Turkish as spoken in Turkey (Ir-TR), Spanish as spoken in United States, Canada, Puerto Rico, or The Dominican Republic (es-US, es-CA, es-PR, es-DO), Persian as spoken in Tran (fa-IR), and Nepali spoken in Nepal (ne-NP). We not able to provide language variety information for Hindi since the data comes from Twitter.
E Detailed Results for Cross-Lingual Transferring to a New Language
Speaker Demographic
We provide supplemental results in Table 10 that further include Precision (P), Recall (R) and F1- score (F) for our cross-lingual learning experiments. This table complements Table 6.
Our corpora are collected from various sources. We are not able to provide their speaker demographic since such information is absent in the original sources. However, all the corpora are spoken (or written) by native speakers.
F Experimental Results with XLM-Align
Annotator Demographic
We present the experimental results obtained with XLM-Align in Table 11. The results are listed in three blocks and each block comparable Table 6. Table 7 and Table respectively. We observe a similar trend in the results with both models, while ULM-RoBERTE outperforms XLM-Alige in most
We recruit 12 annotators consisting of two women and ten men. Their ages range from 18 to 30 years old. Among them, three individuals are native speakers of Chinese, two of Spanish, two of Bangla, one of Persian (and proficient in Turkish), one of Nepali (and proficient in Hindi), one of Hindi, one
cases.

---

Hindi
Korean
Chinese
Bangla
P
R
F
R
F
P
R
F
P
R
F
Baselines
Majority
0.20
0.45
0.27
0.28
0.53
0.37
0.15
0.38
0.21
0.13
0.35
0.18
Random
0.38
0.33
0.35
0.39
0.33
0.34
0.34
0.34
0.34
0.31
0.31
0.31
XIMMRRBERTa trained with
Circa
0.51
0.50
0.42
0.61
0.64
0.58
0.35
0.43
0.34
0.51
0.47
0.42
SWDA-IA
0.49
0.45
0.29
0.54
0.64
0.56
0.30
0.44
0.34
0.22
0.34
0.23
Circa SWDA-IA
0.40
0.44
0.33
0.67
0.65
0.60
0.45
0.44
0.36
0.55
0.47
0.45
Turkish
Spanish
Nepali
Persian
P
R
F
P
R
F
P
R
F
P
R
F
Baselines
Majority
0.19
0.44
0.27
0.25
0.51
0.34
0.21
0.47
0.30
0.21
0.46
0.29
Random
0.19
0.44
0.27
0.40
0.34
0.36
0.39
0.36
0.37
0.38
0.33
0.34
XIMMRRBERTa trained with
Cirea
0.61
0.54
0.59
0.51
0.45
0.51
0.53
0.51
0.59
0.57
0.55
SWDA-IA
0.46
0.56
0.47
0.30
0.48
0.36
0.52
0.59
0.51
0.57
0.53
0.47
Circa + SWDA-IA
0.59
0.61
0.55
0.51
0.49
0.43
0.56
0.57
0.53
0.58
0.57
0.55
Table 10: Detailed Results obtained multiple languages with XLM-RoBERTa, multilingual transformer, trained with English corpora. This table complements Table 6 by including precision and recall values.
Hindi
Korean
Chinese
Bangla
Turkish
Spanish
Nepali
Persian
XLM-Aliga trained with
Cirea
0.40
0.57
0.43
0.48
0.54
0.43
0.56
0,50
SWDA-IA
0.27
0.37
0.21
0.17
0.27
0.34
0.42
0.42
Circa + SWDA-IA
0.37
0.54
0.42
0,49
0.51
0.46
0.56
0.51
XLM-Align trained with English corpora + one new language (Table 7)
0.37
0.57
0.52
n/a
0.61
0.43
XLM-Aliga trained with English corpora language combination (Table 8)
0.40
0.58
0.54
0.51
0.56
0.53
0.60
0.50
Table II: Results (F) obtained with XIMM-Algg, trained with three strategies (Section 5). For the third strategy (the third block), adopt the same language(s) train with XLMMRoBERTa blend with. Comparing XLM-RoBERTa (Table 6, Table 7 and Table 8), most results are worse. However, both models obtain similar trend.
G Training Details
imately it takes 10 minutes train epoch.
Referring to Section 5, we conduct our experiments on an off-the-shelf XLMM-RBBRRTa-base (parame- ters: 279M) and XLM-Align- model (parame-
We tune with several hyperparameters to obtain the experimental results with X base. We list them in Table 12. We also tune the blending factor a (i.e., the ratio of data from distant supervi- sion added each training epoch). We list the a each benchmark in Table 13. For experiments with XLM-Align --iii---nne,, base, we follow the same hyper- parameters and blending factors.
ters: 279M) from HuggingFace (Wolf et al., 2020). Both models multilingual transformers with at- tention mechanism which has been utilized in vari- ous domains (Liu et al., 2023; Mnih et al., 2014). We run the experiments on single NVIDIA Tesla V100 (32GB) GPU. Depending on the size of train- ing datasets, the training time may vary, but approx-

---

Table
Table 7
Table 8
Maximum epochs
10
30
30
Warmup steps
0.01 training steps
0.01 training steps
0.01 training steps
Batch size
32
16
16
Optimizer
AdamW
A
Adam W
Learning rate
2e-5
2e-5
2e-5
Weight decay
le-2
le-2
le-2
Gradient clipping
1.0
1.0
1.0
Table
Tuned hyperparameters our experiments with WL.M-RBBERTa-bas model.
Hindi
Korean
Chinese
Bangla
Turkish
Spanish
Nepali
Persian
Blending factor a
0.2
0.5
0.2
0.2
0.2
0.2
0.2
0.2
Table 13: Tuned blending factor for our experiments with XLM-RoBERTa-base model.
I need you to help me understand the underlying meanings of indirect answers to yes-no questions. You can choose from Yes, No, or Middle. You should reply to me with Yes, No or Middle based on your interpretation of the answer.
Question: "<Question from
benchmarks>"
Answer: "<Answer from benchmarks>"
Does the answer mean Yes, No or Middle?
Figure 2: Prompt used with ChatGPT
H Details of In-context Learning with ChatGPT
We test 30 random samples for each language with ChatGPT (lext-davinci-002) and our best model, explained in Section 5.5. Figure 2 presents the prompt. In order give ChatGPT the most eredit possible, we manually map the generated answers from ChatGPT into Yes, No, and Middle,


Reviews:
[{"review_id": "GFqDugR7ag", "reviewer": "Reviewer_DJgs", "summary": "", "questions": "A. I may have overlooked this detail, but could the authors elaborate on the segment where indirect answers are identified during the distant supervision phase? While the rules for direct answers and yes-no questions are provided, the methodology for pinpointing non-direct answers to those questions remains unclear. Could you please clarify this aspect?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful review. We truly appreciate your comments.\n\n**Questions for The Authors**\n- **RE: I may have overlooked this detail, but could the authors elaborate on the segment where indirect answers are identified during the distant supervision phase?**\n   - We develop rules to identify yes-no questions with direct answers (l. 255). Answers that are not identified as direct are considered candidate indirect answers. Note that the annotation process to create the benchmarks in each language includes a validation step to check whether the candidate indirect answers are actually indirect. This validation step takes place before annotators assign the interpretation to the indirect answers. Based on this question, we will add these details in the extra page if accepted.\n\n**Typos Grammar Style and Presentations Improvements**\n- **RE: The \"Related Work\" section appears overly broad.**\n    - Thank you for the suggestion, we agree after reading it again. We will include multilingual QA (outside yes-no questions) as suggested, including the following papers:\n       - Learning Disentangled Semantic Representations for Zero-Shot Cross-Lingual Transfer in Multilingual Machine Reading Comprehension (Wu et al., ACL 2022)\n       - Synthetic Data Augmentation for Zero-Shot Cross-Lingual Question Answering (Riabi et al., EMNLP 2021)\n       - Automatic Spanish Translation of SQuAD Dataset for Multi-lingual Question Answering (Carrino et al., LREC 2020)\n   - Regarding the generic QA references, we will rephrase to emphasize the difference in the tasks (finding answers or interpreting the answers).\n- **RE: I encountered certain phrases that seemed awkward.**\n   - Thank you for the suggested improvements. We will update the paper and recruit a native speaker to find more improvements.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper's primary contribution is the dataset, introducing a novel expertise level to multilingual QA datasets. The authors have evidently undertaken comprehensive work to devise potential rules for the distant supervision phase across multiple languages. This process offers replicability potential for other languages not addressed in the paper, promising value for the broader multilingual QA community.\n\nIn my view, the authors' approach to collecting unannotated corpora for each language for the distant supervision phase is commendable. It serves as an exemplary method for constructing intricate datasets and tasks using available data. Well done.", "reasons_to_reject": "The manuscript could benefit from further refinement in terms of writing quality to meet the standard expectations of a scientific publication. Some statements appear to be general and would be strengthened with additional citations. There are instances where the wording could be more formalized or where unconventional terminology is used. A subsequent round of proofreading and editing might help enhance the presentation of the content.\n\n", "questions_to_authors": "A. I may have overlooked this detail, but could the authors elaborate on the segment where indirect answers are identified during the distant supervision phase? While the rules for direct answers and yes-no questions are provided, the methodology for pinpointing non-direct answers to those questions remains unclear. Could you please clarify this aspect?", "paper_topic_and_main_contributions": "The paper focuses on indirect answers, a specific type of response to yes-no questions. The authors argue that these types of answers are more commonly observed in human language. The study primarily addresses non-English indirect answers, which have been less explored in previous research. The authors curate new datasets of indirect questions using distant supervision based on rules provided by native speakers to identify yes-no questions with indirect answers. Additionally, they introduce a novel annotated benchmark, annotated by native speakers for Hindi, Korean, Chinese, Bangla, Turkish, Spanish, Nepali, and Persian. Finally, the authors conduct several experiments with multilingual pretrained transformers to assess the task in a supervised setup using their new datasets and benchmark.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The \"Related Work\" section appears overly broad. Specifically, the paragraph on question-answering seems to merely cite general publications about QA without any clear relevance to your specific work. How does this enhance the reader's understanding? Similarly, the portion discussing multilingual pretraining appears generic. It would be beneficial to the audience to delve deeper into indirect answers and to provide more information about multilingual benchmarks for QA, especially those curated similarly to your approach.\n\nI encountered certain phrases that seemed awkward, including \"We build multilingual models\" and \"computational efforts to date are limited to working with English\". For enhanced clarity and formality, consider revising these to sound more precise. This would bolster the reader's confidence in your work. For instance, you might say, \"We have developed multilingual models,\" or \"Previous efforts have predominantly focused on English.\""}}, {"review_id": "N5MvgNcYRs", "reviewer": "Reviewer_jtqq", "summary": "", "questions": "See above.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful review. We truly appreciate your comments.\n\n**Reasons to Reject**\n- **RE: I'm a bit confused about the experiments and final results. [...] I'm not sure how this data is combined in experiments. Is it combined across all splits, or the test set only includes annotated data? If distant supervision data is included in test set, then the answers are direct and easier to classify.**\n    - As explained in l. 194, the instances obtained with distant supervision are only used for training. The test set only includes the instances from benchmarks that were annotated manually (l. 377). There is no overlap between the instances in the training set (i.e., obtained with distant supervision) and the test set; the selection criteria (l. 308) guarantees so. We will add further explanations to make this more clear. \n- **RE: Multiple experiments were done, but there is a lack of analysis and explanation of the results. Scores are not very high which is ok, but a subsection explaining what were some of the weaknesses/strengths of the model, with some examples, would have made the paper stronger IMO.**\n   - Thank you for the suggestion. We will include the predictions of our models for each language in the public repository and include examples of errors for each language in the appendices.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Great yes-no Q resource covering multiple languages.\n- Most of the data was automatically collected, but there were annotation efforts on 300 samples per 8 languages. \n- Many experimental results are reported comparing the 0-shot setting with finetuning on one language and finetuning on multiple languages.\n- Interesting analyses on including which languages in training data can help get better results on another language.", "reasons_to_reject": "- I'm a bit confused about the experiments and final results. There are two sets of data, one which is extracted based on some rules (is noisy and questions have direct answers), and one set is annotated. I'm not sure how this data is combined in experiments. Is it combined across all splits, or the test set only includes annotated data? If distant supervision data is included in test set, then the answers are direct and easier to classify. IMO, the results should have been broken down into annotated samples, and distant supervision data. The main goal is interpreting \"indirect Answers\", and I can't tell how successful the approach and different training setups are based on current reported results. If the test set only included annotated data, please make that more clear in the writeup.\n- Multiple experiments were done, but there is a lack of analysis and explanation of the results. Scores are not very high which is ok, but a subsection explaining what were some of the weaknesses/strengths of the model, with some examples, would have made the paper stronger IMO.", "questions_to_authors": "See above.", "paper_topic_and_main_contributions": "This paper presents a yes-no questions resource with their interpretation (yes, no, middle), over 8 languages: Hindi, Korean, Chinese, Bangla, Turkish, Spanish, Nepali, and Persian. The majority of the samples in the dataset were collected based on applying rules on corpora in that language, but there were also annotation efforts on 300 samples per 8 languages.\n\nXLM-RoBERTa is trained on multiple different settings (English 0-shot, train on one language, train on a combination of multiple languages) and results are reported and compared. The authors also show which languages are most helpful if used during training for another language.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fdEtSfmcPc", "reviewer": "Reviewer_KQFS", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful review. We truly appreciate your comments.\n\n**Reasons To Reject**\n- **RE: The experimental results are somewhat expected, as the addition of more training data from other languages typically leads to improved results.**\n  - The statement is inaccurate according to our results (and thus perhaps the results are not somewhat expected). We observe that adding additional languages (i.e., more data) does not necessarily improve the results (Section 5.3, Table 8). We stop the greedy strategy when adding a new language is no longer beneficial. Our hypothesis is that such improvements are more likely correlated to language similarity (see l. 469 - 486).\n\n- **RE: In general, the task is relatively simple.**\n  - The task is simple to formulate, but it is not simple to solve. Supervised models obtained modest results (Tables 5-8, most F1-scores are below 0.60), and ChatGPT does worse (Section 5.4).\n\n - **RE: The settings where the models were trained on (Circa + SWDA-IA) could be better described as a cross-lingual setting as well instead of \"zero-shot\" since they were already exposed to the task of interpreting answers to yes/no questions.**\n   - Thank you for the rewording suggestion. We will change the word accordingly in the next version of the paper. “crosslingual” is indeed the correct term.\n\n- **RE: Table 3 appears a bit confusing; it might be more beneficial to have separate tables for the final statistics of the benchmark and the evaluation of the rules to enhance clarity and readability.**\n   - Thank you for the suggestion. We will have a new block to differentiate the statistics and evaluation of the rules, or have two tables if there is enough space once we can use an extra page.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The benchmark covers multiple low-resource languages, broadening the evaluation scope to include diverse linguistic contexts.\n\n* Native speakers were involved in writing the rules. \n\n* Some of the shown examples require a form of knowledge or commonsense reasoning for a correct interpretation, making it more challenging.", "reasons_to_reject": "* The experimental results are somewhat expected, as the addition of more training data from other languages typically leads to improved results.\n\n* In general, the task is relatively simple.\n\n* The settings where the models were trained on (Circa + SWDA-IA) could be better described as a cross-lingual setting as well instead of  \"zero-shot\" since they were already exposed to the task of interpreting answers to yes/no questions. \n\n* Table 3 appears a bit confusing; it might be more beneficial to have separate tables for the final statistics of the benchmark and the evaluation of the rules to enhance clarity and readability.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper introduces a new multilingual benchmark consisting of eight languages for interpreting indirect answers to yes/no questions. Among them, five languages have both training and evaluation data, while the remaining three languages are provided with evaluation data only. The data collection was achieved using distant supervision, primarily relying on lexical matching as the initial set of rules. The authors conducted experiments with various settings, including training with only English data or training with multiple languages while evaluating on unseen language. The findings indicate that combined training with multiple languages yields better results compared to using a single language for training. Additionally, training with direct answers improves the model's ability to interpret indirect answers.  ", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors introduce a new dataset for interpreting indirect yes/no questions. A training set was constructed using distant supervision based on rules covering in 8 languages, and small test sets of human-annotated examples were created for 3 languages. They also run experiments to provide some baseline results using the data.

The reviewers are in agreement that the data collection seems reasonable and useful for the task, but they did not find it overall exciting beyond that.
Decision: Accept-Findings