Interpreting Answers to Yes-No Questions in User-Generated Content
Authors: Shivam Mathur
Keun Hee Park
Dhivya Chinnappa
Saketh Kotamraju
Eduardo Blanco
Venue: EMNLP 2023 Findings (2023)
Keywords: yes-no questions
question answering
TL;DR: Understanding what answers to yes-no type questions mean in user-generated content

Abstract: Interpreting answers to yes-no questions in social media is difficult. Yes and no keywords are uncommon, and the few answers that include them are rarely to be interpreted what the keywords suggest. In this paper, we present a new corpus of 4,442 yes-no question-answer pairs from Twitter. We discuss linguistic characteristics of answers whose interpretation is yes or no, as well as answers whose interpretation is unknown. We show that large language models are far from solving this problem, even after fine-tuning and blending other corpora for the same problem but outside social media.

Interpreting Answers to Yes-No Questions in User-Generated Content
Shivam Mathur," Keun Hee Park, 2 Dhivya Chinnappa,"
Saketh Kotamraju, and Eduardo Blanco
Walmart Global Tech, 2 PArizona State University, JJP Morgan Chase Co.,
*University of Texas at Austin, SUniversity of Arizona shi www.mmattttwwaaaaaamcmamaamaaaae com, kpark53@asu w k@ut edu,
chivya.
eduardoblaacco@rrize
Abstract
Do preheat the oven before you bang something in? My girifriend doesn't and I've always found very strange Simply give it an extra 3 minutes cook Fan
Interpreting answers to yes-no questions in social media difficult. Yes and no keywords uncommon, and the few answers that include them are rarely to be interpreted what keywords suggest. present corpus of 4,442 yes-no question-answer pairs from Twitter. We discuss linguistic characteristics of answers whose interpretation is yes answers whose interpretation is wnknown. We show that large language models far from solving this problem, even after fine-tuning and blending other corpora for the same problem but outside social media.
decent like that anyway
want to answer Yes and No.
this paper,
Yes-No question from Twitter and three an- swers. Interpreting answers cannot be reduced finding keywords (e.g., yes, of course). Answers ought to be in- terpret as yes (despite the negation), no (commonsense and reasoning needed), and unknown respectively.
Figure 1:
as well as
Consider the Twitter thread in Figure 1. The question is mundane: whether people preheat the oven before using it. The answers, however, are complicated decipher. The first answer states that very strange that somebody else (the au- thor's girlfriend) does not preheat the oven. Note that the correct interpretations is yes (or probably yes) despite the negation. The second answer ex- plains why it is unnecessary to preheat the oven- without explicitly saying so-thus the correct inter- pretation is no. The third answer includes both yes and yet the correct interpretation is unknown, the author does not commit to either option.
1 Introduction
Social media has become a global town hall where people discuss whatever they care about in real time. Despite its challenges and potential mis- use, the language of social media can be used approach many problems, including mental health (Harrigian et al., 2020), rumor detection (Ma and Gao, 2020), and fake news (Mehta et al., 2022). There is extensive literature on question answer- ing, but few previous efforts work with social me- dia. In particular, yes-no questions are unexplored in social media. Figuring out the correct interpre- tation of answers yes-no questions (yes, no, or somewhere in the middle) from formal texts (Clark et al., 2019), short texts (Louis et al., 2020), and conversations (Choi et al., 2018; Reddy et al., 2019) is challenging. This is the case in any domain, but especially in social media, where informal lan- guage and irony are common. Additionally, unlike previous work on yes-no questions outside social media, we find that yes and keywords are only rare in answers, but also that they are poor in- dicators of the correct interpretation of the answer.
In this paper, we tackle the problem of interpret- ing natural answers to yes-no questions posted on Twitter by real users, as exemplified above. The main contributions are: (a) Twitter-YN new cor- pus of 4,442 yes-no questions and answers from Twitter along with their interpretations; (b) anal- ysis discussing the language used in the answers, and in particular, showing that keyword presence in answers is a bad predictor of their interpretations, (c) experimental results showing that the problem is challenging-eee cluding GPT3 blending with related corpora useful; and (d) qualitative analysis of the most common errors made by our best model.
for large language models, in-
'Twitter-YN and implementation available at http::/ggitbb.com/hhhaammmmtttt
Wook done while at Arizona State University.

---

Motivation Traditionally, question answering as- sumes that there are (a) correct answers ques- tions or (b) no answers (Kwiatkowski et al., 2019). In the work presented in this paper, answers are available-the problem is to interpret what they mean. As Figure 1 shows, answers capture personal preferences rather than correct answers supported by commonsense or common practice (e.g., oven manuals state that preheating is recommended). In- terpreting answers to yes-no questions opens the door to multiple applications. For example, dia- logue systems for social media (Sheng et al., 2021) must understand answers to yes-no question to avoid inconsistencies (e.g., praising the author of the second reply in Figure 1 for preheating the oven is nonsensical). Aggregating interpretations could also assist in user polling (Lampos et al., 2013). As the popularity of chat-based customer support grows (Cui et al., 2017), automatic interpretation of answers to yes-no questions could enhance the customer experience. Lastly, interactive question answering and search, in which systems ask clar- ification questions to better address users' infor- mation needs, would benefit from this work (Li et al., 2017). Clarification questions are often yes- no questions, and as shall see, people rarely use yes, no or similar keywords in their answers. Rather, people answer with intricate justifications that are challenging to interpret.
taken. More recent works (de Marneffe et al., 2009) target questions from SWDA (Stolcke et al., 2000).
Clark et al. (2019) present yes-no questions sub- mitted to a search engine, and Louis et al. (2020)
present a corpus of crowdsourced yes-no questions and answers. Several efforts work with yes-no questions in dialogues, a domain in which they are common. Annotation efforts include crowdsourced dialogues (Reddy et al., 2019; Choi et al., 2018) and transcripts of phone conversations (Sanagavarapu et al., 2022) and Friends (Damgaard et al., 2021).
Question Answering in the Social Media Do- main is mostly unexplored. TweetQA is the only corpus available (Xiong et al., 2019). It consists of over 13,000 questions and answers about tweet. Both the questions and answers were written by crowdworkers, and the creation process ensured that it does not include any yes-no question. In this paper, we target yes-no no questions in social media for the first time. We do not tackle the problem of extracting answers to questions. Rather, we tackle the problem of interpreting answers to yes- no questions, where both questions and answers are posted by genuine social media users rather than paid crowdworkers.
Twitter- YN: A Collection of Yes-No Questions and Answers from Twitter
Since previous research has not targeted yes-no questions in social media, our first step to cre- ate a new corpus, Twitter- Retrieving yes-no questions and their answers takes into account the intricacies of Twitter, while the annotation guide- lines are adapted from previous work.
2 Previous Work
Question Answering Outside Social Media has a long tradition. There are several corpora, includ- ing some that require commonsense (Talmor et al.,
2019) or understanding tables (Cheng et al., 2022). simulate open-book exams (Mihaylov et al., 2018), and include questions submitted by "real" users (Yang et al., 2015; Kwiatkowski et al., 2019). Mod- els include those specialized to infer answers from large documents (Liu et al., 2020), data augmen- tation (Khashabi et al., 2020), multilingual mod- els (Yasunaga et al., 2021), and efforts to build multilingual models (Lewis et al., 2020). None of them target social media or yes-no questions.
3.1 Retrieving Yes-No Questions and Answers
We define a battery of rules to identify yes-no ques- tions. These rules were defined after exploring existing corpora with yes-no questions (Section 2), but we adapted them to Twitter. Our desidera- tum is to collect a variety of yes-no questions, so we prioritize precision over recall. This way, the annotation process is centered around interpreting answers rather than selecting yes- nos-no questions from the Twitter fire hose. The process is as follows: 1. Sclect tweets that (a) contain a question mark ('?') and the bigram do you, is it, does it or can it; and (b) do not contain wh-words (where, who, why, how, whom, which, when, and whose). Let us name the text between the bigram and '?" a candidate yes-no question.
Yes-No Questions and interpreting their answers have been studied for decades. Several works target exclusive answers not containing yes or no (Green and Carberry, 1999; Hockey et al., 1997). Rossen- Knill et al. (1997) work with yes-no questions in the context of navigating maps (Carletta et al., 1997) and find that answers correlate with actions

---

Question
Answer
Intpn.
Do you think it is rude to say to somebody you would look better if you lost weight?
Not only rude, but potentially lethal depending on who you say it to!
yes
Is ok to read book at a bar?
Sure, you don't mind getting beaten up.
Ro
Is it big coat weather?
the news is reporting weather then it's got to be cold ey
prob. yes
Is bad if brake with my left foot while driving?
It's how racecar drivers do it
prob.
Do you want Daylight savings time permanent?
Don't care have bigger issues to worry about
unknown
Table 1: Examples of yes-no questions and answers from our corpus. Some answers include negative keywords but their interpretation yes (first example), whereas others include positive keywords but their interpretation is no (second example). Answers imposing conditions are interpreted probubly yes or probably no (third and fourth examples). Negation does not necessarily indicate that the author leans to either yes or no (fifth example).
2. Exclude candidate questions unless they (a) are between and 100 tokens long; (b) do not span more than one paragraph; and (c) do not contain named entities or numbers.
Temporal Spans: Old and New We collect ques- tions and their answers from two temporal spans: older (years 2014-2021) and newer (first half of 2022). Our rationale is to conduct experiments in realistic setting, i.e., training and testing with ques- tions posted in non-overlapping temporal spans.
3. Exclude candidate questions with links, hash- tags, or from unverified accounts.
The retrieval process resulted in many yes-no question-answer pairs. We set to annotate 4,500 pairs. Because we wanted to annotate all an- swers to the selected questions across both tem- poral spans, Twitter- YN consists of 4,442 yes-no question-answer pairs (old: 2,200, new: 2,242). The total number of questions are 1,200 (3.7 an- swers per question on average).
The first step identifies likely yes-no questions. We found that avoiding other questions with wh- words is critical to increase precision. The sec- ond step disregards short and long questions. The reason discard questions with named entities is to minimize subjective opinions and biases dur- ing the annotation process most yes-no questions involving named entities ask for opinions about the entity (usually celebrity). We discard ques- tions with numbers because the majority are about cryptocurrencies or stocks and we want to avoid having tweets about the financial domain (out of a random sample of 100 questions, 68% of ques- tions with numbers were about cryptocurrencies or stocks). The third step increases precision by con- sidering only verified accounts, which are rarely spam. Avoiding hashtags and links allows us to avoid answers whose interpretation may require external information including trending topics.
3.2 Interpreting Answers to Yes-No Questions
After collecting yes-no questions and answers, we manually annotate their interpretations. We work with the five interpretations exemplified in Table Our definitions are adapted from previous work (de Marneffe et al., 2009; Louis et al., 2020; Sana- gavarapu et al., 2022) and summarized as follows: Yes: The answer ought to be interpreted yes without reservations. In the example, the answer doubles down on the comment that it is rude suggest that somebody would look better if they lost weight (despite also using not) by characterizing the comment as lethal. No: The answer ought to be interpreted as no without reservations. In the example, the author uses irony to effectively state that one should not read books in bars.
Once yes-no questions are identified, we con- sider as answers all replies except those that con- tain links, more than one user mention @@ @user), no verb, less than 6 or more than 30 tokens, or ques- tion marks (??". The rationale behind these filters is to avoid answers whose interpretation require external information (links) or are unresponsive. We do not discard answers with yes or no keywords because as we shall see (and perhaps unintuitively) they are not straightforward to interpret (Section 4).
Probably yes: The answer is yes under certain condition(s) (including time constraints) or it shows arguments for yes from third party. the example, the answer relies on the weather report to mean that it will be big coat weather.

---

36,40
1.40
0.50
0.20
1.44
3.24
0.50
0.23
0.05
35
% Contains Keyword
30
-30
sduuiiitoo
1:01
5.07
0.52
0.35
0.50
call
1.41
6:30
0.11
Interpretation
%
positive
negative
0:30
13.35
1.33
2.34
0.57
12.22
0.30
20
yes
40.79
13.42
11.98
15
15
no
30.23
2.23
18.91
probably yes
8.04
1.33
2.43
25.10
0.50
1.24
probably no unknown
3,85
0.33
2.39
onnotator
brnotocco
17.09
0.92
5.70
Figure 2: Differences between adjudicated interpreta- tions and interpretations by each annotator. There few disagreements: the diagonals (i.e., the adjudicated interpretation and the one by each annotator) account for 84.72% and 84.37%. Additionally, most disagreements are minor: they are between (a) yes (no) and probably yes (probably no) or (b) unknown and another label.
Table Distribution of interpretations in our corpus (Columns 1 and 2) and rryword-bbsee analysis (Col- umn and 4), Only 13.42% (18.91%) of answers with yes (no) interpretations include a positive (negative) keyword. Interpreting answers cannot be reduced to checking for positive or negative keywords.
24.67
6.15
2:90
4,00
2.58
Probably no: The answer is a no under certain condition(s) shows arguments for no held by third party, the author shows hesitancy. In the example, the author indicates that is not bad to brake with your left foot while driving if you are a racecar driver.
Keyword-easse -asd---dddddddeeeee-de Rules
3.02
0.40
1.19
50
58.50
-30
3.15
4.78
5.53
9.36
7.58
20
9,77
5.00
11.59
19.08
37.53
-10
Unknown: The answer disregards the ques- tion (c.g., changes topics) or addresses the question without an inclination toward yes or no. In the example, the author states that the question is irrelevant without answering it. Appendix presents the detailed guidelines. Annotation Process and Quality We recruited four graduate students to participate in the defi- nition of the guidelines and conduct the annota- tions. The 4,442 yes-no question- answers pairs were divided into batches of 100. Each batch was annotated by two annotators, and disagreements were resolved by an adjudicator. In order to en- sure quality, we calculated inter-annotator agree- ment using linearly weighted Cohen's k. Overall agreement was バ = 0.68. Note that x coefficients above 0.6 are considered substantial (Artstein and Poesio, 2008) (above 0.8 are considered nearly per- fect). Figure 2 shows the disagrecments between the interpretations by annotators and the adjudica- tor. Few disagreements raise concerns. The per- centage of disagreements between yes and is small: 1.40, 1.44, 1.28, and 0.88. We refer the reader to the Data Statement in Appendix B for further details about Twitter-Ye YN.
Ground Truth
Figure 3: Heatmap comparing the interpretations pre- dicted with keyword-basse rules (Section 4) and the ground truth. Rules are insufficient for this problem.
keyword, and those that do often are not interpreted what the keyword suggests. For example, 11.98% of answers labeled yes contain a negative keyword, while only 13.42% contain a positive keyword. Are Keywords Enough to Interpret Answers? No, they are not. We tried simple keyword-based rules to attempt to predict the interpretations of an- swers. The rules we consider first check for yes (yes, yea, yeah, sure, right, you bet, of course, cer- tainly, definitely, or uh uh), no (no, nope, no way, never, or n n't), and uncertainty (maybe, may, some- times, can, perhaps, or if) keywords. Then they return (a) yes (or no) if we only found keywords for yes (or no), (b) probably yes (or probably no) if we found keywords for uncertainty and yes (or no), (c) unknown otherwise. As Figure 3 shows, such a strategy does not work. Many instances are wrongly predicted unknown because no keyword is present, and keywords also fail to distinguish between no and probably no interpretations.
aunswrs
4 Analyzing Twitter-Yi YN
4.1 Linguistic Analysis
Twitter-Yi YN is an unbalanced dataset: the interpreta- tion of 70% of answers is either yes or no (Table 2). More interestingly, few answers contain a yes or no
Despite all yes-no no questions include one of four bigrams (do you, is it, does it or can it, Section 3. I),

---

Bigram
Verb (base form, top-10)
Linguistic Feature
yes vs. no
unk vs other
do you (66.4%)
think (9.5), have (7.8), believe (5.4),
Tokens
want (3.7), use (2.6), get (2.4), do (2.3),
Cardinal numbers
like (2.1); go (1.8), and feel (1.7)
Verbs
it (30.6%)
have (5.3), go (4.9), drive (3.6), ask (3.3), look (3.0), put (2.5), want (2.3), (2.2),
verb.communication
M
verb.creation
wear (2.0), and eat (1.8)
verb.motion
f
verb.perception
M
does it (2.5%)
make (19.1), have (6.0), hear (5.5), mat- ter (5.5), get (5.1), take (4.7), tear (4.7),
would
count (4.3), work (4.3), and exist (3.0)
must
shall
can
offer (32.0), ask (32.0). under- stand (32.0). reply (2.0), and wait (2.0)
1st person pronoun
(0.5%)
me
M
Table 3: Most common bigrams and verbs in the ques- tions. Most questions include the bigrams you or it (97%). The verb percentages show that the questions are diverse. Indeed, the top-10 most frequent verbs with you only account for 39.3% of questions with do you.
2nd person pronoun he
tM
you
11
Negation cues Negative sentiment Emojis
411
41
and 97% include do you or is it, Twitter- YN has a wide variety of questions. Indeed, the distribution of verbs has a long tail (Table 3). For example, less than 10% of questions containing do you have the verb think, Further, the the top-10 most frequent verbs account for 39.3% of questions starting with do you. Thus, the questions in Twitter-YN YN are diverse from a lexical point of view.
Table 4: Linguistic analysis comparing answers inter- preted as (a) yes and no and (b) unknown and the other labels. Number of arrows indicate the p-value (t-test; one: p<0.05, two: pc0.01, and three: p<0.001). Arrow direction indicates whether higher counts correlate with the first interpretation (yes unknown) the second.
We also conduct analysis to shed light on the language used in answers to yes-no questions (Ta- ble 4). Verbs and pronouns are identified with spaCy (Honnibal and Montani, 2017). We use WordNet lcxical files (Miller, 1995) as verb classes, the negation cue detector by Hossain et al. (2020), and the sentiment lexicon by Crossley et al. (2017).
fly etc.), and perception (see, hear etc.) verbs ap- pear more often with unknown answers. Finally, unknown answers rarely have negation cues.
5 Automatically Interpretation of Answers to Yes-No Questions
We experiment with simple baselines, RoBERTa- based classifiers, and GPT3 (zero- and few-shot).
First, we compare answers interpreted as yes and no. We observe that cardinal numbers are more common in yes rather than no interpretations. Ob- Experimental settings Randomly splitting Twitter- YN into train, validation, and test splits would lead to valid but unrealistic results. Indeed, in the real world model to interpret answers to yes-no questions ought be trained with (a) answers to different questions than the ones used for evaluation, and (b) answers to questions from non-overlgniing temporal spans.
ject pronoun me is also more common with yes interpretations: most of the time it is used to de- scribe personal experiences. Despite the low perfor- mance of the keyword-based rules, negation cues and negative sentiment more frequent with interpretations. Finally, emojis are more frequent in humorous answers meaning yes rather than no. Second, we compare unknown and other interpre- tations (i.e., when the author leans towards neither yes nor no). We observe that longer answers less likely to be unknown, while more verbs indi- cate unknown. All verb classes are indicators of unknown. For example, communication (say, tell, etc.), creation (bake, perform etc.), motion (walk,
In the main paper, we experiment with the most realistic setting: unmatched question and un- matched temporal span (train: 70%, validation: 15%, test: 15%). We refer the reader Appendix C for the other two settings: matched question and matched temporal span and unmatched question and matched temporal span.

---

Baselines We present three baselines. The ma- jority class always chooses the most frequent label in the training split (i.e., yes). The reyword-bused rules are the rules presented in Section 4. We also experiment with a version of the rules which the lexicon of negation cues is replaced with a negation cue detector (Hossain et al., 2020).
Prompt-Derived Knowledge. Liu al., 2022 have shown that integrating knowledge derived from GPT3 in question-answerin models is bene- ficial. We follow a similar approach by prompting GPT3 to generate a disambiguation text given a question-answer pair from Twitter-Yp YN. Then we complement the input to the RoBERTa-based clas- sifier with the disambiguation text. For example, given Q: Do you still trust anything the media says? A: Modern media can make a collective lie to be- come the truth, GPT3 generates the following dis- ambiguation text: Mentions how media can ma- nipulate the truth, they do not trust anything the media says. We refer the reader to Appendix G for the specific prompt and examples of correct and nonsensical disambiguation texts.
5.1 RoBERTa-based Classifiers
We experiment with RoBERTa (Liu al., 2019) as released by TensorFlow (Abadi et al., 2015) Hub. Hyperparameters were tuned using the train and validation splits, and we report results with the test split. We refer readers to Appendix D for details about hyperparameters and the tuning process.
Pretraining and Blending In addition to train- ing with our corpus, Twitter-YN, we also explore combining related corpora. The most straightfor- ward approach is pretraining with other corpora and then fine-tuning with Twitter- YN. Doing so is sound but obtains worse results (Appendix E) than blending, so we will focus here on the latter.
5.2 GPT3: Zero- and Few-Shot
Given the success of of large language models and prompt engineering in many tasks (Brown et al., 2020). we are keen to see whether they are suc- cessful at interpreting answers to yes-no questions from Twitter. We experiment with various prompt- based approaches and GPT3. Our prompts do not exactly follow previous work, but they are inspired by previous work (Liu et al., 2021):
Pretraining could be seen as a two-step finc- tuning process: first with related corpora and then with Twitter-YN YN. On the other hand, blend- ing (Shnarch et al., 2018) combines both during the same training step. Blending starts fine-tuning with the combination of both and decreases the portion of instances from related corpora after each epoch by an ratio. In other words, blending starts tun- ing with many yes-no questions (from the related corpus and Twitter-YY YN) and finishes only with the ones in Twitter- YN.. The a hyperparameter is tuned like any other hyperparameter; see Appendix D.
Zero-Shot. We prefix prompts (Li and Liang, 2021; Lester et al., 2021) since such prompts work well with generative language models such as GPT3 (Liu et al., 2021). As shown by Efrat and Levy, 2020, we also experiment with the same zero- shot prompt including the annotation guidelines. Few-Shot. We experiment with longer versions of the zero-shot prompts that include two examples from the training split per label (10 examples total; two versions: with and without the guidelines).
We experiment with pretraining and blending with the following corpora. All of them include yes-no questions and manual annotations indicating the interpretation of their answers. However, none of them are in the Twitter domain:
We refer the reader to Appendix G for (a) addi- tional details about the prompts and examples and (b) specifics about the GPT3 experimental setup.
BOOLQ (Clark et al., 2019), 16,000 natural yes-no questions submitted to a search engine and (potential) answers from Wikipedia; Circa (Louis et al., 2020), 34,000 yes-no ques- tions and answers falling into 10 predefined scenarios and written by crowdworkers:
Results and Analysis
We present results the unmatched questions and unmatched temporal span setting in Table 5. Ap- pendix C presents the results in the other two set- tings. We will focus the discussion on average Fl scores as Twitter YN is unbalanced (Section 4). Regarding the baselines, using a negation cue detector to identify no keywords rather than our predefined list is detrimental (F1: 0.27 vs. 0.37), especially with yes label. This result leads to the conclusion that many answers with ground truth
SWDA-IA (Sanagavarapu 22,500 yes-no questions from transcriptions of telephone conversations; and FRIENDS-QU QIA (Damgaard et al., 2021), =6,000 yes-no questions from scripts of the popular Friends TV show.
al., 2022),

---

All
yes
по
pyes
pno
unk
Accuracy
P
R
F1
F1
F1
F1
F1
F1
Baseline, Majority Class
0.39
0.15
0.39
0.22
0.56
0.00
0.00
0.00
0.00
Baseline, Keyword-Bar -ooodd Rules with negation cue detector
0.37
0.51
0.37
0.37
0,40
0.48
0.13
0.15
0.34
0.30
0.48
0.30
0.27
0.22
0.44
0.08
0.18
0.33
RoBERTa trained with yes-no questions corpora
BoolQ
0.44
0.29
0.43
0.32
0.58
0.35
0.00
0.00
0.00
Circa
0.56
0.56
0.56
0.53
0.70
0.59
0.32
0.06
0.35
SwDA-IA
0.50
0.53
0.50
0.47
0.67
0.52
0.26
0.02
0.15
Friends-QUI QIA
0.53
0.53
0.53
0.51
0.67
0.55
0.22
0.00
0.44
Twitter-YN (our corpus)
Question only
0.38
0.41
0.38
0.29
0.53
0.28
0.00
0.00
0.02
Answer only
0.53
0.50
0.53
0.48
0.65
0.56
0.19
0.00
0.32
Question and Answer
0.58
0.58
0.58
0.56
0.68
0.61
0.39
0.22
0.40
RoBERTa blending Twitter-YN
BoolQ (a = 0.5)
0.57
0.45
0.57
0.55
0.70
0.60
0.31
0.06
0.44
Cirea (ax = 0.5)
0.60
0.59
0.60
0.58#
0.72
0.63
0.37
0.25
0.44
+disambiguation text (GPT3)
0.62
0.62
0.62
0.61
0.75
0.67
0.35
0.22
0.57
SWDA-IA (a 0.5)
0.56
0.57
0.56
0.55
0.67
0.59
0.36
0.22
0.44
Friends-QUl 0.8)
0.60
0.59
0.60
0.58
0.71
0.63
0.38
0.20
0.45
+disambiguation text (GPT3)
0.56
0.56
0.56
0.56
0.70
0.63
0.36
0.24
0.37
GPT3, Zero-Shot
0.44
0.55
0.44
0.46
0.56
0.51
0.16
0.13
0.43
GPT3, Few-Shot
0.54
0.55
0.54
0.53
0.67
0.55
0.14
0.30
0.49
Table 5: Results in the unmatched question and unmatched temporal span setting. Only training with other corpora obtains modest results, and taking into account the question in addition to the answer beneficial (Twilter-YN, Fl: 0.56 vs. 0.48). Blending Circa and Friends- QTA beneficial (FI: 0.58 vs. 0.56). Adding disambiguation text from GPT3 with Circa is also beneficial (F1: 0.61). Few-Shot prompt with GPT3 obtains results close to a supervised model trained with Twitter YN (FI: 0.53 vs. 0.56), but underperforms blending Circa using the disambiguation text from GPT3 (0.61). Statistical significance with respect to training with Twitter-Yh YN and is indicated with and with respect to few-shot GPT3 with 'f" (McNemar's Test (McNemar, 1947), p<0.05).
interpretation yes include negation cues such as affixes im- and -less, which are identified by the cue detector but are not in our list. Zero- and Few- Shot GPT3 outperforms the baselines, and obtains moderately lower results than the supervised model trained on Twitter- YN (Question and Answer; F1: 0.53 0.56). Including the annotation guidelines in the zero and few-shot prompts obtains worse results (Appendix G, Table 18).
It is observed that Friends- QIA and Circa are the only corpora worth blending with Twitter-Yn Both obtain better results than training with only Twitter- YN (F1: 0.58 both vs. 0.56). Finally, including in the input the disambiguation texts au- tomatically generated by GPT3 further improves the results (F1: 0.61) blending with Circa. On the other hand, disambiguation texts detrimental when blending with Friends-QIA. These results were surprising to us as disambiguation texts are often nonsensical. Appendix G (Table 17) provides several examples of disambiguation texts generated by GPT3, including sensical and nonsensical texts.
The supervised RoBERTa-based systems trained with other corpora yield mixed results. Circa, SWDA-IA, and Friends-QIA are the ones which outperform the keyword-baute based baseline (0.53, 0.47 and 0.51 vs. 0.37). We hypothesize that this is due to the fact that BoolQ includes yes-no questions from formal texts (Wikipedia). Unsurprisingly, training with Twitter-YN yields the best results. The question alone is insufficient (F1: 0.29), but combining it with the answer yields better results than the answer alone (0.56 vs. 0.48).
Regarding statistical significance, it is important to note that blending Circa and using disambigua- tion texts yields statistically significantly better sults than few-shot GPT3, GPT3, however, by no means useless: provides a solid few-shot base- line and the disambiguation texts can be used to strengthen the supervised models.

---

Error Type
Question
Answer
P.G
Yes distractor (4.4%)
y.n
Do you own bandsaw?
Its on my things to acqure list sure
No distractor (28.6%)
n.y
you like scary movies?
don'tt scary movies scary movies
Social media lexicon (20.7%)
Is foolish to assume that people are always rational?
The keyword is within #assume
Humor (13.2%)
if you heven't fishad a fruit fly out your coffee KEPT
you coffee?
fruit fly to provide protein.
pn,py
Condition or contrast (29.1%)
you have calling life?
Well I should hope so, otherwise there'd be birds all my spartment.
u.py
External entities (14.9%)
much road construction here because they built this rock and roil?
it possible
You must be downtown on the corner of 3rd and Jefferson Starship
u.y
Unresponsive (11,4%)
y,u
you want "old things" away?
You'll always inspire your example than words.
Table 6: Most frequent errors made by the best performing system in the unmatched question and unmatched temporal span setting (Table 5). Error types may overlap, and and G stand for prediction and ground truth respectively. In addition yes and no distractors, social media lexicon (emojis, abbreviation, etc.), humor (including irony and sarcasm), presence of (a) conditions or contrasts or (b) external entities are common errors.
6.1 Error Analysis
7 Conclusions
Table 6 illustrates the most common errors made by our best performing model (RoBERTa blending Twitter-YN (with disambiguation text generated by GPT3) and Circa, Table 5). note that error types may overlap. For example, wrongly predicted instance may include humor and a no distractor).
We have presented Twitter-YN YN, the first corpus of yes-no questions and answers from social media an- notated with their interpretations. Importantly, both questions and answer were posted by real users rather than written by crowdworkers on demand. As opposed to traditional question answering, the problem is not to find answers questions, but rather interprer answers yes-no questions.
Yes and no keywords are not good indicators of answer interpretations (Section 4), and they act as distractors. In the first and second examples, the swers include sure and n'I, but their interpretations are no and yes respectively. Indeed, we observe them in 33% of errors, especially no distractors (28.6%). Many errors (20.7%) contain social media lexicon items such as emojis and hashtags, making their interpretation more difficult. Humor includ- ing irony is also challenge (13.2% of errors). In the example, which also includes contrast, the answer implies that the author likes coffee as the fruit fly has a positive contribution.
Our analysis shows that answers to yes-no ques- tions vary widely and they rarely include a yes or no keyword. Even if they do, the underlying inter- pretation is rarely what one may think the keyword suggests. Answers to yes-no questions usually are long explanations without stating the underlying interpretation (yes, probably yes, unknown, proba- bly no no). Experimental results show that the problem challenging for tfate-of-the-ar models, including large language models such as GPT3. Our future plans include exploring combina- tion of neural- and knowledge-based approaches. In particular, we believe that commonsense re- sources such ConceptNet (Speer et al., 2017), ATOMIC (Sap et al., 2019), and Common- senseQA (Talmor al., 2019) may be helpful. The challenge is to differentiate between common recommended behavior (or commonly held com- monsense) and what somebody answered in social medía. Going back to the example Figure I, oven
Answers with conditions and contrasts are rarely to be interpreted unknown, yet the model often does so. Mentioning external entities also poses challenge as the model does not have any explicit knowledge about them. Finally, unresponsive an- swers are always to be interpreted unknown, yet the model sometimes (1 (1..4 of errors) fails to prop- crly interpret unresponsive answers and mislabels them as either yes or no.

---

manuals and recipes will always instruct people to preheat the oven. The problem is not to obtain uni- versal ground truth-it does not exist. Instead, the problem is to leverage commonsense and reason- ing to reveal nuances about how indirect answers to yes-no questions ought to be interpreted.
Ethics Statement
While tweets are publicly available, users may not be aware that their public tweets can be used for virtually any purpose. Twitter-YN will comply with the Twitter Terms of Use. Twitter- YN will not include any user information, although we acknowl- edge that it is easily accessible given the tweet id and basic programming skills. The broad temporal spans we work with minimize the opportunities for monitoring users, but is still possible.
Limitations
Twitter-Yh Minter-YN,,aee like any other annotated corpus, is not necessarily representative (and it certainly is not a complete picture) of the problem at hand in our case, interpreting answers to yes-no questions in user generated content. Twitter-YN YN may not transfer to other social media platforms. More im- portantly, our process to select yes-no questions and answers disregards many yes-no questions. In addition working only with English, we disre- gard yes-no questions without question marks (e.g., wonder what y' I'lll think about preheating the oven); the questions we work with are limited to those containing four bigrams. We exclude un- verified accounts to avoid spam and maximize the chances of finding answers (verified accounts are more popular), but this choice ignores many yes-no questions. Despite the limitations, we believe that the variety of verbs and including two temporal spans alleviates this issue. Regardless of size, no corpus will be complete.
While it is far from our purposes, models to in- terpret answers to yes-no questions could be used for monitoring individuals and groups of people. Malicious users could post questions about sensi- tive topies, including politícs, health, and deeply held believes. In turn, unsuspecting users may re- ply with their true opinion, thereby exposing them- selves to whatever the malicious users may want to do (e.g., deciding health insurance quotes based on whether somebody enjoys exercising). That said, this hypothetical scenario also opens the door to users to answer whatever they wish, truthful ant-andiet in the process, mislead the hypothetical monitoring system. The work presented here is about interpreting answers to yes-no questions in Twitter, not figuring the "true" answer. If some- body makes a Twitter persona, we make no attempt to reveal the real person underneath.
A Acknowledgments
Another potential limitation is the choice of la- bels. We are inspired by previous work (Section 2 and 3), some of which work with five labels and others with as many as nine. We found no issues us- ing five labels during the annotation effort, but we certainly have no proof that five is the best choice.
This material is based upon work supported by the National Science Foundation under Grant No. 1845757. Any opinions, findings, and con- clusions or recommendations expressed in this ma- terial are those of the authors and do not necessarily reflect the views of the NSF We would like to knowledge the Chameleon platform (Keahey al., 2020) for providing computational resources.
Lastly, we report negative results using several prompting GPT3
prompts and is the best approach if zero or few training exam- ples are available. Previous work on prompt en- gineering and recent advances in large language models makes us believe that is possible that researchers specializing in prompts may come up with better prompts (and results) than we did. Our effort on prompting and GPT3, however, is an hon- est and thorough effort to the best of our ability with the objective of empirically finding the opti- mal prompts. Since these large models learn from random and maliciously designed prompts (Weh- son and Pavlick, 2022), we acknowledge that other prompts may yield better results.
References
Martín Abadi, Ashish Agarwal, Paul Barham, Bugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow. Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefow- icz, Lukasz Kaiser, Manjunath Kudlur, Josh Leven- berg, Dandelion Mané, Rajat Monga, Sherry Moore. Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, llya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas. Oriol Vinyals, Pete Warden, Mar- tin Wattenbrrg, Martin Wicke, Yuan Yu, and Xiao-

---

qiang Zheng. 2015. TensorFlow: Large-scale ma- chine learning on heterogeneous systems. Software available from tensorflow.org.
Lei Cui, Shaohan Huang, Furu Wei, Chuanqi Tan, Chao- qun Duan, and Ming Zhou. 2017. SuperAgent: A customer service chatbot for E-commerce websites. In Proceedings of ACL 2017, System Demonstrations, pages 97-102, Vancouver, Canada. Association for Computational Linguistics.
Ron Artstein and Massimo Poesio. 2008. Survey article: Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4):555-596.
Cathrine Damgaard, Paulina Toborek, Trine Eriksen, and Barbara Plank. 2021. "I'll be there for you": 'he one with understanding indirect answers. In Pro- ceedings of the 2nd Workshop on Computational Ap- proaches to Discourse, pages 1-11, Punta Cana, Do- minican Republic and Online. Association for Com- putational Linguistics.
Emily M. Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6:587-604
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler. Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. CoRR, abs/2005.1411 14165.
Marie-Cathering de Marneffe, Scott Grimm, and Christopher Potts. 2009. Not a simple yes or no: Uncertainty in indirect answers. In Proceedings of the SIGDIAL 2009 Conference, pages 136,143, Lon- don, UK. Association for Computational Linguistics.
Avia Efrat and Omer Levy. 2020. The turking test: Can language models understand instructions? CoRR, abs/2010. 1982.
Nancy Green and Sandra Carberry. 1999. Interpreting and generating indirect answers. Computational Lin- guistics, 25(3):389 435.
Jean Carletta, Amy Isard, Stephen Isard, Jacqueline Kowtko, Gwyneth Joherty-Sneddon and Anne H. Anderson. 1997. The reliability of a dialogue struc- ture coding scheme. Computational Linguistics. 13-31.
Keith Harrigian, Carlos Aguirre, and Mark Dredze. 2020. Do models of mental health based social media data generalize? In Findings of the Associa- rion for Computational Linguistics: EMNLP 2020, pages 3774-3788, Online, Association for Computa- tional Linguisties.
Zhoujun Cheng. Haoyu Dong. Zhiruo Wang, Ran Jia, Jiaqi Guo, Yan Gao, Shi Han, Jian-Guang Lou, and Dongmei Zhang. 2022. HiTab: hicrarchical table dataset for question answering and natural language generation. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers). pages 1094-11 1110, Dublin, Ireland. Association for Computational Linguistics.
Beth Ann Hockey, Dehorah Rossen-Knill Beverly Spe- jewski, Matthew Stone, and Stephen Isard. 1997. Can you predict responses to yes/no questions? yes, no. and stuff. pages 2267-2270. Publisher Copyright: O 1997 5th European Conference Speech Commu- nication and Technology, EUROSPEECH 1997. All rights reserved.; 5th European Conference on Speech Communication and Technology, EUROSPEECH 1997 Conference date: 22-09-1997 Through 25- 09-1997.
Eunsol Choi, He He, Mohit lyyer, Mark Yatskar, Wen- tau Yih, Yejin Choi, Percy Liang, and Luke Zettle- moyer. 2018. QuAC: Question answering in context. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Lnguage Processing, pages 2174-2184, Brussels, Belgium. Association for Com- putational Linguisties.
Matthew Honnibal and Ines Montani. 2017. spaCy 2: Natural language understanding with Bloom embed- dings, convolutional neural networks and incremental parsing. To appear.
Christopher Clark, Kenton Lee, Ming- Miig-Wei Chang, Tom Kwiatkowski. Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chap- ler of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 19924-2966 Minneapolis. Min- nesota. Association for Computational Linguistics.
Md Mosharaf Hossain, Kathleen Hamilton, Alexis Palmer, and Eduardo Blanco. 2020. Predicting the focus of negation: Model and error analysis. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 8389- 8401, Online. Association for Computational Lin- guistics.
Scott A Crossley, Kristopher Kyle, and Danielle S Mc- Namara. 2017. Sentiment analysis and social cogni- tion engine (seance): An automatic tool for sentiment, social cognition, and social-order analysis. Behavior research methods, 8933)8803-821.
Kate Keahey, Jason Anderson. Zhuo Zhen, Pierre Riteau, Paul Ruth, Dan Stanzione, Men Cevik, Ja- cob Colleran, Haryadi S. Gunawi, Cody Hammock, Joe Mambretti, Alexander Barnes, François Hal- bach, Alex Rocha, and Joe Stubbs. 2020. Lessons

---

learned from the chameleon testbed. In Proceedings of the 2020 USENIX Annual Technical Conference (USENIX ATC 20). USENIX Association.
question answering. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics, pages 6762-6771, Online, Association for Computational Linguistics.
Daniel Khashabi, Tushar Khot, and Ashish Sabharwal. 2020. More bang for your buck: Natural perturba- tion foruus question answering, In Proceedings of the 2020 Conference on Empirical Methods in Natu- ral Language Processing (EMNLP), pages 163-170, Online. Association for Computational Linguistics.
Jiacheng Liu, Alisa Liu, Ximing Lu, Sean Welleck, Pe- ter West, Ronan Le Bras, Yejin Choi, and Hannanch Hajishirzi. 2022. Generated knowledge prompting for commonsense reasoning. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pages 3154-3169, Dublin, Ireland. Association for Compu- tational Linguistics.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins. Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Ken- ton Lee, Kristina Toutanova, Llion Jones, Matthew Keleey, Ming- Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natu- ral questions: A benchmark for question answering research. Transactions of the Association for Compu- lational Linguistics, 7:452-466
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig, 2021. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language processing. CoRR, abs/2107. 13586.
Vasileios Lampos, Daniel Preotiuc-Piette and Trevor Cohn. 2013. A user-centric model of voting intention from social media. In Proceedings of the 51st Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 993-1003, Sofia, Bulgaria. Association for Computational Lin- guistics.
Yinhan Liu, Myle Ott. Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 1907.1190711199
Annie Louis, Dan Roth, and Filip Radlinski. 2020. 'I'd rather just go to bed": Understanding indirect an- swers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 741 1-425,, Online, Association for Computational Linguistics
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficiee prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045-3059, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Jing Ma and Wei Gao. 2020. Debunking rumors on Twit- with tree transformer. In Proceedings of the 28th International Conference on Computational Linguis- rics, pages 5455-5466 Barcelona. Spain (Online). International Committee on Computational Linguis- tics.
Patrick Lewis. Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2020. MILQA: Evalu- ating longulln extractive question answering. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics. pages 7315- 7330, Online. Association for Computational Lin- guistics.
Quinn McNemar. 1947. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2):153-1577
Huayu Li. Martin Renqiang Min, Yong Ge, and Asim Kadav. 2017. A context-aware attention network for
Nikhil Mehta, Maria Leonor Pacheco, and Dan Gold- wasser. 2022. Tackling fake news detection by con- tinually improving social context representations us- ing graph neural networks. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 1563-1380, Dublin, Ireland. Association for Compu- tational Linguistics.
interactive question answering. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 17, page 927-935, New York, NY, USA. Association for Computing Machinery.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 1 1ith International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 4582- 4597, Online. Association for Computational Lin- guistics.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct cloc- tricity? new dataset for open book question swering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381-2391, Brussels, Belgium. Association for Computational Linguistics.
Dayiheng Liu, Yeyun Gong, Jie Fu, Yu Yan, Jiusheng Chen, Daxin Jiang, Jiancheng Lv, and Nan Duan. 2020. RikiNet: Reading Wikipedia pages for natural
George A Miller. 1995. Wordnet: a lexical database for english. Communications of the ACM, 38(1 1):39441.

---

Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019. CoQA: A conversational question answering challenge. Transactions of the Association for Com- putational Linguistics, 7:249-266.
the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume (Long and Short Pupers), pages 4149-4158, Minneapolis, Minnesota. Association for Computational Linguistics.
Deborah Rossen-Knill, Beverly Spejewski, Beth Hockey, Stephen Isard, and Matthew Stone. 1997. Yes/no questions and answers in the map task corpus. IRCS Technical Reports Series.
Albert Webson and Ellie Pavlick. 2022. Do prompt- based models really understand the meaning of their prompts? In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, pages 2300-2344, Seattle, United States. Association for Computational Linguistics.
Krishna Sanagavarapu, Jathin Singaraju, Anusha Kak- ileti, Anirudh Kaza, Aaron Mathews, Helen Li, Nathan Brito, and Eduardo Blanco. 2022. Disentan- gling indirect answers to yes-no questions in real con- versations. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 4677-4695, Seattle, United States. Association for Computational Linguisties.
Wenhan Xiong, Jiawei Wu, Hong Wang, Vivek Kulka- mi, Mo Yu, Shiyu Chang, Xiaoxiao Guo, and William Yang Wang. 2019. TWEETQA: A social media focused question answering dataset. In Pro- ceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 5020- 5031, Florence, Italy. Association for Computational Linguistics.
Maarten Sup, Ronan Le Bras, Emily Allaway, Chan- dra Bhagavatula. Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A. Smith, and Yejin Choi. 2019. Atomic: An atlas machine commonsense for ih-then reasoning. In Proceedings of the Thirty- Third AAAI Conference
Yi Yang, Wen-tau Yih, and Christopher Meek, 2015. WikiQA: A challenge dataset for open-domain ques- tion answering. In Proceedings of the 2015 Con- ference on Empirical Methods in Natural Language Processing, pages 2013-2018, Lisbon, Portugal. As- sociation for Computational Linguistics.
Artificial Intelligence and Thirty-Fire First Innovative Applications of Artificial In- telligence Conference and Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, AAAT 19/AA/19//ATT 19/EAA/ 19. AAAI Press.
Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut. Percy Liang, and Jure Leskovec. 2021. QA-GNN: Reasoning with language models and knowledge graphs for question answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 535- 535,,46, Online. Association for Computational Linguistics.
Emily Sheng, Kai-Wei Chang, Prem Natarajan, and Nanyun Peng. 2021, "nice try, kiddo": Investigating ad hominems dialogue responses. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguisties: Human Language Technologies, pages 750-767, On- line. Association for Computational Linguistics.
Eyal Shnarch, Carlos Alzate, Lena Dankin, Mar- tin Gleize. Yufang Hou, Leshem Choshen, Ranit Aharonov, and Noam Slonim. 2018, Will it blend? blending weak and strong labeled data in a neural network for argumentation mining. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Melbourne, Australia, Association for Computational Linguistics.
A Annotation Guidelines
We give four annotators the following annotation guidelines for each of the five labels. These guide- lines are also used in some of the zero-shot and few-shot prompts.
pages
1. yes (y)
The reply is an affirmative reply to the ques- tion without reservations.
Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of gen- eral knowledge. Thirty-first AAAI conference on artificial intelligence.
Or, the reply states the question in affirmative terms. For example, Mexican food is delicious. states Do you like Mexican food? in positive terms.
Andreas Stolcke, Klaus Ries. Noah Coccaro, Eliza- beth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van Ess- Bss-Dyyema,, and Marie Meteer. 2000. Dialogue act modeling for au- tomatic tagging and recognition of conversational speech. Computational Linguistics, 26(3):339-37.
2. probably yes (py)
The reply is an affirmative reply under certain conditions.
Or, the reply states the question in hopeful terms. For example, am keen on eating tacos this weekend states Do you like Mexican food? in hopeful terms.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A ques- tion answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference
Or, the reply is affirmative but refers to a point

---

of time in the future.
A new of 100 random instances is then collected based on the refined rules. This is once again given for annotation to the four annotators. This process continues until annotators find that the questions and answers identified with the rules are actual ycs-no questions and answers to the questions.
Or, the reply states that an entity (person, group of people, place, organization, etc.) leans towards yes. For example, Q: Do you like Mexican food? A: My whole family loves it.
3. no (n)
The final version of the rules used to scrape yes- no questions and answers from Twitter are detailed in Section 3.1. Section 3.2 provides descriptions of the five interpretations annotators choose from (i.e., annotation guidelines), and Table presents examples. All question-answer pairs (4,442) were annotated by two annotators independently. Four annotators participated in the annotations. Inter- annotator agreement (linearly weighted Cohen's x) 0.68, indicating substantial agreement (Artstein and Poesio, 2008); over 0.8 would be nearly perfect. A After the independent double annotation process, the adjudicator resolves the disagreements to create the ground truth.
The reply is negative reply to the question
without reservations.
Or, the reply states the question in negative terms. For example, Mexican food bland states Do you like Mexican food? in negative terms.
4. probably no (pn)
The reply is negative reply under certain con- ditions.
Or, the reply states that entity (person, group of people, place, organization, etc.) leans towards no. For example, Q: Do you like Mexican food? A: My friends avoid Mexi- can restaurants.
5. unknown (uk)
B.2 Language Variety
A reply that is responsive to the question but does not lean towards yes or no.
The data collection process carried out from May to July 2022. Question tweets are in English ('en' as per the Twitter API). We also use SpaCy to confirm that the reply is in English. Information on the specific type English (American, British, Australian etc.) is not available.
Or, reply that does not address the question at all.
B Data Statement
As per the recommendations by Bender and Fried- man (2018), provide here Data Statement for better understanding of Twitter- YN.
B.3 Speaker Demographic
The questions come from 1,200 unique question tweets posted by verified twitter accounts (as of July 2022). We look specifically at verified users only, since such tweets are more popular and are therefore more likely to have replies. We do not require replies to come from verified accounts. As per Twitter age restrictions, the minimum user age is 13 years for authors of both questions and an- swers. Speakers are not reachable in our scenario, thus demographic information is limited.
B.1 Curation Rationale
We develop Twitter-Ye YN to build and evaluate mod- els to interpret answers to ycs-no questions user- generated content. Twitter-YN. includes yes-no questions and answers from Twitter along with an- notations indicating the interpretation of the an- swers.
Questions come from main tweets (i.e., any tweet except replies and retweets). Answers are reply tweets to the question tweets (retweets do not count as replies). Questions and answers are selected based on manually defined rules. These rules are developed and refined through an iterative process. We scrape Twitter for question-ansee -nnwerr pairs and assign 100 instances four annotators (each annotator gets the same instances). Based on (a) feedback by each of these four annotators on the questions and answers and (b) agreement scores (linearly weighted Cohen's s k), the rules are refined.
B.4 Annotator Demographic
Four annotators (including one adjudicator) are part of the annotation process and development of an- notation guidelines. All of the individuals are grad- uate students who fluent in English. Annotators three men and one woman, and their ages range from 18 40 years old. Ethnic backgrounds are as follows: one Asian and three South Asian. An- notators reported that they are from a middle class economic background.

---

The final version of the annotation guidelines is developed by considering the annotator feedback during pilot annotations. There is no overlap be- tween the questions and answers used in the pilot annotations and the ones that are Twitter- Ww.tter-YN.
second setting, models are trained with answers the same questions seen during training.
Since the test splits are not the same, the results in Tables and 8 are not comparable. Because of the same reason, the results in Table 5 and either Table 7 or cannot be compared either.
B.5 Speaker Situation
D Hyperparameters
Text in Twitter- is retrieved from Twitter be- tween May and July of 2022. Modality of text written (typed by speaker). Twitter allows users (speakers) to edit what they tweet. We use the version of the tweet available as of July 2022 re- gardless of the original posting date. Twitter- YN in- cludes asynchronous interactions since reply tweets are posted only after the tweet with the question is posted. Questions and answers cannot be appear in Twitter simultaneously. The intended audience for the text could be anyone on the Internet.
All RoBERTa-based classifiers were tuned using the train and validation splits. Results (Tables 5 and 7-8, 12-14), were obtained with the test split after the tuning process. We experimented with the following ranges for hyperparameters, and chose the optimal values based on the loss calculated with the validation split:
Learning rate: 1e-5, 2c-5, 3e-5
Epochs: Up to 200 with early stopping (pa-
tience = 3, mhin_delta = 0.01)
Batch size: 16,
B.6 Text Characteristics
Blending factor (a:): 0.2-0.8
Genre of text is in the social media domain. Qusseern-answer instances are generally based on topics such as general advice, day-to-day chores, health, food, finance, music, policies, and poli- tics. The text is informal. Social media cues such as shorter forms of phrases (e.g., lol: laughing out loud), slang, and emojis are common. Most tweets are text only (including Unicode symbols such as emojis), although a few also contain images. Twitter-Yi YN does not include these images.
Tables 9-11 present the tuned hyperparameters in the three experimental settings. These tables complement the results presented in Tables 5, 7, and 8. Hyperparameters were tuned with the train and validation splits.
E Results Pretraining with Related Corpora
Tables 12-14 present results pretraining with yes- no questions and then fine-tuning with Twitter- YN rather than blending. These tables complement Ta- bles 5, 7, and 8. Although pretraining sometimes outperforms blending in two settings, blending al- ways outperforms pretraining in the most important setting: unmatched question and unmatched tem- poral span.
B.7 Recording Quality
N/A
B.8 Other
N/A
B.9
Provenance Appendix
F Detailed Error Analysis
N/A
Tables and 16 detail the error types exemplified in Table 6. Specifically, we provide the percent- ages of errors for each combination of ground truth (gold) and predicted interpretations.
C Results with the other Two Experimental Settings
Tables 7 and present results with the unmatched question and matched temporal span and matched question and matched temporal span settings. These tables complement the results discussed in Section 5. While valid, these settings unrealis- tic in terms of what the model trained and eval- uated with. the first setting, models are trained with question-an question-answer answer pairs posted during the same temporal span than those in the test split. In the
G Experiments Using Prompts and GPT3
Given the success of of large language models and prompt engineering in many tasks (Brown et al., 2020),
keen to see whether they are successful interpreting answers to yes-no ques- tions from Twitter. We experiment with various prompt-based approaches and GPT3. Specifically,

---

All
yes
pyes
pno
unk
Accuracy
F1
Baseline, Majority Class
0.41
0.17
0.41
0.24
0.58
0.00
0.00
0.00
0.00
Baseline, Keyword- Based Rules
0.38
0.55
0.38
0.39
0.38
0.49
0.19
0.25
0.35
with negation cue detector
0.32
0.50
0.32
0.30
0.23
0.44
0.12
0.29
0.31
RoBERTa trained with yes-no questions corpora
BoolQ
0.44
0.36
0.44
0.32
0.59
0.29
0.00
0.00
0.00
Circa
0,58
0.56
0.58
0.54
0.70
0.61
0.38
0.10
0.26
SwDA-IA
0.53
0.45
0.53
0.44
0.65
0.56
0.00
0.00
0.09
Friends-QIA
0,56
0.53
0.56
0.53
0.71
0.58
0.30
0.00
0.28
Twitter-Ye YN (our corpus)
Question only
0.41
0.27
0.41
0.30
0.57
0.24
0.00
0.00
0.00
Answer only
0.53
0.47
0.53
0.47
0.64
0.59
0.00
0.00
0.25
Question and Answer
0.58
0.54
0.58
0.56
0.70
0.60
0.23
0.00
0.45
RoBERTa blending Twitter-YN YN (question and answer) and other yes-no questions corpora
BoolQ 0.5)
0.60
0.61
0.60
0.60s
0.71
0.62
0.37
0.25
0.52
Circa (a = 0.2)
0.61
0.58
0.61
0.58
0.72
0.67
0.30
0.14
0.38
SWDA-IA (a 0.5)
0.58
0,56
0.58
0.57
0.68
0.63
0.26
0.22
0.45
Friends-QLA (a= 0.8)
0.57
0.57
0.57
0.57
0.70
0.62
0.31
0.18
0.40
Table 7: Results with the test split of Twitter YN (our corpus) in the unmatched question and matched temporal span setting, We present results with (a) baselines, (b) training with yes-no questions corpora (other corpora and the training split of Twitter-YNN YN), and (c) blending the training split of Twitter-YY YN and other corpora. While direct comparison is unsound because the training and test splits differ, we observe either the same results or just slightly higher than the ones in the unmatched question and unmatched temporal span setting (Table 5), especially when blending (last block). In other words, training with questions and answers of the new temporal span does not present advantage. Statistical significance with respect to training RoBERTa trained with Twitter-Y YN (Question and Answer, second block) is indicated with and asterisk (McNemar's 's Test (MeNemar, 1947). < 0.05)

---

All
yes
no
pyes
pno
unk
Accuracy
R
F1
F1
F1
F1
F1
F1
Baseline, Majority Class
0.40
0.18
0.40
0.23
0.57
0.00
0.00
0.00
0.00
Baseline, Keyword-Baae sewoodd-Baedd Rules
0.34
0.49
0.34
0.34
0.35
0.45
0.19
0.16
0.29
with negation detector
0.29
0.52
0.29
0.29
0.23
0.45
0.15
0.17
0.27
RoBERTa trained with yes-no questions corpora
BoolQ
0,50
0.36
0.50
0.40
0.63
0.49
0.00
0.00
0.00
Circa
0.55
0.52
0.55
0.51
0.67
0.58
0.31
0.00
0.25
SwDA-IA
0.52
0.60
0.52
0.52
0.68
0.59
0.20
0.13
0.26
Friends- -
0,56
0.55
0,56
0.54
0.65
0.62
0.34
0.00
0.42
Twitter-YN (our corpus)
Question only
0.50
0.40
0.50
0.43
0.63
0.50
0.00
0.00
0.18
Answer only
0.56
0.57
0.56
0.53
0.67
0.58
0.33
0.18
0.32
Question and Answer
0.63
0.74
0.63
0.60
0.71
0.66
0.41
0.13
0,47
RoBERTa blending Twitter-YY (question and answer) and other yes-no questions corpora
BoolQ (a 0.5)
0.66
0.66
0.66
0.64
0.75
0.69
0.40
0.23
0.56
Circa (a 0.5)
0.64
0.61
0.64
0.62
0.76
0.70
0.32
0.14
0.45
SWDA-IA (a = 0.5)
0.64
0.62
0.64
0.62
0.76
0.67
0.38
0.13
0.46
Friends-QIA
0.8)
0.65
0.62
0.65
0.63
0.74
0.71
0.36
0.06
0.56
Table 8: Results with the test split of Twitter-YN (our corpus) in the matched question and matched temporal span setting. We present results with (a) baselines, (b) training with yes-no questions corpora (other corpora and the training split of Twitter-YNN and (c) blending the training split of Twitter-YY YN and other corpora. While direct comparison is unsound because the training and test splits differ, we generally observe higher or the same results than the ones in the unmatched question and unmalched temporal span setting (Table 5). Unsurprisingly. using (different) answers to the same questions in training and testing is beneficial. Note, however, that these results are unrealistic.
Experiment
Learning Rate
Epochs
Batch Size
RoBERTa trained with yes-no questions corpora
BoolQ
3e-5
16
Circa
3e-5
3
16
SWDA-IA
2e-5
Friends-QIA
2e-5
16
Twitter-Y YN (our corpus)
Question
le-5
Answer
3e-5
2
16
Questions and Answer
2c-5
2
16
RoBERTa pretraining with yes-no question corpora + finetuning with Twitter- YN
BoolQ
3e-5
1+1
16
Cirea
3e-5
3+1
16
SWDA-IA
2e-5
3+1
16
Friends-QIA
2e-5
2+1
16
RoBERTa blending Twitter-Ye YN (question answer) and other yes-no questions corpora
BoolQ
3e-5
16
Circa
2e-5
+disambiguation text (GPT3)
2c-5
2
16
SWDA-IA
le-5
2
16
Friends-QIA
2e-5
+disambiguation text (GPT3)
2e-5
2
16
Table Hyperparameters the unmatched question and unmatched temporul span setting found after tuning with the development set. This table complements Table 5.

---

Experiment
Learning Rate
Epochs
Batch Size
RoBERTa trained with yes-no questions corpora
BoolQ
le-5
16
Circa
3e-5
1
16
SWDA-IA
3e-5
2
16
Friends- QIA
3e-5
16
Twitter- YN (our corpus)
Question
le-5
16
Answer
3e-5
2
16
Question and Answer
2c-5
2
16
RoBERTa pretraining with yes-no question corpora finetuning with Twitter YN
BoolQ
le-5
16
Circa
3e-5
1+1
16
SWDA-IA
3e-5
2+2
16
Friends- QIA
3e-5
3+1
16
RoBERTa blending Twitter- YN (question and answer) and other yes-no questions corpora
BoolQ
2e-5
3
16
Circa
3e-5
16
SWDA-IA
le-5
16
Friends-QIA
2e-5
2
16
Table 10: Hyperparameters with the wimatched question and matched temporal span setting found after tuning with the development set. This table complements Table 7.
Experiment
Learning Rate
Epochs
Batch Size
RoBERTa trained with yes-no questions corpora
BoolQ
le-5
3
16
Circa
3e-5
16
SWDA-IA
2e-5
16
Friends-QIA
3e-5
3
16
Twitter YN (our corpus)
Question
2e-5
3
16
Answer
3e-5
2
16
Question and Answer
2e-5
RoBERTa pretraining with yes-no question corpora + finetuning with Twitter-Yi YN
BoolQ
le-5
343
16
Circa
3e-5
1+2
16
SWDA-IA
2e-5
3+2
16
Friends-QIA
3e-5
3+1
16
RoBERTa blending Twitter-YN YN (question and answer) and other yes-no questions corpora
BoolQ
le-5
Circa
2e-5
2
16
SWDA-IA
le-5
2
16
Friends-QIA
2e-5
Table 11: Hyperparameters with the matched question and matched temporal spon setting found after tuning with the development set. This table complements Table 8.

---

Overall
yes
pyes
pno
unk
Accuracy
P
BoolQ
0.51
0.54
0,51
0.41
0.64
0.57
0.03
0.00
0.02
Circa
0.60
0.60
0.60
0.58
0.71
0.62
0.38
0.22
0.45
SwDA-IA
0.55
0.55
0.55
0.51
0.69
0.55
0.36
0.00
0.33
Friends-QUi
0.58
0.55
0.58
0.54
0.70
0.60
0.26
0.00
0.42
Table 12: Results with RoBERTa pretrained relevant yes-no questions corpora and fine-tuned on Twitter-Yi YN in the unmatched question and unmatched temporal span setting. Pretraining obtains worse results than blending (Table 5).
Overall
yes
Pyes
pno
unk
Accuracy
BoolQ
0.59
0.51
0.59
0.54
0.72
0.62
0.00
0.00
0.42
Circa
0.61
0.63
0.61
0.59
0.74
0.63
0.25
0.06
0.47
SwDA-IA
0.57
0.49
0.57
0.51
0.71
0.59
0.00
0.00
0.33
Friends-QIA
0.57
0.55
0.57
0.51
0.69
0.59
0.23
0.00
0.25
Table 13: Results with RoBERTa pretrained relevant yes-no questions corpora and fine-tuned Twitter- YN in the unmatched question and matched temporal span setting.
Overall
yes
no
pyes
pno
unk
Accuracy
P
R
F1
F1
F1
F1
F1
F1
BoolQ
0.59
0.56
0.59
0.56
0.70
0.62
0.19
0.00
0.48
Circa
0.64
0.62
0.64
0.62
0.72
0.71
0.36
0.18
0.47
SwDA-IA
0.62
0.64
0.62
0.61
0.73
0.69
0.39
0.00
0.50
Friends QIA
0.61
0.64
0.61
0.60
0.71
0.67
0.40
0.06
0.50
Table 14: Results with RoBERTa pretrained on relevant yes-no questions corpora and fine-tuned on Twitter- YN in the matched question and matched temporal span setting.

---

Error Type
Ground Truth Predicted
Percentage
Yex distractor
no no
0.88
yes
0.88
probably yes
yes
0.44
yes
probably yes
0.44
unknown
no unknown
0.44
0.44
no
probably yes
0.44
yes
unknown
0.44
No distractor
unknown yes
no unknown
5.29
5.29
yes no
3.52
probably yes
no
3.08
probably
2.20
probably yes
1.76
no
unknown probably yes
0.88
yes
0.88
yes
probably no
0.88
yes
0.88
unknown
yes probably no
0.88
probably yes
0.88
probably
unknown
0.44
probably no unknown
probably yes
0.44
probably no
0.44
probably yes unknown
unknown
0.44
probably yes
0.44
External Entities
yes probably yes probably yes
unknown
3.52
unknown
2.64
yes
2.20
probably yes
no unknown
1.32
no
0.88
unknown
yes
0.88
unknown no
0.88
yes no yes
0.88
0.44
probably no probably no
no yes probably yes
0.44
0.44
yes
0.44
Table 15; Detailed error analysis showing the ground truth and predicted labels of the best model (Part 1). This table complements Table 6.

---

Error Type
Ground Truth
Predicted
Percentage
Social media lexicon
yes probably yes
unknown
5.29
unknown
3.08
unknown yes
no
2.20
1.32
unknown probably yes
1.32
probably
unknown yes no unknown no probably yes
1.32
probably
1.32
probably no
0.88
0.88
prohably yes probably no
0.88
0.44
yes
probably no
0.44
unknown
yes probably no
0.44
0.44
yes
probably yes
0.44
Humor
unknown
2.64
yes
no yes unknown
2.20
1.76
yes
1.32
probably
yes
0.88
probably no
no
0.88
probably no
unknown
0.88
unknown
probably yes
0.44
probably yes
0.44
yes probably no probably yes probably no
probably yes
0.44
yes
0.44
unknown
0.44
probably yes
0.44
Condition contrast
unknown
3.52
probably yes
yes
3.52
yes
3.08
probably no yes
2.64
probably yes
2.64
yes
unknown probably no no unknown no prohably yes yes probably yes
2.64
1.76
unknown probably yes yes
1.76
1.32
1.32
unknown probably no no probably no probably yes unknown
0.88
0.88
0.88
probably yes
0.44
0.44
yes
0.44
probably no probably
unknown
0.44
probably no
0.44
Unresponsive
unknown
6.16
unknown yes probably yes
3.52
unknown
1.76
Table 16: Detailed error analysis showing the ground truth and predicted labels of the best model (Part 2). This table complements Table 6.

---

we experiment with GPT3's text-davinci-002 en- gine, temperature=0.7. frequency petaelcyypppeOOO,,,. pres- ence_ @nce_penalyy=0 and top_ pop_ Our prompts do not exactly follow previous work, but they are inspired by previous work (Liu et al., 2021):
0.58 (Table 5). All the systems using GPT3 obtain
Fls under 0.58. Including the disambiguation to RoBERTa is detrimental in case of Friends-QIA (FI: 0.56) but beneficial in case of Circa (F1: 0.61). Unsurprisingly, few-shot approaches obtain better results than zero-shot prompts (0.53, 0.50 vs. 0.46, 0,41). Perhaps surprisingly. including the annota- tion guidelines is detrimental (zero-shot: 0.46 vs. 0,41, few-shot: 0.53 vs. 0.50).
Zero-Shot. We use prefix prompts (Li and Liang, 2021; Lester al., 2021) since such prompts work well with generative language models such as GPT3 (Liu et al., 2021). As shown by Efrat and Levy, 2020, we also experiment with the same zero-shot prompt including the annotation guide- lines. Specifically, we experiment with two zero- shot prompts:
We acknowledge that prompt engineering may lead to better results with GPT3. We refer the reader to the Limitations section for a discussion.
The first zero-shot prompt includes (a) the yes-no question and answer, and (b) a plain English question asking GPT3 to interpret the answer: What does the given answer mean? Choose only from the following options: yes, probably yes, no, probably no, unknown. Fig- ure 4 illustrates this prompt.
The second zero-shot prompt includes the notation guidelines (Appendix A) for each label, and then the same content the first prompt (i.e., yes-no question and answer fol- lowed by the plain English question asking GPT3 to interpret the answer). Figure illus- trates this prompt.
Few-Shot. We experiment with longer versions the zero-shot prompts that include two exam- ples from the training split per label. Specifically, we add two examples per label (total: 10 exam- ples) to create few-shot versions of the zero-shot prompts detailed above. Figures 6 and 7 illustrate the few-shot prompts (with and without guidelines respectively).
Prompt-Derived Knowledge. Liu et al. (2022) have shown that integrating knowledge derived from GPT3 in question- question-answering models is bene- ficial. We follow similar approach by prompting GPT3 generate disambiguation text given question-answer pair from Twitter-YN. Then we complement the input to the RoBERTa-beel sifier with the disambiguation text. Figure illus- trated this prompt, and Table 17 provides examples of disambiguation texts. As the table shows, certain disambiguation texts are invalid.
clas-
G.1 Results
Table 18 presents the results using GPT3. The best RoBERTa-baen system obtains an overall Fl of

---

Question: <yes-no question tweet> Answer: <reply tweet>
What does the given yes, probably yes, no, probably no, unknown. <answer space>
mean? Choose only from the following options:
Figure 4: Template to generate zero-shot prompts for GPT3
yes: (guidelines for yes]
probably yes: [guidelines for probably yes]
no: (guidelines for no]
ppobbbbly no:
(guidelines for probably no]
unknown:
[guidelines for unknown]
Question: <yes-no question tweet>
Answer: <reply tweet>
What does the given answer mean? Choose only from the following options: yes, probably yes, no, probably no, unknown. <answer space>
Figure 5: Template generate zero-shot prompts with annotation guidelines for GPT3. We refer the reader Appendix A for the annotation guidelines for each class.

---

Question: Do you have any weird things that you' re really really good at?
Answer: can balance things, like tennis rackets or golf clubs, my nose.
The given answer can be interpreted as yes.
Question: Do you guys believe in ghosts?
Answer: i do believe in ghost and i'm not afraid of ghost. The given answer can interpreted as yes.
Question: Do you think that inmates should get access to free university education while in prison?
Answer: if our intent to rehabilitate, we should absolutely offer higher education to inmates.
The given answer can be interpreted as probably yes.
Question: Do you like cereal?
Answer: sometimes, i have to be in the mood for it. The given answer can interpreted as probably yes.
Question: Is it theft take the pen from your hotel room? Answer: It's in the marketing budget. They want you to take the swag.
The given answer can be interpreted as no.
Question: Is it too early for pizza? Answer: every time of day can be pizza time. The given answer can be interpreted as no.
Question: On a work zoom, is it unprofessional to obviously be in your bed?
Answer: anyone who would be working with you needs to know bed boy.
The given answer can be interpreted as probably no.
Question: Is it a crime to get extra vaccination shots?
Answer: we'll all be doing for the rest of our lives so hope not.
The given answer can be interpreted as probably no.
Question: Do you pee in the ocean?
Answer: tell gio to cut his hair, he looks like an idiot. be interpreted unknown.
The given answer
Question: Do you believe your phone is listening to you? Answer: even if it does, i don't mind. i've got nothing to hide. The given answer be interpreted unknown.
Question: <yes-no question tweet>
Answer: <reply-tweet>
Choose the correct interpretation: yes, no, probably yes, probably no, unknown. <answer space>
Figure 6: Template to generate few-shot prompts for GPT3

---

yes: (guidelines for yes]
probably yes: [guidelines for probably yes]
no: [guidelines for no]
probably no:
(guidelines for probably no]
uunknow:
[guidelines for unknown]
Examples:
(examplel for yes] (example2 for yes]
[examplel for probably yes] [example2 for probably yes]
(example1 for no] [example2 for no]
[examplel for probably no] [example2 for probably no]
[examplel for unknown] (example2 for unknown]
Question: <yes-no question tweet>
Answer: <reply tweet>
What does the given answer mean? Choose only from the following options: yes, probably yes, no, probably no, unknown. <answer space>
Figure 7: Template to generate few-shot prompts with annotation guidelines for GPT3. refer the reader Appendix A for the annotation guidelines for each class, and Figure 6 for examples.
Question: Do you have any weird things that you're really really good at? Answer: i can balance things, like tennis rackets or golf clubs, on my nose.
The given answer can be interpreted as "yes", because the
answerer mentions a list of weird things he or she is really good at.
Question: <yes-no question tweet> Answer: <reply tweet> <answer space>
Figure 8: Template generate the prompt elicit knowledge (or disambiguation text) regarding the interpretation of an answer with GPT3 (i.c., justification for the interpretation). The prompt includes an example of disambiguation text (e-g., the answerer mentions a list of weind things he or she really good at), GPT3 always provides answers following the template The given answer be interpret [label], because the answerer /disambiguation text). We use the disambiguation text as an additional input for the RoBERTa-baace classifier.

---

Question
Answer
G.
Disambiguation (from GPT3)
Valid?
My name regularly gets auto- corrected to "Saataism". Get surprising number of messages addressed Do you have an autocorrect hell?
An old phone of mine used change yorkshire workshy. outrageous
y
Does not mention having an au- False tocorrect hell
Do you still trust anything the media says?
Modern media make collective lie to become the truth
Mentions how media manip- True ulate the truth, they do not trust anything the media says
Do you still super follow mc?
had to chose between spo- tify and superfollowing
uk
Chose spotify over superfollow- False ing the asker
Do you have the courage to be disliked?
am not able to answer this question.
Admits they cannot answer the True question, indicating they do not have the courage to be disliked
Do you bring your phone while using the bathroom?
Depends how long I think will be there!
py
Implies that they do bring False their phone while using the bath- room
Table 17: Disambiguation texts produced by GPT3. 'G' indicates the gold label. "Valid?" indicates whether the disambiguation is valid based on the gold label. In the first example, the answer mentions an "auto correct hell," however, the disambiguation disregards this critical term. In the second and fourth examples, the answer correctly interpreted by GPT3 by specifying media manipulation and capturing the fact that not answering the question shows lack courage respectively. In the third and fifth examples, GPT3 incorrectly interprets that the author of the answer prefers Spotify and does not bring phone in the bathroom respectively. These examples show that GPT3 sometimes produces invalid disambiguation texts. As Table 5 shows, the disambiguation texts can be both beneficial and detrimental.
All
yes
no
pyes
pno
unk
Accuracy
P
R
F1
F1
F1
F1
F1
F1
GPT33 Zero-shot
0.44
0.55
0.44
0.46
0.56
0.51
0.16
0.13
0.43
+ annotation guidelines
0.42
0.55
0.42
0.41
0.51
0.48
0.12
0.00
0.40
GPT3, Few-shot
0.54
0.55
0.54
0.53
0.67
0.55
0.14
0.30
0.49
annotation guidelines
0.52
0.57
0.52
0.50
0.66
0.50
0.15
0.16
0.48
RoBERTa blending Twitter-YN (question and answer) and other yes-no questions corpora
Circa (Table 5)
0,60
0.59
0,60
0.58#
0.72
0.65
0.37
0.25
0.44
+ disambiguation text (GPT3)
0.62
0.62
0.62
0.61
0.75
0.67
0.35
0.22
0.57
Friends-QIA (Table
0.60
0.59
0.60
0.58:
0.71
0.63
0.38
0.20
0.45
+ disanbiguation text (GPT3)
0.56
0.56
0.56
0.56
0.70
0.63
0.36
0.24
0.37
Table 18: Results obtained with GPT3 in the unmatched question and unmatched temporal span scenario. These results are moderately worse than the ones obtained with RoBERTa-based classifiers (Table 5), although the zero- shot prompts obtain results better than the baselines without requiring any annotations. Statistical significance with respect few-shot GPT3 (first block) is indicated with an dagger (f) (McNemar's Test, p 0.05).


Reviews:
[{"review_id": "dmHLfXQG93", "reviewer": "Reviewer_9N5z", "summary": "", "questions": "- How were the 4,442 yes-no question-answer pairs curated? Were there any specific criteria or filters applied during the collection process?\n\n- Were there any regional or cultural variations in the way users responded to yes-no questions?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. We appreciate your comments.\n\n**Reasons to Reject:**\n\nRE: Model Details, visual descriptions of architectures, fine-tuning process, etc.:\n- We cover this content in Section 5, 5.1 and 5.2, including citations and hyperparameter search (train / val / test split, etc.). Appendices D include the specific search space for hyperparameters and the final values. Appendix G includes the detailed prompts (Figures 5-8), GPT versions, and parameters (temperature, frequency penalty, etc.). Regarding reproducibility, please see Tables 9-14 with detailed hyperparameters. Also, note that we will release the dataset and our implementation to replicate our results (footnote 1). \n\n    Based on this comment, we will add more details and move some content from the appendix to the main paper. We appreciate the suggestion.\n\nRE: Comparison with Other Platforms: While the focus is on Twitter, it might be interesting to see if the findings are consistent across other social media platforms or if there are platform-specific nuances. \n- We agree. We believe that it is clear from page 1 that we only work with Twitter and we acknowledge it in the Limitations. Thank you for the suggestion. We will add working with other social media platforms to our research agenda. \n\n\n**Questions for the  authors:**\n\nRE: How were the 4,442 yes-no question-answer pairs curated? Were there any specific criteria or filters applied during the collection process??\n- Section 3.1 explains how the corpus was curated, including our rationale. Section 3.2 includes a shorter version of the annotation guidelines and details the annotation process. Appendix A provides a comprehensive Data Statements that provides more details (annotators, additional rationale, etc.)\n    As explained in those sections, we use filters to identify yes-no questions. As previous work has shown (Section 2), identifying yes-no questions in dialogue is not challenging. The challenging problem is interpreting the answers to those questions.\n\n\nRE: Were there any regional or cultural variations in the way users responded to yes-no questions?\n- Probably yes. This is true of any NLP problem, especially problems that work with dialogues. That said, this question is outside of the scope of the current paper as Twitter does not provide detailed regional or cultural information.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- New Corpus Creation: The paper presents a corpus of 4K+ binary question-answer pairs from Twitter. This dataset can serve as a valuable resource for the social media as well as QA research.\n\n- Linguistic Analysis: The papers analyzes the linguistic characteristics of answers, whether they lean towards 'yes', 'no', or are 'unknown'. The linguistic perspective present the nuanced understanding  required for this corpus as well as  the challenges in interpreting user-generated content.\n\n- Detailed Model Evaluation: The paper evaluates the performance of large language models, after fine-tuning and blending with other corpora. The experimentation presents a comprehensive comparison with of the current state-of-the-art models.", "reasons_to_reject": "Model Details: The paper lacks specifics about model design large language models were tested. A more detailed analysis of the models, visual description of their architectures, and the fine-tuning process would be beneficial for reproducibility and a deeper understanding of the results.\n\nComparison with Other Platforms: While the focus is on Twitter, it might be interesting to see if the findings are consistent across other social media platforms or if there are platform-specific nuances.", "questions_to_authors": "- How were the 4,442 yes-no question-answer pairs curated? Were there any specific criteria or filters applied during the collection process?\n\n- Were there any regional or cultural variations in the way users responded to yes-no questions?", "paper_topic_and_main_contributions": "The paper delves into the intriguing challenge of interpreting answers to yes-no questions in the context of social media, specifically Twitter. Given the informal and often ambiguous nature of user-generated content, this is a pertinent issue in the realm of question answering.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZMaEeRKZPa", "reviewer": "Reviewer_8ozY", "summary": "", "questions": "Questions For The Authors\n\nA. Introduction & Motivation: The introduction does not adequately motivate the task. Could you clarify the research gap and whether other works cover it?\n\nB. Line 064: Regarding the qualitative analysis of errors made by your model, what significant conclusions were derived?\n\nC. Line 069: The statement about answers being available is a bit unclear. How did you ascertain that the answers are present?\n\nD. Line 137: What do you mean by \"interpreting answers\" in this context? Are you classifying answers into yes/no/unknown categories?\n\nE. Section 3: Why didn't you use a classifier trained on BOOLQ yes-no (which identifies questions with high precision) as opposed to a heuristic approach?\n\nF. Line 172 & 182: You mention that avoiding other questions with 'wh' words is crucial for precision. Could you quantify this improvement?\n\nG. Line 174: The repetition of the word \"avoid\" makes the sentence cumbersome. Consider rephrasing.\n\nH. Line 178: What proportion of questions with numbers pertains to cryptocurrencies or stocks? Have any clustering, topic modelling, or classification methods been applied?\n\nI. Line 201: While the approach you've mentioned is commendable, how did you ensure the prevention of data leakage, especially considering duplicates or paraphrased questions from different years?\n\nJ. Line 243: Was there a research ethics agreement in place for recruiting graduate students? What were their demographics, and were they native speakers?\n\nK. Line 270: The title \"Are Keywords Enough to Interpret Answers?\" seems overstated, given that the keyword list omits obvious examples like \"yep\" for yes.\n\nL. Line 304: Your analysis indicates that the \"me\" pronoun is more common with \"Yes\". Why is this significant or interesting? Could you provide a discussion on this?\n\nM. Line 354: You mention that the results are reported using the test split. Could you specify where these results are reported or provide a direct reference/link within the paper?\n\nN. Line 371: The statement about blending is not clear. Could you clarify what you mean by \"many yes-no questions\" and where they originate from?\n\nO. GPT Experiments: Can you provide more detailed descriptions of your experiments involving the GPT model?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review and questions.\n\n**Reasons to reject**\n\nRE: The paper currently reads more like a draft than a polished conference submission. It requires thorough restructuring and proofreading due to unclear sentences, repetitions, and confusing sections.\n- We would need specifics to address this weakness. The other three reviewers appear to disagree.\n\nRE: The paper fails to convey the significance of the task effectively. There's a lack of context and clear delineation of the research gaps in the introduction. A more comprehensive introduction that outlines the context, importance, and absence of similar resources would benefit a broader audience.\n- Please see Section 1, line 066 onward (Motivation). Additionally, recent previous work (Section 2) supports that the problem is worthwhile. Finally, the results are relatively low even with large language models, demonstrating that the problem is challenging (Sections 6 and 7). Section 2 distinguishes between traditional question answering (including social media, line 127-149), previous work on yes-no questions, and our work.\n\nRE: Several claims in the paper lack supporting evidence. Examples include ambiguous statements about improvements in precision without concrete numbers and criticisms of keyword-based methods based on a limited set of keywords.\n- Assuming “improvements in precision without concrete numbers” refers to line 172 (avoiding questions with wh-words to identify yes-no questions), the precision is 100% in our sample. We did not observe any yes-no questions with a wh-word. Just to clarify, wh-words are who, where, etc. (line 162-163) (and they expect a person, location, etc. for an answer; not a yes or no). Thank you for pointing this out. We will include the number.\n\n- Our goal is to present an unsupervised keyword-based baseline. Any keyword-based method will be based on a limited set of keywords by definition. Additionally, note that many answers that contain yes keywords must be interpreted No and vice versa (Figure 3). Adding more keywords would not bypass this issue. We believe that explaining this point (line 270-285) is an important analysis which helps understand the dataset and difficulty of the problem.\n\nRE: Shallow analysis: The paper provides a shallow linguistic dataset analysis without clear takeaways or conclusions. If the analysis isn't enhancing the paper's value or clarity, it might be better suited for the appendix. Conversely, essential details in the appendix, such as dataset splitting justifications, should be integrated into the main text.\n- We believe it is important to justify the diversity (bigrams, many verbs, etc.) of the new dataset and provide a linguistic analysis (Section 4.1). We acknowledge that the importance of analyses is subjective, but we do argue that the analyses belong to the main paper.\n\nRE: Conversely, essential details in the appendix, such as dataset splitting justifications, should be integrated into the main text.\n- The dataset splits (train / validation / test) using the three settings (matched/unmatched; question/temporal) (and the justification) are described in the main paper (line 328-342). The main paper includes results with the most realistic setting, and Appendix C includes results with the two least realistic settings.\nWe are confident this is the right content for the main paper and appendices.\n\nRE: The methods proposed may need reconsideration, especially given potential issues with dataset splitting.\n- The results presented in the paper are for the “unmatched question unmatched temporal span” setting (as stated in line 330-339). As explained there, this means that (by definition) the question-answer pairs in the training and test split (a) do not have any overlapping question and (b) were published in non-overlapping temporal spans.\nTherefore, there are no issues with dataset splitting.\n\nRE: The proposed dataset is quite small.\n- The dataset has 4,442 question-answers pairs (line 006). This is subjective, but we acknowledge the corpus size in the Limitations. No corpus accounts for everything. Further, like everybody else, we are bound by a limited annotation budget. That said, we want to point out that the experimental results show that our models learn from the dataset (Table 5).\n\n\n**Questions for the authors**\n\nA. \nRE: Introduction & Motivation: The introduction does not adequately motivate the task. Could you clarify the research gap and whether other works cover it?\n- Please see Section 1, line 066 onward (Motivation). Additionally, recent previous work (Section 2) supports that the problem is worthwhile. Finally, the results are relatively low even with large language models, demonstrating that the problem is challenging (Sections 6 and 7).\nSection 2 distinguishes between traditional question answering (including social media, line 127-149), previous work on yes-no questions, and our work.\n\nB. \nRE: Line 064: Regarding the qualitative analysis of errors made by your model, what significant conclusions were derived?\n- We describe and exemplify the most frequent error types in Section 6, we refer the reviewer to that section for an answer.\n\nC. \nRE: Line 069: The statement about answers being available is a bit unclear. How did you ascertain that the answers are present?\n- Let us make it more clear: We collect yes-no questions from Twitter. The reply tweets to the tweets containing the yes-no questions are the answers to those questions. So, the answers are available (i.e., they do not need to be retrieved from a document collection). The problem is to figure out the correct interpretation of the answers. Note that generally question answering is about finding the answer to a question (line 066-071).\n\n    We will clarify this; thank you for pointing out your confusion.\n\nD. \nRE: Line 137: What do you mean by \"interpreting answers\" in this context? Are you classifying answers into yes/no/unknown categories?\n- Yes, as explained in line 027-034 and Figure 1.\n\nE. \nRE: Section 3: Why didn't you use a classifier trained on BOOLQ yes-no (which identifies questions with high precision) as opposed to a heuristic approach?\n- BoolQ contains only yes-no questions, so we would still need negative examples. While your suggestion is valid (and training such a classifier would not be difficult), our heuristic approach requires less effort and is very effective: the annotation process did not identify issues. We also would like to point out that identifying the yes-no questions is not a major contribution; it is only a step towards the creation of our dataset.\n\nF. \nRE: Line 172 & 182: You mention that avoiding other questions with 'wh' words is crucial for precision. Could you quantify this improvement?\n- The precision is 100% in our sample. We did not observe any yes-no questions with a wh-word. Just to clarify, wh-words are who, where, why, etc. (line 162-163) (and they expect a person, location, reason, etc. for an answer; not a yes or no).\n\n    Thank you for the question, we will include the number and reword.\n\nG. \nRE: Line 174: The repetition of the word \"avoid\" makes the sentence cumbersome. Consider rephrasing.\n- Thank you for helping us improve readability. We will use “discarding tweets with X” and other similar paraphrases to minimize potential confusions.\n\nH. \nRE: Line 178: What proportion of questions with numbers pertains to cryptocurrencies or stocks? Have any clustering, topic modeling, or classification methods been applied?\n- Out of a random sample of 100 instances, 68% of questions with numbers pertained to cryptocurrencies and stocks. The remaining ones covered politics (e.g. number of votes, years) and sports (e.g., scores). We will include these details in the paper. This is a manual analysis. The methods suggested by the reviewer scale but would introduce errors.\n\nI. \nRE: Line 201: While the approach you've mentioned is commendable, how did you ensure the prevention of data leakage, especially considering duplicates or paraphrased questions from different years?\n- By definition (line 330-339), the “unmatched question unmatched temporal span” setting does not have in the training and test splits (a) questions published in overlapping temporal spans or (b) the same questions. Therefore, there is no data leakage.\nWhile this is not a proof, the results from training with Twitter-YN (question+answer, F1: 0.56) indicate that there is no data leakage, otherwise the F1 would be much higher.\n\nJ. \nRE: Line 243: Was there a research ethics agreement in place for recruiting graduate students? What were their demographics, and were they native speakers?\n- We are employed by large universities. The research presented in the paper was done in accordance to our research guidelines and regulations, including ethics. We have mandated training and are highly regulated.\nPlease find details about the annotators in the Data Statement (Appendix B). They are not native speakers but have lived in an English-speaking country for years. Let us also point out that not everyone in social media using English is a native speaker, so the benefit of having native speakers as annotators is not obvious to us. Annotators are co-authors as they are a key contributor to the paper.\n\nK. \nRE: Line 270: The title \"Are Keywords Enough to Interpret Answers?\" seems overstated, given that the keyword list omits obvious examples like \"yep\" for yes.\n- As stated earlier, no list of keywords is complete. Additionally, note that many answers that contain _yes_ keywords must be interpreted No and vice versa (Figure 3). So more than keyword matching is needed: no matter how long the list of keywords, as stated in the paper ( line 263-269), keyword matching is not enough because keywords are misleading.\n\nL. \nRE: Line 304: Your analysis indicates that the \"me\" pronoun is more common with \"Yes\". Why is this significant or interesting? Could you provide a discussion on this?\n- This is an empirical finding. We did not anticipate this finding. We hesitate to make strong claims, but the field of Psychology has made very interesting insights about pronoun usage, in particular first person pronouns (see, for example, the work by James Pennenbaker).\n\nM. \nRE: Line 354: You mention that the results are reported using the test split. Could you specify where these results are reported or provide a direct reference/link within the paper?\n- “[...] we report results with the test split” means that all the results in the paper (Table 5 and appendices) are the results obtained with the test split (after training and validating with the training and validation splits).\n\nN. Line 371: \nRE: The statement about blending is not clear. Could you clarify what you mean by \"many yes-no questions\" and where they originate from?\n- Thank you, we will clarify this. “Many” here refers to the data coming from related corpora and Twitter-YN (line 365-366).\n\nO. \nRE: GPT Experiments: Can you provide more detailed descriptions of your experiments involving the GPT model?\n- As stated in Section 5.2, we provide the exact version, arguments, and other necessary details in Appendix F. Figures 4-8 detail the exact prompts we use. We are confident stating that  that’s all that is needed to use the GPT API. \n\n    Additionally, we will also release our code (footnote 1).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The task addressed in the paper is genuine, and the curated dataset can potentially be valuable for both the academic community and industry.\n* By evaluating several baseline models, the authors provide valuable insights into the challenges of the task.", "reasons_to_reject": "* The paper currently reads more like a draft than a polished conference submission. It requires thorough restructuring and proofreading due to unclear sentences, repetitions, and confusing sections.\n* The paper fails to convey the significance of the task effectively. There's a lack of context and clear delineation of the research gaps in the introduction. A more comprehensive introduction that outlines the context, importance, and absence of similar resources would benefit a broader audience.\n* Several claims in the paper lack supporting evidence. Examples include ambiguous statements about improvements in precision without concrete numbers and criticisms of keyword-based methods based on a limited set of keywords.\n* Shallow analysis: The paper provides a shallow linguistic dataset analysis without clear takeaways or conclusions. If the analysis isn't enhancing the paper's value or clarity, it might be better suited for the appendix. Conversely, essential details in the appendix, such as dataset splitting justifications, should be integrated into the main text.\n* The methods proposed may need reconsideration, especially given potential issues with dataset splitting.\n* The proposed dataset is quite small.", "questions_to_authors": "Questions For The Authors\n\nA. Introduction & Motivation: The introduction does not adequately motivate the task. Could you clarify the research gap and whether other works cover it?\n\nB. Line 064: Regarding the qualitative analysis of errors made by your model, what significant conclusions were derived?\n\nC. Line 069: The statement about answers being available is a bit unclear. How did you ascertain that the answers are present?\n\nD. Line 137: What do you mean by \"interpreting answers\" in this context? Are you classifying answers into yes/no/unknown categories?\n\nE. Section 3: Why didn't you use a classifier trained on BOOLQ yes-no (which identifies questions with high precision) as opposed to a heuristic approach?\n\nF. Line 172 & 182: You mention that avoiding other questions with 'wh' words is crucial for precision. Could you quantify this improvement?\n\nG. Line 174: The repetition of the word \"avoid\" makes the sentence cumbersome. Consider rephrasing.\n\nH. Line 178: What proportion of questions with numbers pertains to cryptocurrencies or stocks? Have any clustering, topic modelling, or classification methods been applied?\n\nI. Line 201: While the approach you've mentioned is commendable, how did you ensure the prevention of data leakage, especially considering duplicates or paraphrased questions from different years?\n\nJ. Line 243: Was there a research ethics agreement in place for recruiting graduate students? What were their demographics, and were they native speakers?\n\nK. Line 270: The title \"Are Keywords Enough to Interpret Answers?\" seems overstated, given that the keyword list omits obvious examples like \"yep\" for yes.\n\nL. Line 304: Your analysis indicates that the \"me\" pronoun is more common with \"Yes\". Why is this significant or interesting? Could you provide a discussion on this?\n\nM. Line 354: You mention that the results are reported using the test split. Could you specify where these results are reported or provide a direct reference/link within the paper?\n\nN. Line 371: The statement about blending is not clear. Could you clarify what you mean by \"many yes-no questions\" and where they originate from?\n\nO. GPT Experiments: Can you provide more detailed descriptions of your experiments involving the GPT model?", "paper_topic_and_main_contributions": "The paper presents research on understanding yes-no questions that are written freely in the social media domain. The authors introduce a new dataset with over 4,000 question-answer pairs and human-annotated yes-no labels. They also test some basic models to show the task's challenges.\n\n", "excitement": 2, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "See \"Questions For The Authors\""}, "extra_text": {}}, {"review_id": "ZyzyHztjbe", "reviewer": "Reviewer_heYF", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Your recommendation of our work to EMNLP 2023 is highly appreciated. We thank you for your positive reviews.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper focuses on an interesting problem that even SOTA models don’t seem to be able to do well on. It is an interesting avenue of research and the dataset seems valuable. The methodology of the work is sound, and the authors have clearly made an effort to outline their experimental and dataset-related details in the Appendix.\n", "reasons_to_reject": "N/A. I think it is a great paper and it should be presented at EMNLP.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents a dataset that focuses on interpreting answers to yes/no questions in social media. This is challenging because the goal isn’t to search for a keyword within the response, but to interpret it. The paper presents this hand-annotated dataset and experiments with several baselines to showcase that this problem warrants further exploration.\n", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "bFwYC4lDMy", "reviewer": "Reviewer_v4HH", "summary": "", "questions": "Will the corpus be made available? What will be the license of use?\n\nHow was the statistical confidence calculated?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Your recommendation of our work to EMNLP 2023 is highly appreciated. We thank you for your positive reviews. \n\n**Questions for the authors**\n\nRE: Will the corpus be made available? What will be the license of use?\n- Yes. Twitter-YN will be made available (as stated in the footnote of page 1). We will follow Twitter Terms of Service and release our annotations using “CC BY.”\n\nRE: How was the statistical confidence calculated?\n- We refer the reviewer to the McNemar’s Test (McNemar, 1947), as cited in the caption for Table 5.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The corpus. It follows the standard approaches for data gathering and annotation.\n\nLinguistic profile of yes/no question answering.\n\nThe explored models are reasonable and well-described, and the error analysis may help improve the model.", "reasons_to_reject": "None", "questions_to_authors": "Will the corpus be made available? What will be the license of use?\n\nHow was the statistical confidence calculated?", "paper_topic_and_main_contributions": "The authors propose a new yes/no QA corpus for exploring the type of answer (yes, no, probably yes, probably no and unknown). This corpus comprises 4,442 pairs of QA from Twitter (1.200 questions; about 3.7 answers per question). The authors provide a linguistic profile of the corpus. Then, they compare different methods: baselines (majority and rule-based), RoBERTa and GPT3. In addition, they tried fine-tuning with another corpus and then the proposed one, as well as blending the corpora.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a dataset for the task of interpreting responses to yes/no questions asked on Twitter. This can often be challenging because the responses may not explicitly contain the words "yes" or "no", and the interpretation might require commonsense knowledge. The authors created a dataset of 4K+ QA pairs and interpretations representing this task. The reviewers generally agreed that the dataset can serve as a valuable resource. Reviewer 8ozY had concerns about the motivation, experimental methodology, and the organization of the paper, but the authors' responses seem convincing to me.

There are existing datasets for this task, e.g.: BoolQ, but not in this domain (social media, particularly Twitter), making this a potentially useful resource.
Decision: Accept-Findings